https://flyte.org
Join Slack
Hi there! I'm trying to deploy Flyte in Azure and when I try to run my first workflow from the examp...
# flyte-support
f
Hi there! I'm trying to deploy Flyte in Azure and when I try to run my first workflow from the examples repo, I get this error:
Copy code
$ pyflyte run --project flytesnacks --domain development --remote basics/hello_world.py my_wf

Failed with Exception Code: USER:ValueError
Value error!  Received: 400. Request to send data <azure blob storage URI> failed.
Has anyone seen this before? Is it 
flytekit
that tries to upload to blob storage or is this error coming from 
flyteadmin
? I haven't set up any kind of authentication yet, so that could be related as well. 
flyteadmin
also doesn't show any logs even with a high log level Thanks in advance!
g
Did you update the signed URL in the flyteadmin configmap
Copy code
signedUrl:
        stowConfigOverride:
          endpoint: <http://flyte:30084>
you should use your dns
f
Ah no I haven't! Is that under 
flyteadmin:
?
g
yes
f
ah thanks! let me try
do you point that to your ingress?
or just to your service name like 
<http://flyteadmin:30084>
g
yes, ingress
f
didn't work 😕 here's the rest of my storage config, in case it helps:
Copy code
storage:
  type: custom
  bucketName: flyte
  custom:
    container: "flyte"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "${storage_account_name}"
        key: "${storage_account_key}"
  enableMultiContainer: false
The two variables are replaced by terraform
is it 
flytectl
that tries to upload something to blob storage? or does it talk to flyteadmin which then tries to upload it?
g
pyflyte tries to upload the user code to s3 by using data proxy in the flyteadmin.
is there any error in flyteadmin
f
thanks. No errors in 
flyteadmin
logs 😕 I can't get flyteadmin to show more logs despite increasing the log level
(i'm on azure btw)
how is that port exposed? I can't see any service or ingress rule for it also I'm using nginx as an ingress controller and it's a bit flaky with http2/grpc calls in my current environment (self-signed cert) -- in case it could be related
Alright, I’m giving up for tonight. If you think of anything let me know and I’ll try in the morning! Thanks for your help today 🙏
Ping @millions-queen-16335 maybe as I’ve seen you discuss similar issues on discuss.flyte.org 😊
g
could you use pyflyte --verbose run? you will show more error message
f
Will do 👍
g
thanks! btw, you’re using Azure Blob Storage, right? if so, you don’t need to add stowConfigOverride. I thought you use minio.
f
I do! Ok good to know
Hi there! I figured out the issue. It seems like there is a missing header in `flytekit`https://github.com/flyteorg/flytekit/blob/master/flytekit/remote/remote.py#L800 When enabling 
--verbose
I saw that my 400 was due to this error:
Copy code
<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>MissingRequiredHeader</Code>
    <Message>
        An HTTP header that's mandatory for this request is not specified.
        RequestId:ebd5dfbc-f01e-001d-5074-cb4f26000000
        Time:2023-08-10T10:24:58.6553767Z
    </Message>
    <HeaderName>x-ms-blob-type</HeaderName>
</Error>
I manually patched the code to add 
x-ms-blob-type: BlockBlob
as a header and the file was successfully uploaded. However, I still got an exception: 
FlyteValueException: Value error!  Received: 201. Request to send data
because this check is too strict. Happy to make a PR to to at enable passing additional headers if that helps
g
Thank you @kind-kite-58745. good to know you’ve fixed it.
k
I was mentioned by accident, forwarding @flat-exabyte-79377
🙏 2
c
@flat-exabyte-79377 I just ran into this issue and your fix worked for me! Thanks. Looks like all you need to do is sign your commits to get them to master?
f
I'm glad it helped! thanks for the reminder, I've justed signed it off now
211 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by