https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
v

Victor Delépine

08/09/2023, 1:36 PM
Hi there! I'm trying to deploy Flyte in Azure and when I try to run my first workflow from the examples repo, I get this error:
Copy code
$ pyflyte run --project flytesnacks --domain development --remote basics/hello_world.py my_wf

Failed with Exception Code: USER:ValueError
Value error!  Received: 400. Request to send data <azure blob storage URI> failed.
Has anyone seen this before? Is it 
flytekit
that tries to upload to blob storage or is this error coming from 
flyteadmin
? I haven't set up any kind of authentication yet, so that could be related as well. 
flyteadmin
also doesn't show any logs even with a high log level Thanks in advance!
k

Kevin Su

08/09/2023, 5:28 PM
Did you update the signed URL in the flyteadmin configmap
Copy code
signedUrl:
        stowConfigOverride:
          endpoint: <http://flyte:30084>
you should use your dns
v

Victor Delépine

08/09/2023, 5:30 PM
Ah no I haven't! Is that under 
flyteadmin:
?
k

Kevin Su

08/09/2023, 5:30 PM
yes
v

Victor Delépine

08/09/2023, 5:30 PM
ah thanks! let me try
do you point that to your ingress?
or just to your service name like 
<http://flyteadmin:30084>
k

Kevin Su

08/09/2023, 5:52 PM
yes, ingress
v

Victor Delépine

08/09/2023, 6:18 PM
didn't work 😕 here's the rest of my storage config, in case it helps:
Copy code
storage:
  type: custom
  bucketName: flyte
  custom:
    container: "flyte"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "${storage_account_name}"
        key: "${storage_account_key}"
  enableMultiContainer: false
The two variables are replaced by terraform
is it 
flytectl
that tries to upload something to blob storage? or does it talk to flyteadmin which then tries to upload it?
k

Kevin Su

08/09/2023, 7:18 PM
pyflyte tries to upload the user code to s3 by using data proxy in the flyteadmin.
is there any error in flyteadmin
v

Victor Delépine

08/09/2023, 7:20 PM
thanks. No errors in 
flyteadmin
logs 😕 I can't get flyteadmin to show more logs despite increasing the log level
(i'm on azure btw)
how is that port exposed? I can't see any service or ingress rule for it also I'm using nginx as an ingress controller and it's a bit flaky with http2/grpc calls in my current environment (self-signed cert) -- in case it could be related
Alright, I’m giving up for tonight. If you think of anything let me know and I’ll try in the morning! Thanks for your help today 🙏
Ping @Mathias Andersen maybe as I’ve seen you discuss similar issues on discuss.flyte.org 😊
k

Kevin Su

08/09/2023, 9:52 PM
could you use pyflyte --verbose run? you will show more error message
v

Victor Delépine

08/09/2023, 9:52 PM
Will do 👍
k

Kevin Su

08/09/2023, 9:53 PM
thanks! btw, you’re using Azure Blob Storage, right? if so, you don’t need to add stowConfigOverride. I thought you use minio.
v

Victor Delépine

08/09/2023, 9:54 PM
I do! Ok good to know
Hi there! I figured out the issue. It seems like there is a missing header in `flytekit`https://github.com/flyteorg/flytekit/blob/master/flytekit/remote/remote.py#L800 When enabling 
--verbose
I saw that my 400 was due to this error:
Copy code
<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>MissingRequiredHeader</Code>
    <Message>
        An HTTP header that's mandatory for this request is not specified.
        RequestId:ebd5dfbc-f01e-001d-5074-cb4f26000000
        Time:2023-08-10T10:24:58.6553767Z
    </Message>
    <HeaderName>x-ms-blob-type</HeaderName>
</Error>
I manually patched the code to add 
x-ms-blob-type: BlockBlob
as a header and the file was successfully uploaded. However, I still got an exception: 
FlyteValueException: Value error!  Received: 201. Request to send data
because this check is too strict. Happy to make a PR to to at enable passing additional headers if that helps
k

Kevin Su

08/10/2023, 4:33 PM
Thank you @Victor Churikov. good to know you’ve fixed it.
v

Victor Churikov

08/10/2023, 4:51 PM
I was mentioned by accident, forwarding @Victor Delépine
g

Gopal Vashishtha

08/16/2023, 4:03 PM
@Victor Delépine I just ran into this issue and your fix worked for me! Thanks. Looks like all you need to do is sign your commits to get them to master?
v

Victor Delépine

08/16/2023, 4:04 PM
I'm glad it helped! thanks for the reminder, I've justed signed it off now
16 Views
Powered by