https://flyte.org logo
Join Slack
Powered by
Hi there, I'm deploying Flyte to my K8s cluster us...
# flyte-v1-support
h
Hi there, I'm deploying Flyte to my K8s cluster using `flyte-binary`; the service is happily running but every time I try to start a "hello world" workflow using 
pyflyte run
the execution fails with this error:
Copy code
flytekit.exceptions.system.FlyteDownloadDataException: SYSTEM:DownloadDataError: error=Failed to get data from s3://<the bucket>/flytesnacks/development/K3QHP5EWEVPQRJJUI4CPMYMZQU======/fast5078eeda28546327dc735d6cad471fa3.tar.gz to ./ (recursive=False).

Original exception: Unable to locate credentials
My understanding is that the task pod does not have the credentials to access the metadata bucket. How is it supposed to get them in the first place? Since they are supplied via the Helm chart, I figured they would be automatically provisioned in the namespace where the task is running (
flytesnacks-development
in that case). Note that I had a similar issue with container images as I'm using a private registry. I was able to fix it by installing the credentials in a secret in the execution namespace and editing the serviceaccount the task pod is run under to such that it can make use of these credentials. In that case though, the format for the credentials and the way they are consumed is standard Docker access in K8s; however in the bucket case I have no idea how to supply the credentials to the execution pod - for example, in which format does it expect them? Thanks for your help!
c
I know in 
flyte-core
you need to configure the credentials in a few places... I'm not sure about 
flyte-binary
but I can look
Hmm. in 
flyte-binary
it looks like there is only one place to configure them
You should be able to inspect the pod to see what credentials are being passed through
h
my pod is not started using the 
flyte-copilot
binary but `pyflyte-fast-execute`:
Copy code
- args:
    - pyflyte-fast-execute
    - --additional-distribution
    - s3://<the bucket>/flytesnacks/development/K3QHP5EWEVPQRJJUI4CPMYMZQU======/fast5078eeda28546327dc735d6cad471fa3.tar.gz
    - --dest-dir
    - .
    - --
    - pyflyte-execute
    - --inputs
    - s3://<the bucket>/metadata/propeller/flytesnacks-development-ahfjrff9f7dv8bqbvddc/n0/data/inputs.pb
    - --output-prefix
    - s3://<the bucket>/metadata/propeller/flytesnacks-development-ahfjrff9f7dv8bqbvddc/n0/data/0
    - --raw-output-data-prefix
    - s3://<the bucket>/data/kg/ahfjrff9f7dv8bqbvddc-n0-0
    - --checkpoint-path
    - s3://<the bucket>/data/kg/ahfjrff9f7dv8bqbvddc-n0-0/_flytecheckpoints
    - --prev-checkpoint
    - '""'
    - --resolver
    - flytekit.core.python_auto_container.default_task_resolver
    - --
    - task-module
    - src.hello_world
    - task-name
    - say_hello
It also gets a few env variables starting with 
FLYTE_INTERNAL_
but none are about storage credentials.
c
I see, yeah the copilot might be for containers tasks. Let me look into it a bit more
🙏 1
we do something similar with pod templates which inject the credentials into every one of our pods but I thought that might have been related to our fork
I'm kinda surprised this isn't documented anywhere hmm
h
yeah that looks like the exact same problem
c
I suppose you could also run pods with a service account that has web identity and access to S3
h
regardless of the solution it will require some secret in the namespace where the pod is run, and that namespace depends on the project/environment combo that the user picked
I'm not quite sure how I can ensure that the requisite secrets are always present in these namespaces given that they are created dynamically
a
@hundreds-iron-56210 so if you describe the execution Pod, what envVars do you see?
h
Copy code
FLYTE_INTERNAL_EXECUTION_WORKFLOW
FLYTE_INTERNAL_EXECUTION_ID
FLYTE_INTERNAL_EXECUTION_PROJECT
FLYTE_INTERNAL_EXECUTION_DOMAIN
FLYTE_ATTEMPT_NUMBER
FLYTE_INTERNAL_TASK_PROJECT
FLYTE_INTERNAL_TASK_DOMAIN
FLYTE_INTERNAL_TASK_NAME
FLYTE_INTERNAL_TASK_VERSION
FLYTE_INTERNAL_PROJECT
FLYTE_INTERNAL_DOMAIN
FLYTE_INTERNAL_NAME
FLYTE_INTERNAL_VERSION
c
Yeah those are all pretty standard
h
@average-finland-92144 @clean-glass-36808 what would you recommend I try next?
c
My company uses a per namespace pod templates that includes references to a namespace/project specific set of blob storage credentials.
h
I'm not even at the point where I have to deploy project-specific credentials: right now it's the credentials to the object storage that Flyte uses to (IIUC) stash registration data that are not propagated to the execution pod (whereas the service pod is able to access the object storage).
c
I understand. Did you try the solution in https://github.com/flyteorg/flyte/issues/6415? Obviously not ideal since passwords are stored directly in the config but it would allow you to test
👍 1
h
hey, I ended up giving it a try and it worked
now I'm going to have to find a way to make this all work with secret references
c
We use the namespace pod template for that so the secrets aren’t leaked through the pod spec
👍 1
c
Hi @hundreds-iron-56210 I'm curious how you ended up solving your issue. I ran into the same problem and @clean-glass-36808 kindly shared this message. Specifically I'd like to avoid sharing the blob storage credentials in the execution namespace which is accessible by our end users. Ideally I would have a per project/domain blob credentials but I don't know to set this up Thanks for your help!
h
Sorry, I did not find a solution for this problem, I moved on from it.
🥲 1
22 Views
Open in Slack
PreviousNext
Powered by