https://flyte.org logo
Join Slack
Powered by
Hey! Trying to run a simple job using a non-defaul...
# flyte-on-gcp
a
Hey! Trying to run a simple job using a non-default docker image (containing our dependencies), made after the image in flytekit. When I don't specify an image (and running the default one), the job successfully works -- it manages to write data to GCS to pass it between tasks. When I'm passing a custom image with additional dependencies, I'm getting 401 from GCS without container reaching my code. It feels like there is something wrong/off with the workload identity, but I'm not 100% sure. Any ideas about how to debug it efficiently?
Following the docs https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#verify_the_setup, everything seems to be correct -- the pod gets a correct service account. The custom images don't work with 401 unauthorized though, failing with fetching the 
script_mode.tar.gz
archive
What's interesting is that I can't get access to GCS with the flyteworkers SA: gives me 403 when I'm trying to 
gsutil ls
. I've verified that the WI is configured, https://cloud.google.com/kubernetes-engine/docs/troubleshooting/authentication#pod_cant_authenticate_to outputs are correct
d
hi @Artyom Korkhov, it this custom image stored on Artifact Registry?
a
Yes!
The trace is
Copy code
FlyteAssertion: Failed to get data from 
gs://<BUCKET>/flytesnacks/development/DCN2DB4Y4LNBVAO3Y
FQJQ3KCSU======/script_mode.tar.gz to /root/ (recursive=False).

Original exception: 401, message='Unauthorized', 
url=URL('<https://storage.googleapis.com/download/storage/v1/b/><BUCKET>/o/flytesnacks%2Fdevelopment%2FDCN2DB4Y4LNBVAO3YFQJQ3KCSU======%2
Fscript_mode.tar.gz?alt=media')
(
<BUCKET>
is a replaced value)
My suspicion is that the docker image is packaged incorrectly. I'm trying to iterate locally with 
pyflyte run
and can also see in the line above that the tar.gz is being copied in the 
/root/
folder. My docker image very likely expects different paths. Could this be the problem, but masking differently?
d
it's possible yes. Right now, Flyte will use /root as the WORKDIR. You can specify a different path if you add 
--destination-dir
to your pyflyte command. It should enable Flyte to download the code to whatever location you indicate as WORKDIR. Does that make sense?
a
Yes, thank you! This was indeed the case -- the docker image built was using different paths. Fixed that and got things running. Thank you! 🙂
4 Views
Open in Slack
PreviousNext
Powered by