Bernhard Stadlbauer
04/27/2022, 11:43 AMflytectl==0.5.21
. The workflows were serialized using pyflyte
(from `flytekit=1.0.0`; using fast
). The storage configuration looks like this:
...
storage:
type: stow
stow:
kind: google
config:
project_id: "<project>"
scopes: <https://www.googleapis.com/auth/devstorage.read_write>
container: "<bucket>"
Now, when running flytectl register files tmp/workflows/* -p acorn -d development
, we see:
Error: failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID
{"json":{},"level":"error","msg":"failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID","ts":"2022-04-27T13:37:26+02:00"}
GOOGLE_APPLICATION_CREDENTIALS
env is set to a service account which has access to the bucket. Registering against flyteadmin in version 1.0.0Prafulla Mahindrakar
04/27/2022, 12:49 PMPrafulla Mahindrakar
04/27/2022, 12:49 PMPrafulla Mahindrakar
04/27/2022, 12:52 PMPrafulla Mahindrakar
04/27/2022, 12:57 PMBernhard Stadlbauer
04/27/2022, 1:36 PMremoteData
config map to the following:
❯ kubectl -n flyte describe configmap flyte-admin-config | grep remoteData -A 7
remoteData.yaml:
----
remoteData:
region: us-west-1
scheme: gcs
signedUrls:
durationMinutes: 3
enabled: true
signingPrincipal: gsa-flyteadmin@<my-project>.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>
But am still seeing the same error. I've checked and the config.yaml also get's injected into the flyteadmin pod. Am I missing something here?Prafulla Mahindrakar
04/27/2022, 1:37 PMPrafulla Mahindrakar
04/27/2022, 1:43 PMGoogleAccessID
Bernhard Stadlbauer
04/27/2022, 1:47 PM❯ kubectl -n flyte logs pods/flyteadmin-7b9c8cd58c-2g6ft
time="2022-04-27T13:35:20Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/namespace_config.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
Interestingly, nothing shows up here (I have re-run the register step before getting the logs). Should/can I increase the log level?
Yes, the error is still the same:
{
"json": {},
"level": "error",
"msg": "failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID",
"ts": "2022-04-27T15:46:11+02:00"
}
Bernhard Stadlbauer
04/27/2022, 1:49 PMflytectl get projects
works as expectedPrafulla Mahindrakar
04/27/2022, 3:20 PMPrafulla Mahindrakar
04/27/2022, 4:26 PMPrafulla Mahindrakar
04/27/2022, 4:28 PMPrafulla Mahindrakar
04/27/2022, 4:29 PMHaytham Abuelfutuh
storage: missing required GoogleAccessID
is pointing to an issue with the storage config in admin side..
After digging through that, it’s caused by an outdated Google Cloud library… apologies about this. Here is a fix: https://github.com/flyteorg/stow/pull/5Prafulla Mahindrakar
04/28/2022, 5:03 AMBernhard Stadlbauer
04/28/2022, 12:17 PMPrafulla Mahindrakar
04/28/2022, 12:53 PMBernhard Stadlbauer
04/28/2022, 1:07 PMPrafulla Mahindrakar
05/02/2022, 4:09 PM