https://flyte.org logo
b

Bernhard Stadlbauer

04/27/2022, 11:43 AM
Hi! We are currently trying to fast-register workflows using
flytectl==0.5.21
. The workflows were serialized using
pyflyte
(from `flytekit=1.0.0`; using
fast
). The storage configuration looks like this:
Copy code
... 
storage:
  type: stow
  stow:
    kind: google
    config:
      project_id: "<project>"
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
  container: "<bucket>"
Now, when running
flytectl register files tmp/workflows/* -p acorn -d development
, we see:
Copy code
Error: failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID
{"json":{},"level":"error","msg":"failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID","ts":"2022-04-27T13:37:26+02:00"}
GOOGLE_APPLICATION_CREDENTIALS
env is set to a service account which has access to the bucket. Registering against flyteadmin in version 1.0.0
p

Prafulla Mahindrakar

04/27/2022, 12:49 PM
Hi @Bernhard Stadlbauer, with the latest changes to flytectl fast register , it uses signed url feature which require no storage level configuration on client side and instead the client is provided a signed url to upload the artifacts. You would need add your service account in flyteadmin remote storage config Eg default config here https://github.com/flyteorg/flyteadmin/blob/9d9194ccd99972e56c467defcff05fc459976a56/pkg/runtime/application_config_provider.go#L50 You will need to add SigningPrincipal to the value of your service account In yaml values file https://github.com/flyteorg/flyte/blob/0d7d93368032b2cc8b8916bfd4e53de9615cf6b6/charts/flyte/values.yaml#L559
cc : @Haytham Abuelfutuh
You can still use older flytectl to avoid using this feature anything flytectl<=0.5.8
Update to <=0.5.8
b

Bernhard Stadlbauer

04/27/2022, 1:36 PM
@Prafulla Mahindrakar Thank you for the quick reply! Not having to have local credentials seems really nice! I have changed my
remoteData
config map to the following:
Copy code
❯ kubectl -n flyte describe configmap flyte-admin-config | grep remoteData -A 7
remoteData.yaml:
----
remoteData:
  region: us-west-1
  scheme: gcs
  signedUrls:
    durationMinutes: 3
    enabled: true
    signingPrincipal: gsa-flyteadmin@<my-project>.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>
But am still seeing the same error. I've checked and the config.yaml also get's injected into the flyteadmin pod. Am I missing something here?
p

Prafulla Mahindrakar

04/27/2022, 1:37 PM
Can you provide the logs from admin .
Does it still complain about the
GoogleAccessID
b

Bernhard Stadlbauer

04/27/2022, 1:47 PM
These are the logs:
Copy code
❯ kubectl -n flyte logs pods/flyteadmin-7b9c8cd58c-2g6ft
time="2022-04-27T13:35:20Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/namespace_config.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
Interestingly, nothing shows up here (I have re-run the register step before getting the logs). Should/can I increase the log level? Yes, the error is still the same:
Copy code
{
  "json": {},
  "level": "error",
  "msg": "failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID",
  "ts": "2022-04-27T15:46:11+02:00"
}
Getting projects with
flytectl get projects
works as expected
p

Prafulla Mahindrakar

04/27/2022, 3:20 PM
Also @Bernhard Stadlbauer yes increasing the log level shoudl help here and strange that configmap didn’t take effect . Did you rollout and restart flyteadmin
Hi @Bernhard Stadlbauer my bad , i think remoteDataConfig is deprecated one and the new one is https://github.com/flyteorg/flytestdlib/blob/6185cd0d84cc1e0eaf3af1ed7ef42c144d2f6165/storage/config.go#L64
Let me check this on my end first.
Downgrading flytectl should help to unblock you though.
h

Haytham Abuelfutuh

04/28/2022, 12:22 AM
This
storage: missing required GoogleAccessID
is pointing to an issue with the storage config in admin side.. After digging through that, it’s caused by an outdated Google Cloud library… apologies about this. Here is a fix: https://github.com/flyteorg/stow/pull/5
p

Prafulla Mahindrakar

04/28/2022, 5:03 AM
Thanks Haytham for fixing this . We should probably remove the deprecated fields as they erroneously lead to me to this code.
b

Bernhard Stadlbauer

04/28/2022, 12:17 PM
Perfect, thank you! I have downgraded to <= 5.8.0 right now 👍 Which one would be the right config option in the future, do I need the SignedURLConfig or the SigningPrincipal?
p

Prafulla Mahindrakar

04/28/2022, 12:53 PM
Hi @Bernhard Stadlbauer you won’t need any config change for this issue .If you get the latest admin which has the fix then you can use the newest flytectl version which removes the dependency on storage config on client side .
b

Bernhard Stadlbauer

04/28/2022, 1:07 PM
Perfect, thank you! I'll wait for the latest admin then 👍
p

Prafulla Mahindrakar

05/02/2022, 4:09 PM
The following version of admin https://github.com/flyteorg/flyteadmin/releases/tag/v1.0.1 contains the fix. For GCP additional perms need to be added for the flyteadmin role https://github.com/flyteorg/flyte/pull/2435/files You would need something similar for the cloud provider that you use for your flyte deployment
12 Views