Hi! We are currently trying to fast-register workflows using

flytectl==0.5.21

. The workflows were serialized using

pyflyte

(from `flytekit=1.0.0`; using

fast

). The storage configuration looks like this:

... 
storage:
  type: stow
  stow:
    kind: google
    config:
      project_id: "<project>"
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
  container: "<bucket>"

Now, when running

flytectl register files tmp/workflows/* -p acorn -d development

, we see:

Error: failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID
{"json":{},"level":"error","msg":"failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID","ts":"2022-04-27T13:37:26+02:00"}

GOOGLE_APPLICATION_CREDENTIALS

env is set to a service account which has access to the bucket. Registering against flyteadmin in version 1.0.0

Hi @agreeable-kitchen-44189, with the latest changes to flytectl fast register , it uses signed url feature which require no storage level configuration on client side and instead the client is provided a signed url to upload the artifacts. You would need add your service account in flyteadmin remote storage config Eg default config here https://github.com/flyteorg/flyteadmin/blob/9d9194ccd99972e56c467defcff05fc459976a56/pkg/runtime/application_config_provider.go#L50 You will need to add SigningPrincipal to the value of your service account In yaml values file https://github.com/flyteorg/flyte/blob/0d7d93368032b2cc8b8916bfd4e53de9615cf6b6/charts/flyte/values.yaml#L559

@icy-agent-73298 Thank you for the quick reply! Not having to have local credentials seems really nice! I have changed my

remoteData

config map to the following:

❯ kubectl -n flyte describe configmap flyte-admin-config | grep remoteData -A 7
remoteData.yaml:
----
remoteData:
  region: us-west-1
  scheme: gcs
  signedUrls:
    durationMinutes: 3
    enabled: true
    signingPrincipal: gsa-flyteadmin@<my-project>.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>

But am still seeing the same error. I've checked and the config.yaml also get's injected into the flyteadmin pod. Am I missing something here?

Can you provide the logs from admin .

These are the logs:

❯ kubectl -n flyte logs pods/flyteadmin-7b9c8cd58c-2g6ft
time="2022-04-27T13:35:20Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/namespace_config.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"

Interestingly, nothing shows up here (I have re-run the register step before getting the logs). Should/can I increase the log level? Yes, the error is still the same:

{
  "json": {},
  "level": "error",
  "msg": "failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID",
  "ts": "2022-04-27T15:46:11+02:00"
}

Also @agreeable-kitchen-44189 yes increasing the log level shoudl help here and strange that configmap didn’t take effect . Did you rollout and restart flyteadmin

This

storage: missing required GoogleAccessID

is pointing to an issue with the storage config in admin side.. After digging through that, it’s caused by an outdated Google Cloud library… apologies about this. Here is a fix: https://github.com/flyteorg/stow/pull/5

Thanks Haytham for fixing this . We should probably remove the deprecated fields as they erroneously lead to me to this code.

Perfect, thank you! I have downgraded to <= 5.8.0 right now 👍 Which one would be the right config option in the future, do I need the SignedURLConfig or the SigningPrincipal?

Hi @agreeable-kitchen-44189 you won’t need any config change for this issue .If you get the latest admin which has the fix then you can use the newest flytectl version which removes the dependency on storage config on client side .

Perfect, thank you! I'll wait for the latest admin then 👍

The following version of admin https://github.com/flyteorg/flyteadmin/releases/tag/v1.0.1 contains the fix. For GCP additional perms need to be added for the flyteadmin role https://github.com/flyteorg/flyte/pull/2435/files You would need something similar for the cloud provider that you use for your flyte deployment