Hi! We are currently trying to fast-register workf...
# announcements
b
Hi! We are currently trying to fast-register workflows using
flytectl==0.5.21
. The workflows were serialized using
pyflyte
(from `flytekit=1.0.0`; using
fast
). The storage configuration looks like this:
Copy code
... 
storage:
  type: stow
  stow:
    kind: google
    config:
      project_id: "<project>"
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
  container: "<bucket>"
Now, when running
flytectl register files tmp/workflows/* -p acorn -d development
, we see:
Copy code
Error: failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID
{"json":{},"level":"error","msg":"failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID","ts":"2022-04-27T13:37:26+02:00"}
GOOGLE_APPLICATION_CREDENTIALS
env is set to a service account which has access to the bucket. Registering against flyteadmin in version 1.0.0
p
Hi @Bernhard Stadlbauer, with the latest changes to flytectl fast register , it uses signed url feature which require no storage level configuration on client side and instead the client is provided a signed url to upload the artifacts. You would need add your service account in flyteadmin remote storage config Eg default config here https://github.com/flyteorg/flyteadmin/blob/9d9194ccd99972e56c467defcff05fc459976a56/pkg/runtime/application_config_provider.go#L50 You will need to add SigningPrincipal to the value of your service account In yaml values file https://github.com/flyteorg/flyte/blob/0d7d93368032b2cc8b8916bfd4e53de9615cf6b6/charts/flyte/values.yaml#L559
cc : @Haytham Abuelfutuh
You can still use older flytectl to avoid using this feature anything flytectl<=0.5.8
Update to <=0.5.8
b
@Prafulla Mahindrakar Thank you for the quick reply! Not having to have local credentials seems really nice! I have changed my
remoteData
config map to the following:
Copy code
❯ kubectl -n flyte describe configmap flyte-admin-config | grep remoteData -A 7
remoteData.yaml:
----
remoteData:
  region: us-west-1
  scheme: gcs
  signedUrls:
    durationMinutes: 3
    enabled: true
    signingPrincipal: gsa-flyteadmin@<my-project>.<http://iam.gserviceaccount.com|iam.gserviceaccount.com>
But am still seeing the same error. I've checked and the config.yaml also get's injected into the flyteadmin pod. Am I missing something here?
p
Can you provide the logs from admin .
Does it still complain about the
GoogleAccessID
b
These are the logs:
Copy code
❯ kubectl -n flyte logs pods/flyteadmin-7b9c8cd58c-2g6ft
time="2022-04-27T13:35:20Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml /etc/flyte/config/namespace_config.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resource_defaults.yaml]"
Interestingly, nothing shows up here (I have re-run the register step before getting the logs). Should/can I increase the log level? Yes, the error is still the same:
Copy code
{
  "json": {},
  "level": "error",
  "msg": "failed to upload source code from [tmp/workflows/fasta438eafa1aa98e76043e131e481a7a23.tar.gz]. Error: failed to create an upload location. Error: rpc error: code = Unknown desc = failed to create a signed url. Error: storage: missing required GoogleAccessID",
  "ts": "2022-04-27T15:46:11+02:00"
}
Getting projects with
flytectl get projects
works as expected
p
Also @Bernhard Stadlbauer yes increasing the log level shoudl help here and strange that configmap didn’t take effect . Did you rollout and restart flyteadmin
Hi @Bernhard Stadlbauer my bad , i think remoteDataConfig is deprecated one and the new one is https://github.com/flyteorg/flytestdlib/blob/6185cd0d84cc1e0eaf3af1ed7ef42c144d2f6165/storage/config.go#L64
Let me check this on my end first.
Downgrading flytectl should help to unblock you though.
h
This
storage: missing required GoogleAccessID
is pointing to an issue with the storage config in admin side.. After digging through that, it’s caused by an outdated Google Cloud library… apologies about this. Here is a fix: https://github.com/flyteorg/stow/pull/5
p
Thanks Haytham for fixing this . We should probably remove the deprecated fields as they erroneously lead to me to this code.
b
Perfect, thank you! I have downgraded to <= 5.8.0 right now 👍 Which one would be the right config option in the future, do I need the SignedURLConfig or the SigningPrincipal?
p
Hi @Bernhard Stadlbauer you won’t need any config change for this issue .If you get the latest admin which has the fix then you can use the newest flytectl version which removes the dependency on storage config on client side .
b
Perfect, thank you! I'll wait for the latest admin then 👍
p
The following version of admin https://github.com/flyteorg/flyteadmin/releases/tag/v1.0.1 contains the fix. For GCP additional perms need to be added for the flyteadmin role https://github.com/flyteorg/flyte/pull/2435/files You would need something similar for the cloud provider that you use for your flyte deployment
118 Views