https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
h

Hud

10/20/2023, 1:26 PM
Hi team, I typically have credentials stored in a 
.env
file, a separate 
config.py
that loads the credentials, and I import that config module into a file. However when running remotely that config module is not found. What is the preferred way to load this?
y

Yee

10/20/2023, 1:28 PM
create secrets from the credentials?
h

Hud

10/20/2023, 1:43 PM
kubectl create secret -n <project>-<domain> --from-env-file=.env
Something like this? @Yee
it says project domain has to be the same as namespace - what should be project and domain for sandbox?
y

Yee

10/20/2023, 1:49 PM
the project and domain where the task will run
so something like flytesnacks development
h

Hud

10/20/2023, 1:50 PM
so its arbitrary in some way? Like myproject-development should be okay?
Ok I see the namespaces via 
kubectl get namespace
Not sure which namespace is the correct one for flyte sandbox, I tried to create the secrets in all of them and ran
Copy code
secret = Secret(
    group="user-info",
    key="user_secret",
    mount_requirement=Secret.MountType.ENV_VAR,
)
SECRET_GROUP = "user-info"
SECRET_NAME = "user_secret"


@task(secret_requests=[Secret(group=SECRET_GROUP, key=SECRET_NAME)])
def secret_task() -> str:
    context = current_context()
    secret_val = context.secrets.get(SECRET_GROUP, SECRET_NAME)
    print(secret_val)
    return secret_val
but got the error when I run `pyflyte run`:
Copy code
Failed with Unknown Exception <class 'ValueError'> Reason: Error encountered while executing 'secret_task':
  Unable to find secret for key user_secret in group user-info in Env Var:_FSEC_USER-INFO_GH_PAT and FilePath: /etc/secrets/user-info/user_secret
Error encountered while executing 'secret_task':
  Unable to find secret for key user_secret in group user-info in Env Var:_FSEC_USER-INFO_GH_PAT and FilePath: /etc/secrets/user-info/user_secret
whereby the 
user_secret
is the key I defined in my 
.env
file e.g. 
user_secret=<my-secret-key>
@Yee
y

Yee

10/21/2023, 1:41 AM
the .env file is not used at all
can you copy paste the output of kubectl get secret in the namespace of your execution?
h

Hud

10/21/2023, 2:21 AM
Copy code
❯ kubectl get secret
NAME        TYPE     DATA   AGE
user-info   Opaque   3      5h55m
@Yee so this only worked when I created it in the 
default
namespace
y

Yee

10/21/2023, 2:21 AM
you need to create the secret in the namespace of the execution.
h

Hud

10/21/2023, 2:23 AM
yeah that's the part I am struggling with - is there a way to determine the namespace?
y

Yee

10/21/2023, 2:23 AM
it’s just project-domain
flytesnack-development, flytesnacks-staging, etc
h

Hud

10/21/2023, 2:25 AM
Copy code
❯ kubectl get namespace
NAME                      STATUS   AGE
default                   Active   29d
kube-system               Active   29d
kube-public               Active   29d
kube-node-lease           Active   29d
flyte                     Active   29d
flytesnacks-development   Active   29d
flytesnacks-staging       Active   29d
flytesnacks-production    Active   29d
So I have created the secret in all of the namespaces - and I still get the error
y

Yee

10/21/2023, 2:25 AM
mmm
can you try something for me please?
h

Hud

10/21/2023, 2:26 AM
sure
y

Yee

10/21/2023, 2:26 AM
can you recreate everything without 
-
and 
_
not the namespaces, those always have a `-`… just the secrets
the secret group and name
and you just need to create the secret in the ns of the execution
h

Hud

10/21/2023, 2:28 AM
ok I can try - I cant seem to 
kubectl delete secret user-info
in other namespaces beside 
default
is there a way to delete those?
y

Yee

10/21/2023, 2:28 AM
nah it’s fine
no need to delete
that’s weird but that’s a different thing… you must not have perms to delete in your cluster.
if this doesn’t work then dump out the logs for the flyte pod webhook
h

Hud

10/21/2023, 2:34 AM
yeah seems it doesnt work
Copy code
Error encountered while executing 'secret_task':
  Unable to find secret for key usersecret in group userinfo in Env Var:_FSEC_USERINFO_USERSECRET and FilePath: /etc/secrets/userinfo/usersecret
@Yee how do I dump the logs for the flyte pod webhook ?
y

Yee

10/21/2023, 2:36 AM
kubectl -n flyte logs webhook pod name
and k get secret one more time?
h

Hud

10/21/2023, 2:38 AM
Copy code
❯ kubectl get secret
NAME       TYPE     DATA   AGE
userinfo   Opaque   1      5m28s
y

Yee

10/21/2023, 2:38 AM
and 
k get userinfo -o yaml
h

Hud

10/21/2023, 2:39 AM
oh that's odd:
Copy code
❯ kubectl get userinfo -o yaml
error: the server doesn't have a resource type "userinfo"
y

Yee

10/21/2023, 2:39 AM
get secret
k…
and the logs?
i assume that’s a test secret right not a real secret
h

Hud

10/21/2023, 2:43 AM
Not sure what's the correct webhook pod name ..
y

Yee

10/21/2023, 2:49 AM
k -n flyte get pod
i find these convenient to have btw
Copy code
alias k="kubectl"
alias kf="kubectl -n flyte"
alias ksd='k -n flytesnacks-development'
alias kga="kubectl get --all-namespaces"
h

Hud

10/21/2023, 2:50 AM
Copy code
flyte-sandbox-postgresql-0                            1/1     Running                  1 (2d9h ago)    11d
flyte-sandbox-proxy-d95874857-4lzjw                   1/1     Running                  1 (2d9h ago)    11d
flyte-sandbox-buildkit-7d7d55dbb-4fdlm                1/1     Running                  1 (2d9h ago)    11d
flyte-sandbox-minio-645c8ddf7c-h84qp                  1/1     Running                  10 (2d9h ago)   29d
flyte-sandbox-docker-registry-759844bc88-ctmng        1/1     Running                  0               11d
flyte-sandbox-7d699df5fc-8qx5r                        1/1     Running                  1 (2d9h ago)    11d
flyte-sandbox-kubernetes-dashboard-6757db879c-zv7b6   1/1     Running                  20 (36h ago)    11d
one of those ?
y

Yee

10/21/2023, 2:50 AM
flyte-sandbox-7d699df5fc-8qx5r
should be running inside single binary
but you’ll need to do some digging
h

Hud

10/21/2023, 2:53 AM
oof quite a large dump - what am I looking for specifically ?
y

Yee

10/21/2023, 2:54 AM
i dunno
was hoping it would have more logging
also maybe change your task to also print os.environ
want to see if it’s being set as something else.
h

Hud

10/21/2023, 3:03 AM
@Yee so print(os.environ) basically shows all of the env vars I have, but not usersecret
Copy code
@task
def secret_task():
    sc = SecretsManager()
    print(sc.get(SECRET_GROUP, SECRET_NAME))
also gave the same error
y

Yee

10/21/2023, 3:03 AM
anything in the logs
?
h

Hud

10/21/2023, 3:10 AM
@Yee hm cant seem to find anything in the logs with secret, env, fsec, inject, .. not sure what else to look for this instant
y

Yee

10/21/2023, 3:12 AM
search for the exec id
d

David Espejo (he/him)

10/30/2023, 3:15 PM
I was able to repro this issue. Basically it happens if the execution is local (instead of using 
pyflyte run --remote ...
) and the env vars described here are not set. Maybe the docs need better guidance on what to configure depending on remote/local execution so secrets discovery works.
y

Yee

10/31/2023, 1:27 AM
ooh nice. thank you @David Espejo (he/him)
2 Views
Powered by