Hey, everyone. Did the in-container path for secre...
# announcements
Hey, everyone. Did the in-container path for secrets change for 1.x.x? I can't access my secrets anymore, and I can't find the directory
that has not been touched
this is weird. I can't even see those internal environment variables,
and such. were they removed?
I create a Secret with:
Copy code
docker exec flyte-sandbox k3s kubectl create secret generic gcloud-credentials --namespace flyte-example-development --from-file=credentials.json=/root/scv.json
And try to retrieve it with:
Copy code
@task(secret_requests=[Secret(group='gcloud-credentials', key='credentials.json', mount_requirement=Secret.MountType.FILE)])
def test_secrets() -> None:
    print(current_context().secrets.get('gcloud-credentials', 'credentials.json'))
But this appears:
Copy code

    Unable to find secret for key credentials.json in group gcloud-credentials in Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON and FilePath: /etc/secrets/gcloud-credentials/credentials.json

User error.
On 0.x.x, secrets were being mounted on `/etc/flyte/secrets`; I was actually retrieving them manually because of how Flyte deals with upper/lower casing for them. But now there's no
directory, nor
, even though the secrets are being requested (I can see on the task definition) and they do exist on the defined namespace (I can see with
kubectl get secrets -n flyte-example-development
and the default SecretsConfig is created from env vars
did you add this as an env var?
cc @Yee do you know whats happening here?
no, I configured as a mounted file
bc of the bug involving env vars and fast registration
I'm avoiding the injection with env vars
@Matheus Moreno can you lauched pod spec
and I think you should try and dump the secret
we should be able to see this in pod spec to see if it even mounted it
from the code pov, it all looks fine
that being said, could be a bug right. sorry for the trouble, but help will be appreciated
yeah 1 second
Copy code
  "config": {},
  "id": {
    "resourceType": 1,
    "project": "flyte-example",
    "domain": "development",
    "name": "tasks.serving.test_secrets",
    "version": "c1826ab9d9ef49bdbeb7f1c9a7920bfe-uuid4"
  "type": "python-task",
  "metadata": {
    "runtime": {
      "type": 1,
      "version": "1.0.1",
      "flavor": "python"
    "retries": {}
  "interface": {
    "inputs": {
      "variables": {}
    "outputs": {
      "variables": {}
  "securityContext": {
    "secrets": [
        "group": "gcloud-credentials",
        "key": "credentials.json",
        "mountRequirement": 2
    "tokens": []
  "container": {
    "command": [],
    "args": [
    "env": [],
    "config": [],
    "ports": [],
    "image": "us-docker.pkg.dev/bi-data-science/dsc-images/flyte-flyte-example:c1826ab",
    "resources": {
      "requests": [],
      "limits": []
this is the flyte config
this is the kubectl describe for the pod
@Matheus Moreno is the sandbox?
if so, then can you tell me
kubectl get pods -n flyte
or is this demo?
cc @Haytham Abuelfutuh
its the demo
the image is flyte-sandbox-lite
can you dump the yaml for the pod as well please?
tried mounting as env var and also no luck
@Yee I think the webhook is not coming up
We can look at the docker logs
But let me try this too
any luck?
hey @Matheus Moreno can you try
flytectl sandbox start
my guess is that
is not starting up in the
cluster for you for some reason
we will figure that out
but you should be able to use sandbox
it is slower but each component is a separate deployment and easier to monitor use
in this you will have to start
and then say
docker logs
we will try to reproduce
yeah i can reproduce the same error on my demo
ketan how do i get logs from ‘propeller’?
i don’t quite understand the error message
Copy code
what is that env var used for?
cuz environment variables are not supposed to have
in them, so some sanitizing needs to happen
I think this involves the SecretManager. I actually posted about this some other day, about how it upper/lower cases the keys depending on the mount type
let me see if I can find my post
can confirm that that’s not the issue at this point though. it doesn’t work if you just do normal characters.
@Yee lets talk
hey @Matheus Moreno - sorry but could you try using the sandbox for now? we’ll take a deeper look and try to fix this. (so just
flytectl demo teardown
and then
flytectl sandbox start
- which does a very similar thing but all the flyte components run inside docker containers on the virtual cluster instead of one binary so it takes longer to load)
@Matheus Moreno
flytectl sandbox start
still works.
is a new type of image that we are building which is faster to start and use and also is almost airgapped (almost). It seems for some reason the Webhook is unable to communicate with K8s api in demo
Ok, I'll keep using the sandbox for now. Thank you everyone!