https://flyte.org
Join Slack
Hey, everyone. Did the in-container path for secrets change for 1.x.x? I can't access my secrets any...
# announcements
s
Hey, everyone. Did the in-container path for secrets change for 1.x.x? I can't access my secrets anymore, and I can't find the directory 
/etc/flyte/secrets
either
f
that has not been touched
s
this is weird. I can't even see those internal environment variables, 
FLYTE_SECRET_ENV_VAR_PREFIX
and such. were they removed?
I create a Secret with:
Copy code
docker exec flyte-sandbox k3s kubectl create secret generic gcloud-credentials --namespace flyte-example-development --from-file=credentials.json=/root/scv.json
And try to retrieve it with:
Copy code
@task(secret_requests=[Secret(group='gcloud-credentials', key='credentials.json', mount_requirement=Secret.MountType.FILE)])
def test_secrets() -> None:
    print(current_context().secrets.get('gcloud-credentials', 'credentials.json'))
But this appears:
Copy code
Message:

    Unable to find secret for key credentials.json in group gcloud-credentials in Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON and FilePath: /etc/secrets/gcloud-credentials/credentials.json

User error.
On 0.x.x, secrets were being mounted on `/etc/flyte/secrets`; I was actually retrieving them manually because of how Flyte deals with upper/lower casing for them. But now there's no 
/etc/flyte/secrets
directory, nor 
/etc/secrets
, even though the secrets are being requested (I can see on the task definition) and they do exist on the defined namespace (I can see with 
kubectl get secrets -n flyte-example-development
).
f
and the default SecretsConfig is created from env vars
did you add this as an env var?
cc @thankful-minister-83577 do you know whats happening here?
s
no, I configured as a mounted file
bc of the bug involving env vars and fast registration
I'm avoiding the injection with env vars
f
@sparse-window-1536 can you lauched pod spec
and I think you should try and dump the secret
we should be able to see this in pod spec to see if it even mounted it
from the code pov, it all looks fine
that being said, could be a bug right. sorry for the trouble, but help will be appreciated
s
yeah 1 second
Copy code
{
  "config": {},
  "id": {
    "resourceType": 1,
    "project": "flyte-example",
    "domain": "development",
    "name": "tasks.serving.test_secrets",
    "version": "c1826ab9d9ef49bdbeb7f1c9a7920bfe-uuid4"
  },
  "type": "python-task",
  "metadata": {
    "runtime": {
      "type": 1,
      "version": "1.0.1",
      "flavor": "python"
    },
    "retries": {}
  },
  "interface": {
    "inputs": {
      "variables": {}
    },
    "outputs": {
      "variables": {}
    }
  },
  "securityContext": {
    "secrets": [
      {
        "group": "gcloud-credentials",
        "key": "credentials.json",
        "mountRequirement": 2
      }
    ],
    "tokens": []
  },
  "container": {
    "command": [],
    "args": [
      "pyflyte-execute",
      "--inputs",
      "{{.input}}",
      "--output-prefix",
      "{{.outputPrefix}}",
      "--raw-output-data-prefix",
      "{{.rawOutputDataPrefix}}",
      "--checkpoint-path",
      "{{.checkpointOutputPrefix}}",
      "--prev-checkpoint",
      "{{.prevCheckpointPrefix}}",
      "--resolver",
      "flytekit.core.python_auto_container.default_task_resolver",
      "--",
      "task-module",
      "tasks.serving",
      "task-name",
      "test_secrets"
    ],
    "env": [],
    "config": [],
    "ports": [],
    "image": "us-docker.pkg.dev/bi-data-science/dsc-images/flyte-flyte-example:c1826ab",
    "resources": {
      "requests": [],
      "limits": []
    }
  }
}
this is the flyte config
this is the kubectl describe for the pod
kubectl_describe.txt
f
@sparse-window-1536 is the sandbox?
if so, then can you tell me 
kubectl get pods -n flyte
or is this demo?
cc @high-park-82026
s
its the demo
the image is flyte-sandbox-lite
t
can you dump the yaml for the pod as well please?
s
pod.yaml
pod.yaml
tried mounting as env var and also no luck
f
@thankful-minister-83577 I think the webhook is not coming up
We can look at the docker logs
But let me try this too
s
any luck?
a
hey @sparse-window-1536 can you try 
flytectl sandbox start
f
my guess is that 
webhook
is not starting up in the 
demo
cluster for you for some reason
we will figure that out
but you should be able to use sandbox
it is slower but each component is a separate deployment and easier to monitor use 
kubectl
in this you will have to start 
demo
and then say 
docker logs
we will try to reproduce
t
yeah i can reproduce the same error on my demo
ketan how do i get logs from ‘propeller’?
i don’t quite understand the error message
Copy code
Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON
what is that env var used for?
cuz environment variables are not supposed to have 
.
and 
-
in them, so some sanitizing needs to happen
s
I think this involves the SecretManager. I actually posted about this some other day, about how it upper/lower cases the keys depending on the mount type
let me see if I can find my post
t
can confirm that that’s not the issue at this point though. it doesn’t work if you just do normal characters.
f
@thankful-minister-83577 lets talk
.
t
hey @sparse-window-1536 - sorry but could you try using the sandbox for now? we’ll take a deeper look and try to fix this. (so just 
flytectl demo teardown
and then 
flytectl sandbox start
- which does a very similar thing but all the flyte components run inside docker containers on the virtual cluster instead of one binary so it takes longer to load)
f
@sparse-window-1536 
flytectl sandbox start
still works. 
Demo
is a new type of image that we are building which is faster to start and use and also is almost airgapped (almost). It seems for some reason the Webhook is unable to communicate with K8s api in demo
s
Ok, I'll keep using the sandbox for now. Thank you everyone!
168 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by