https://flyte.org logo
Powered by
#announcements
Title
# announcements
m

Matheus Moreno

05/11/2022, 4:10 PM
Hey, everyone. Did the in-container path for secrets change for 1.x.x? I can't access my secrets anymore, and I can't find the directory 
/etc/flyte/secrets
either
k

Ketan (kumare3)

05/11/2022, 4:11 PM
that has not been touched
m

Matheus Moreno

05/11/2022, 4:24 PM
this is weird. I can't even see those internal environment variables, 
FLYTE_SECRET_ENV_VAR_PREFIX
and such. were they removed?
I create a Secret with:
Copy code
docker exec flyte-sandbox k3s kubectl create secret generic gcloud-credentials --namespace flyte-example-development --from-file=credentials.json=/root/scv.json
And try to retrieve it with:
Copy code
@task(secret_requests=[Secret(group='gcloud-credentials', key='credentials.json', mount_requirement=Secret.MountType.FILE)])
def test_secrets() -> None:
    print(current_context().secrets.get('gcloud-credentials', 'credentials.json'))
But this appears:
Copy code
Message:

    Unable to find secret for key credentials.json in group gcloud-credentials in Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON and FilePath: /etc/secrets/gcloud-credentials/credentials.json

User error.
On 0.x.x, secrets were being mounted on `/etc/flyte/secrets`; I was actually retrieving them manually because of how Flyte deals with upper/lower casing for them. But now there's no 
/etc/flyte/secrets
directory, nor 
/etc/secrets
, even though the secrets are being requested (I can see on the task definition) and they do exist on the defined namespace (I can see with 
kubectl get secrets -n flyte-example-development
).
k

Ketan (kumare3)

05/11/2022, 6:03 PM
and the default SecretsConfig is created from env vars
did you add this as an env var?
cc @Yee do you know whats happening here?
m

Matheus Moreno

05/11/2022, 6:05 PM
no, I configured as a mounted file
bc of the bug involving env vars and fast registration
I'm avoiding the injection with env vars
k

Ketan (kumare3)

05/11/2022, 6:09 PM
@Matheus Moreno can you lauched pod spec
and I think you should try and dump the secret
we should be able to see this in pod spec to see if it even mounted it
from the code pov, it all looks fine
that being said, could be a bug right. sorry for the trouble, but help will be appreciated
m

Matheus Moreno

05/11/2022, 6:15 PM
yeah 1 second
Copy code
{
  "config": {},
  "id": {
    "resourceType": 1,
    "project": "flyte-example",
    "domain": "development",
    "name": "tasks.serving.test_secrets",
    "version": "c1826ab9d9ef49bdbeb7f1c9a7920bfe-uuid4"
  },
  "type": "python-task",
  "metadata": {
    "runtime": {
      "type": 1,
      "version": "1.0.1",
      "flavor": "python"
    },
    "retries": {}
  },
  "interface": {
    "inputs": {
      "variables": {}
    },
    "outputs": {
      "variables": {}
    }
  },
  "securityContext": {
    "secrets": [
      {
        "group": "gcloud-credentials",
        "key": "credentials.json",
        "mountRequirement": 2
      }
    ],
    "tokens": []
  },
  "container": {
    "command": [],
    "args": [
      "pyflyte-execute",
      "--inputs",
      "{{.input}}",
      "--output-prefix",
      "{{.outputPrefix}}",
      "--raw-output-data-prefix",
      "{{.rawOutputDataPrefix}}",
      "--checkpoint-path",
      "{{.checkpointOutputPrefix}}",
      "--prev-checkpoint",
      "{{.prevCheckpointPrefix}}",
      "--resolver",
      "flytekit.core.python_auto_container.default_task_resolver",
      "--",
      "task-module",
      "tasks.serving",
      "task-name",
      "test_secrets"
    ],
    "env": [],
    "config": [],
    "ports": [],
    "image": "us-docker.pkg.dev/bi-data-science/dsc-images/flyte-flyte-example:c1826ab",
    "resources": {
      "requests": [],
      "limits": []
    }
  }
}
this is the flyte config
this is the kubectl describe for the pod
kubectl_describe.txt
k

Ketan (kumare3)

05/11/2022, 7:01 PM
@Matheus Moreno is the sandbox?
if so, then can you tell me 
kubectl get pods -n flyte
or is this demo?
cc @Haytham Abuelfutuh
m

Matheus Moreno

05/11/2022, 7:11 PM
its the demo
the image is flyte-sandbox-lite
y

Yee

05/11/2022, 7:22 PM
can you dump the yaml for the pod as well please?
m

Matheus Moreno

05/11/2022, 7:24 PM
tried mounting as env var and also no luck
k

Ketan (kumare3)

05/11/2022, 7:33 PM
@Yee I think the webhook is not coming up
We can look at the docker logs
But let me try this too
m

Matheus Moreno

05/11/2022, 8:50 PM
any luck?
k

katrina

05/11/2022, 9:57 PM
hey @Matheus Moreno can you try 
flytectl sandbox start
k

Ketan (kumare3)

05/11/2022, 9:57 PM
my guess is that 
webhook
is not starting up in the 
demo
cluster for you for some reason
we will figure that out
but you should be able to use sandbox
it is slower but each component is a separate deployment and easier to monitor use 
kubectl
in this you will have to start 
demo
and then say 
docker logs
we will try to reproduce
y

Yee

05/11/2022, 10:05 PM
yeah i can reproduce the same error on my demo
ketan how do i get logs from ‘propeller’?
i don’t quite understand the error message
Copy code
Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON
what is that env var used for?
cuz environment variables are not supposed to have 
.
and 
-
in them, so some sanitizing needs to happen
m

Matheus Moreno

05/11/2022, 10:09 PM
I think this involves the SecretManager. I actually posted about this some other day, about how it upper/lower cases the keys depending on the mount type
let me see if I can find my post
y

Yee

05/11/2022, 10:34 PM
can confirm that that’s not the issue at this point though. it doesn’t work if you just do normal characters.
k

Ketan (kumare3)

05/11/2022, 10:40 PM
@Yee lets talk
.
y

Yee

05/11/2022, 11:21 PM
hey @Matheus Moreno - sorry but could you try using the sandbox for now? we’ll take a deeper look and try to fix this. (so just 
flytectl demo teardown
and then 
flytectl sandbox start
- which does a very similar thing but all the flyte components run inside docker containers on the virtual cluster instead of one binary so it takes longer to load)
k

Ketan (kumare3)

05/11/2022, 11:58 PM
@Matheus Moreno 
flytectl sandbox start
still works. 
Demo
is a new type of image that we are building which is faster to start and use and also is almost airgapped (almost). It seems for some reason the Webhook is unable to communicate with K8s api in demo
m

Matheus Moreno

05/12/2022, 5:17 PM
Ok, I'll keep using the sandbox for now. Thank you everyone!
7 Views
Powered by