Thread
#announcements
    Matheus Moreno

    Matheus Moreno

    4 months ago
    Hey, everyone. Did the in-container path for secrets change for 1.x.x? I can't access my secrets anymore, and I can't find the directory
    /etc/flyte/secrets
    either
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    that has not been touched
    Matheus Moreno

    Matheus Moreno

    4 months ago
    this is weird. I can't even see those internal environment variables,
    FLYTE_SECRET_ENV_VAR_PREFIX
    and such. were they removed?
    I create a Secret with:
    docker exec flyte-sandbox k3s kubectl create secret generic gcloud-credentials --namespace flyte-example-development --from-file=credentials.json=/root/scv.json
    And try to retrieve it with:
    @task(secret_requests=[Secret(group='gcloud-credentials', key='credentials.json', mount_requirement=Secret.MountType.FILE)])
    def test_secrets() -> None:
        print(current_context().secrets.get('gcloud-credentials', 'credentials.json'))
    But this appears:
    Message:
    
        Unable to find secret for key credentials.json in group gcloud-credentials in Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON and FilePath: /etc/secrets/gcloud-credentials/credentials.json
    
    User error.
    On 0.x.x, secrets were being mounted on /etc/flyte/secrets; I was actually retrieving them manually because of how Flyte deals with upper/lower casing for them. But now there's no
    /etc/flyte/secrets
    directory, nor
    /etc/secrets
    , even though the secrets are being requested (I can see on the task definition) and they do exist on the defined namespace (I can see with
    kubectl get secrets -n flyte-example-development
    ).
    and the default SecretsConfig is created from env vars
    did you add this as an env var?
    cc @Yee do you know whats happening here?
    Matheus Moreno

    Matheus Moreno

    4 months ago
    no, I configured as a mounted file
    bc of the bug involving env vars and fast registration
    I'm avoiding the injection with env vars
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    @Matheus Moreno can you lauched pod spec
    and I think you should try and dump the secret
    we should be able to see this in pod spec to see if it even mounted it
    from the code pov, it all looks fine
    that being said, could be a bug right. sorry for the trouble, but help will be appreciated
    Matheus Moreno

    Matheus Moreno

    4 months ago
    yeah 1 second
    {
      "config": {},
      "id": {
        "resourceType": 1,
        "project": "flyte-example",
        "domain": "development",
        "name": "tasks.serving.test_secrets",
        "version": "c1826ab9d9ef49bdbeb7f1c9a7920bfe-uuid4"
      },
      "type": "python-task",
      "metadata": {
        "runtime": {
          "type": 1,
          "version": "1.0.1",
          "flavor": "python"
        },
        "retries": {}
      },
      "interface": {
        "inputs": {
          "variables": {}
        },
        "outputs": {
          "variables": {}
        }
      },
      "securityContext": {
        "secrets": [
          {
            "group": "gcloud-credentials",
            "key": "credentials.json",
            "mountRequirement": 2
          }
        ],
        "tokens": []
      },
      "container": {
        "command": [],
        "args": [
          "pyflyte-execute",
          "--inputs",
          "{{.input}}",
          "--output-prefix",
          "{{.outputPrefix}}",
          "--raw-output-data-prefix",
          "{{.rawOutputDataPrefix}}",
          "--checkpoint-path",
          "{{.checkpointOutputPrefix}}",
          "--prev-checkpoint",
          "{{.prevCheckpointPrefix}}",
          "--resolver",
          "flytekit.core.python_auto_container.default_task_resolver",
          "--",
          "task-module",
          "tasks.serving",
          "task-name",
          "test_secrets"
        ],
        "env": [],
        "config": [],
        "ports": [],
        "image": "us-docker.pkg.dev/bi-data-science/dsc-images/flyte-flyte-example:c1826ab",
        "resources": {
          "requests": [],
          "limits": []
        }
      }
    }
    this is the flyte config
    this is the kubectl describe for the pod
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    @Matheus Moreno is the sandbox?
    if so, then can you tell me
    kubectl get pods -n flyte
    or is this demo?
    cc @Haytham Abuelfutuh
    Matheus Moreno

    Matheus Moreno

    4 months ago
    its the demo
    the image is flyte-sandbox-lite
    Yee

    Yee

    4 months ago
    can you dump the yaml for the pod as well please?
    Matheus Moreno

    Matheus Moreno

    4 months ago
    tried mounting as env var and also no luck
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    @Yee I think the webhook is not coming up
    We can look at the docker logs
    But let me try this too
    Matheus Moreno

    Matheus Moreno

    4 months ago
    any luck?
    k

    katrina

    4 months ago
    hey @Matheus Moreno can you try
    flytectl sandbox start
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    my guess is that
    webhook
    is not starting up in the
    demo
    cluster for you for some reason
    we will figure that out
    but you should be able to use sandbox
    it is slower but each component is a separate deployment and easier to monitor use
    kubectl
    in this you will have to start
    demo
    and then say
    docker logs
    we will try to reproduce
    Yee

    Yee

    4 months ago
    yeah i can reproduce the same error on my demo
    ketan how do i get logs from ‘propeller’?
    i don’t quite understand the error message
    Env Var:_FSEC_GCLOUD-CREDENTIALS_CREDENTIALS.JSON
    what is that env var used for?
    cuz environment variables are not supposed to have
    .
    and
    -
    in them, so some sanitizing needs to happen
    Matheus Moreno

    Matheus Moreno

    4 months ago
    I think this involves the SecretManager. I actually posted about this some other day, about how it upper/lower cases the keys depending on the mount type
    let me see if I can find my post
    Yee

    Yee

    4 months ago
    can confirm that that’s not the issue at this point though. it doesn’t work if you just do normal characters.
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    @Yee lets talk
    .
    Yee

    Yee

    4 months ago
    hey @Matheus Moreno - sorry but could you try using the sandbox for now? we’ll take a deeper look and try to fix this. (so just
    flytectl demo teardown
    and then
    flytectl sandbox start
    - which does a very similar thing but all the flyte components run inside docker containers on the virtual cluster instead of one binary so it takes longer to load)
    Ketan (kumare3)

    Ketan (kumare3)

    4 months ago
    @Matheus Moreno
    flytectl sandbox start
    still works.
    Demo
    is a new type of image that we are building which is faster to start and use and also is almost airgapped (almost). It seems for some reason the Webhook is unable to communicate with K8s api in demo
    Matheus Moreno

    Matheus Moreno

    4 months ago
    Ok, I'll keep using the sandbox for now. Thank you everyone!