Hi all having an issue injecting secrets from k8s into my ta

Question

Hi all having an issue injecting secrets from k8s into my task Here s how I ve set it up I created a test secret like this to try and see if a task can read it ```kubectl n flytesnacks development create secret generic common secrets from literal=TEST SECRET=blah kubectl n flytesnacks development get secret common secrets o json | jq data | to entries | map value= value | base64d gt gt key TEST SECRET value blah ``` Within the flytesnacks development namespace I m running a task like this ``` task secret requests= Secret group= common secrets key= TEST SECRET def print secret gt str secrets = current context secrets return secrets get common secrets TEST SECRET ``` However this fails with the following ```Unable to find secret for key TEST SECRET in group common secrets in Env Var FSEC COMMON SECRETS TEST SECRET and FilePath root secrets common secrets test secret``` From looking at the code in the the SecretManager it looks like it only checks the ENV variable or a file path which does not exist because I m using k8s secrets as an additional provider Am I missing something I ve checked that the k8s secret is in the same namespace as the task being run

Ketan (kumare3) · Answer

Hmm the secret should be Injected by the Flyte secrets injector

Eric Hsiao · Answer

Do you know how I can check when how that runs It says in the docs that it should be injected to the pod when the task starts but that doesn t seem to be the case

Ketan (kumare3) · Answer

Have you looked at this <https docs flyte org projects cookbook en stable auto core containerization use secrets html|https docs flyte org projects cookbook en stable auto core containerization use secrets html>

Eric Hsiao · Answer

Yes that s the document I m using I checked the webhook pod and things seem to be configured correctly ``` kubectl logs pod flyte pod webhook 595f7b6858 7qxdt n flyte time= 2022 08 12T00 31 32Z level=info msg= time= 2022 08 12T00 31 32Z level=info msg= App flytepropeller Version unknown BuildSHA unknown BuildTS 2022 08 12 00 31 32 456181799 +0000 UTC m=+0 033941479 time= 2022 08 12T00 31 32Z level=info msg= time= 2022 08 12T00 31 32Z level=info msg= Detected 4 CPU s n metrics prefix flyte certDir etc webhook certs localCert false listenPort 9443 serviceName flyte pod webhook servicePort 443 secretName flyte pod webhook secretManagerType K8s awsSecretManager sidecarImage <http docker io amazon aws secrets manager secret sidecar v0 1 4|docker io amazon aws secrets manager secret sidecar v0 1 4> resources limits cpu 200m memory 500Mi requests cpu 200m memory 500Mi vaultSecretManager role flyte kvVersion 2 ```

Ketan (kumare3) · Answer

Hmm interesting so let s try to look at one of the launched pods cc < Yee> you recently saw this What annotation should we look for

Eric Hsiao · Answer

Here s the output for describing a task pod that starts up ``` Documents kb describe pod a2wf2sqs2bfw6qr4l92d n0 0 n flytesnacks development Name a2wf2sqs2bfw6qr4l92d n0 0 Namespace flytesnacks development Priority 0 Node ip 10 15 147 196 ec2 internal 10 15 147 196 Start Time Mon 15 Aug 2022 12 52 24 0400 Labels domain=development execution id=a2wf2sqs2bfw6qr4l92d inject flyte secrets=true interruptible=false node id=n0 project=flytesnacks shard key=13 task name=flyte workflows hello world print secret workflow name=flyte workflows hello world wf Annotations false flyte secrets s0 m4zg54lqhiqcey2pnvww52rnonswg3tforzsectlmv3tuibclfavercjl4jukq1sivkcecq eks privileged Status Succeeded IP 10 15 128 66 IPs IP 10 15 128 66 Controlled By flyteworkflow a2wf2sqs2bfw6qr4l92d Containers a2wf2sqs2bfw6qr4l92d n0 0 Container ID 5 Image XXX Image ID XXX Port lt none gt Host Port lt none gt Args pyflyte execute inputs output prefix raw output data prefix checkpoint path prev checkpoint resolver flytekit core python auto container default task resolver task module flyte workflows hello world task name print secret State Terminated Reason Completed Exit Code 0 Started Mon 15 Aug 2022 12 52 25 0400 Finished Mon 15 Aug 2022 12 52 28 0400 Ready False Restart Count 0 Limits cpu 1 memory 1000Mi Requests cpu 1 memory 1000Mi Environment FLYTE INTERNAL EXECUTION WORKFLOW flytesnacks development FLYTE INTERNAL EXECUTION ID a2wf2sqs2bfw6qr4l92d FLYTE INTERNAL EXECUTION PROJECT flytesnacks FLYTE INTERNAL EXECUTION DOMAIN development FLYTE ATTEMPT NUMBER 0 FLYTE INTERNAL TASK PROJECT flytesnacks FLYTE INTERNAL TASK DOMAIN development FLYTE INTERNAL TASK NAME flyte workflows hello world print secret FLYTE INTERNAL TASK VERSION v0 0 5 FLYTE INTERNAL PROJECT flytesnacks FLYTE INTERNAL DOMAIN development FLYTE INTERNAL NAME flyte workflows hello world print secret FLYTE INTERNAL VERSION v0 0 5 AWS STS REGIONAL ENDPOINTS regional AWS DEFAULT REGION us east 1 AWS REGION us east 1 AWS ROLE ARN XXX AWS WEB IDENTITY TOKEN FILE var run secrets eks amazonaws com serviceaccount token Mounts var run secrets eks amazonaws com serviceaccount from aws iam token ro var run secrets kubernetes io serviceaccount from kube api access 7kr64 ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes aws iam token Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 86400 kube api access 7kr64 Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class Guaranteed Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s```

Ketan (kumare3) · Answer

this is weird it has the right secret annotation `flyte secrets s0 m4zg54lqhiqcey2pnvww52rnonswg3tforzsectlmv3tuibclfavercjl4jukq1sivkcecq`

Ketan (kumare3) · Answer

cc < Kevin Su> < Yee> does anyone of you know the problem here is ` ` a problem

Yee · Answer

i m still debugging this

Yee · Answer

i think it s just the caps

Yee · Answer

but now this has been going forever <https playground hosted unionai cloud console projects flytesnacks domains development executions f25523dab45564925bdf duration=all>

Yee · Answer

and not really sure why

Yee · Answer

failure to mount volume this part is a bug

Yee · Answer

if you specify a secret that doesn t exist it shouldn t hang

Ketan (kumare3) · Answer

ohh so it cannot be caps

Yee · Answer

``` Type Reason Age From Message Warning FailedMount 6m29s x28 over 106m kubelet Unable to attach or mount volumes unmounted volumes= mnxw12lpnywxgzldojsxi316 unattached volumes= kube api access pbzq2 mnxw12lpnywxgzldojsxi316 aws iam token timed out waiting for the condition Warning FailedMount 2m13s x60 over 108m kubelet MountVolume SetUp failed for volume mnxw12lpnywxgzldojsxi316 references non existent secret key test secret```

Yee · Answer

i think <https github com flyteorg flytekit blob 994a8bffdc00e6c76546bf7b4b37f92108ca90ff flytekit core context manager py L363|this> shouldn t be lower

Eric Hsiao · Answer

oh should I try it out with a lower case secret

Eric Hsiao · Answer

I added a lower case version of the secret ``` symphony hall git main kubectl n flytesnacks development get secret common secrets o json | jq data | to entries | map value= value | base64d key TEST SECRET value blah key test secret value some value ``` Code change ``` task secret requests= Secret group= common secrets key= test secret def print secret gt str secrets = current context secrets return secrets get common secrets test secret ``` but still got the same error ```Unable to find secret for key test secret in group common secrets in Env Var FSEC COMMON SECRETS TEST SECRET and FilePath root secrets common secrets test secret``` I wonder if the ` ` in `common secrets` is causing issues I also changed the secrets default directory in the Dockerfile Could that cause problems I know the default is ` etc secrets` ```ENV FLYTE SECRETS DEFAULT DIR root secrets```

Yee · Answer

```$ alias ksd ksd= kubectl n flytesnacks development $ ksd create secret generic common secrets from literal=test secret=sosecret secret common secrets created``` with code assuming the flytesnacks repo in cookbook ```$ cat core secret example py from flytekit import Secret task workflow current context task secret requests= Secret group= common secrets key= test secret def print secret secrets = current context secrets s = secrets get common secrets test secret print s workflow def my print print secret ``` ```$ ksd logs f0cc52c90e3ae47a8876 n0 0 tar Removing leading ` from member names sosecret```

Yee · Answer

can you try again

Yee · Answer

now i m not able to repro

Yee · Answer

keep in mind when you change you unf have to change in two places in the secret name in the task decorator and again in the get call itself

Yee · Answer

let me know if you re still having issues and we can hop on a screenshare tomorrow

Yee · Answer

we will also fix the casing bug

Yee · Answer

< Dan Rammer hamersaw > can you take a look at this tomorrow

Eric Hsiao · Answer

< Yee> I took your code and packaged it up but it unfortunately still didn t find the secret very odd

Eric Hsiao · Answer

I ssh ed into the task running container and tried to see if the secret was mounted but it doesn t look like it at least from the envs and file paths it s looking at

Eric Hsiao · Answer

```root alxtm7pdc9zchkqttvtp n0 0 ~ printenv KUBERNETES SERVICE PORT HTTPS=443 FLYTE INTERNAL EXECUTION DOMAIN=development PYTHON VERSION=3 9 13 FLYTE INTERNAL TASK PROJECT=flytesnacks FLYTE INTERNAL VERSION=v0 0 12 FLYTE INTERNAL PROJECT=flytesnacks FLYTE INTERNAL TASK NAME=flyte workflows hello world print secret FLYTE SECRETS DEFAULT DIR= root secrets FLYTE INTERNAL TASK DOMAIN=development FLYTE INTERNAL EXECUTION PROJECT=flytesnacks```

Yee · Answer

can you `ls lRa root secrets`

Yee · Answer

btw are you on gcp

Eric Hsiao · Answer

I m on aws

Eric Hsiao · Answer

` root secrets` didn t get created since the secrets weren t moved over

Yee · Answer

what do you mean moved over could you dump that pod spec

Yee · Answer

actually can you hop on screenshare

Eric Hsiao · Answer

yep

Dan Rammer (hamersaw) · Answer

Hey < Eric Hsiao> quick update here I think I m able to reproduce this locally can you provide some versioning information specifically flytepropeller and k8s

Eric Hsiao · Answer

yes I m in a meeting right now but will send those over in ~15 minutes

Eric Hsiao · Answer

^ < Chris Antenesse>

Chris Antenesse · Answer

flytepropeller is v1 1 0

Chris Antenesse · Answer

k8s version is v1 22

Ketan (kumare3) · Answer

< Dan Rammer hamersaw > is this because of the all CAPS Secret name `TEST SECRET`

Chris Antenesse · Answer

< Eric Hsiao> i added a new secret called `test secret` case issue should be quick to verify on our end

Ketan (kumare3) · Answer

can you please try that

Ketan (kumare3) · Answer

i think this is something to do with case sensitivity

Ketan (kumare3) · Answer

we should be fixing to support both cc < Yee> did some investigations i feel

Eric Hsiao · Answer

I verified that before but that seems like a separate issue

Dan Rammer (hamersaw) · Answer

< Yee> and I discussed the case sensitivity issues earlier today We have a PR out which should fix that I want to make sure it s not a bigger issue with the webhook deployment Can you change the mutating webhook configuration to fail rather than ignore failures

Dan Rammer (hamersaw) · Answer

this can be done by setting `failurePolicy Fail`

Eric Hsiao · Answer

Yep will try that

Dan Rammer (hamersaw) · Answer

This causes FlytePropeller to fail in creating the Pod if the k8s API server is unable to call the mutating webhook which handles secret injection In my test case in the UI the task error was ```Workflow flytesnacks development flytegen core containerization use secrets secret task failed RuntimeExecutionError max number of system retry attempts 11 10 exhausted Last known status message failed at Node corecontainerizationusesecretssecrettask RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container InternalError failed to create resource caused by Internal error occurred failed calling webhook Post service flyte pod webhook not found``` but i just removed the webhook service this should tell us if the webhook is active

Yee · Answer

can we add that to the default deployment < Dan Rammer hamersaw >

Dan Rammer (hamersaw) · Answer

^^ was just thinking the exact same thing

Yee · Answer

though i suspect it s not failing i feel like it s not getting called at all

Chris Antenesse · Answer

where do we change the `failurePolicy` i thought the mutating webhook config was at the service level but not seeing it there

Dan Rammer (hamersaw) · Answer

oh sorry it has it s own resource `kubectl n flyte edit mutatingwebhookconfigurations flyte pod webhook o yaml`

Chris Antenesse · Answer

ok got it

Chris Antenesse · Answer

that s set on our end

Eric Hsiao · Answer

I ll rerun the test flyte workflow

Dan Rammer (hamersaw) · Answer

already set to Fail then like Yee suggested the webhook is not even being called so there is a mismatch between the webhook criteria and the pod definitions

Chris Antenesse · Answer

my bad i set it to `Fail` just now

Chris Antenesse · Answer

it was set to `Ignore` previously slightly smiling face

Dan Rammer (hamersaw) · Answer

yeah we ll have to update this in the default configuration

Eric Hsiao · Answer

I m seeing the task I m kicking off hanging now ~3mins

Dan Rammer (hamersaw) · Answer

ok i m guessing there are webhook call errors and flytepropeller is internally retrying there is a 10s timeout on the mutating webhook be default i think so we ll have to incur that few times maybe

Dan Rammer (hamersaw) · Answer

it should produce an end error that will hopefully pray be a bit more descriptive on what is happening

Dan Rammer (hamersaw) · Answer

is there anything in the propeller logs like `failed calling webhook` on my local testing above the propeller logs are littered with ``` json exec id agbb24545n6vbwknrxqs node corecontainerizationusesecretssecrettask ns flytesnacks development res ver 5504 routine worker 4 src handler go 222 wf flytesnacks development flytegen core containerization use secrets secret task level error msg handling parent node failed with error failed at Node corecontainerizationusesecretssecrettask RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container InternalError failed to create resource caused by Internal error occurred failed calling webhook Post service flyte pod webhook not found ts 2022 08 17T15 45 24 05 00 ``` for each individual retry

Chris Antenesse · Answer

yea

Chris Antenesse · Answer

one sec will grab one

Chris Antenesse · Answer

``` json exec id ancq4kjld5lf6cv88bd8 ns flytesnacks development res ver 10680755 routine worker 8 wf flytesnacks development level error msg Error when trying to reconcile workflow Error failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container InternalError failed to create resource caused by Internal error occurred failed calling webhook failed to call webhook Post context deadline exceeded Error Type errors NodeErrorWithCause ts 2022 08 17T21 22 16Z E0817 21 22 16 167997 1 workers go 102 error syncing flytesnacks development ancq4kjld5lf6cv88bd8 failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container InternalError failed to create resource caused by Internal error occurred failed calling webhook failed to call webhook Post context deadline exceed```

Dan Rammer (hamersaw) · Answer

OK so it looks like it s being called but there is some issue with k8s communication internally I m doing some searching and see <https github com actions runner controller actions runner controller issues 68|potential firewall issues> are common

Eric Hsiao · Answer

This part is a little suspicious`failed calling webhook `<http flyte pod webhook flyte org|flyte pod webhook flyte org>``

Dan Rammer (hamersaw) · Answer

It sounds like the API server is unable to connect to the webhook endpoint but i m not sure the easiest way to debug this

Eric Hsiao · Answer

Should that be pointed at <http flyte org|flyte org>

Dan Rammer (hamersaw) · Answer

so i think that s just the webhook name from the k8s resource ```apiVersion kind MutatingWebhookConfiguration metadata ommitted webhooks omitted name ```

Dan Rammer (hamersaw) · Answer

basically in that configuration the api server will try to call the service defined at ``` service name flyte pod webhook namespace all path mutate v1 pod port 9443```

Chris Antenesse · Answer

so `<https flyte pod webhook flyte svc 443 mutate v1 pod>`

Chris Antenesse · Answer

in our case

Chris Antenesse · Answer

i can exec to the propeller pod and access that hostname port ```kubectl exec n flyte it flytepropeller 74bf956f6c 5zfbh bin sh nc z flyte pod webhook flyte svc 443```

Dan Rammer (hamersaw) · Answer

yeah i am kind of running into a blank here we know that the failure is in calling the webhook it s important to make the distinction that the request to the webhook is not coming from propeller but from the kube api server i believe and propeller just sees the error when trying to create a Pod

Chris Antenesse · Answer

ah ok that makes sense

Dan Rammer (hamersaw) · Answer

can we take a quick look at the propeller config map should be something like ```apiVersion v1 kind ConfigMap metadata name flyte propeller config namespace flyte``` and the pod webhook service ```apiVersion v1 kind Service metadata name flyte pod webhook namespace flyte```

Chris Antenesse · Answer

sure

Chris Antenesse · Answer

the configmap ```kind ConfigMap metadata annotations flyte flyte creationTimestamp 2022 08 12T00 31 18Z labels flyte Helm flyteadmin flyte core v1 1 0 name flyte propeller config namespace flyte```

Chris Antenesse · Answer

and pod webhook service ```apiVersion v1 kind Service metadata annotations flyte flyte grpc creationTimestamp 2022 08 12T00 31 18Z labels Helm name flyte pod webhook namespace flyte```

Dan Rammer (hamersaw) · Answer

can you paste the data section on the webhook as well so the full definition

Chris Antenesse · Answer

oh yes

Dan Rammer (hamersaw) · Answer

i mean the service that you just pasted

Dan Rammer (hamersaw) · Answer

i ve finally lost it

Chris Antenesse · Answer

here s the output of `kubectl get svc flyte pod webhook o yaml n flyt` ```apiVersion v1 kind Service metadata annotations flyte flyte grpc creationTimestamp 2022 08 12T00 31 18Z labels Helm name flyte pod webhook namespace flyte resourceVersion 8525789 uid 0ffd7a94 14d1 44db ae76 34d17dccb0cd spec clusterIP 172 27 91 248 clusterIPs 172 27 91 248 internalTrafficPolicy Cluster ipFamilies IPv4 ipFamilyPolicy SingleStack ports name https port 443 protocol TCP targetPort 9443 selector app flyte pod webhook sessionAffinity None type ClusterIP status loadBalancer ```

Chris Antenesse · Answer

do you want output of the configmap too

Dan Rammer (hamersaw) · Answer

umm we might be alright

Dan Rammer (hamersaw) · Answer

do i see the service port 9443 being mapped to 443 in the pod do we just have the wrong port in the MutatingWebhookConfiguration can you try to update it to 9443 rather than 443 ``` service name flyte pod webhook namespace all path mutate v1 pod port 9443``` `mutatingwebhookconfigurations flyte pod webhook`

Chris Antenesse · Answer

sure

Chris Antenesse · Answer

from ``` service name flyte pod webhook namespace flyte path mutate v1 pod port 443``` to ``` service name flyte pod webhook namespace flyte path mutate v1 pod port 9443```

Chris Antenesse · Answer

that is complete on my end

Dan Rammer (hamersaw) · Answer

yeah if this doesn t work i m going to have to defer to our in house k8s expert the only issue is he is on the other side of the world so his hours can be a bit difficult to overlap

Chris Antenesse · Answer

got it

Chris Antenesse · Answer

should we try and run the task again and show some logs

Eric Hsiao · Answer

yeah I m kicking it off and getting this ``` json exec id aknhc8ds5wn5z7sql6jb ns flytesnacks development res ver 10692263 routine worker 5 wf flytesnacks development level error msg Error when trying to reconcile workflow Error failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container InternalError failed to create resource caused by Internal error occurred failed calling webhook failed to call webhook Post no service port 9443 found for service flyte pod webhook Error Type errors NodeErrorWithCause ts 2022 08 17T22 07 16Z E0817 22 07 16 516247 1 workers go 102 error syncing flytesnacks development aknhc8ds5wn5z7sql6jb failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container InternalError failed to create resource caused by Internal error occurred failed calling webhook failed to call webhook Post no service port 9443 found for service flyte pod webhook json exec id ancq4kjld5lf6cv88bd8 ns flytesnacks development routine worker 4 level warning msg Workflow not found in cache ts 2022 08 17T22 08 06Z json exec id ancq4kjld5lf6cv88bd8 ns flytesnacks development routine worker 4 level warning msg Workflow namespace flytesnacks development name ancq4kjld5lf6cv88bd8 not found may be deleted ts 2022 08 17T22 08 06Z ```

Eric Hsiao · Answer

no service port 9443 found for service

Dan Rammer (hamersaw) · Answer

oh sure that s interesting because that s how our configuration is setup using 9443 as the target port in service and then mutating configuration endpoint

Dan Rammer (hamersaw) · Answer

< Yuvraj> any chance you are seeing something here I m sure you ll take one look at it will be obvious

Chris Antenesse · Answer

we were able to solve this on our end the k8s API was unable to connect to the webhook pod after a security group change allowing the traffic we were able to inject secrets

Yee · Answer

oh all good

Yee · Answer

everything is resolved would you mind describing how you were able to debug

Chris Antenesse · Answer

yep things are working as intended on our end at least for that piece slightly smiling face

Yee · Answer

like how did you determine that did you look at cloudwatch logs what testing did you do sorry if some of this was covered in the messages i missed above

Chris Antenesse · Answer

sure i think there are a few important points being able to have the more descriptive logs for flytepropller was a big help however this line tripped us up a bit it seemed like there was a connectivity issue from flytepropeller to the webhook pod when in fact it was an issue from the k8s API nodes to the webhook pod ```Internal error occurred failed calling webhook failed to call webhook Post context deadline exceed``` we were able to determine that it was an issue from the k8s API gt webhook pod by looking at this diagram

Chris Antenesse · Answer

one we knew that the k8s API nodes were trying to talk to the webhook pods on 443 it was a matter of finding the security group that handled that access and adding a rule to allow that traffic once i did that i kicked off the workflow and things worked as intended

Chris Antenesse · Answer

there is one more hump unrelated to this that we need to get over which has to do with private registry access i ll start a new conversation about that in < CP2HDHKE1|onboarding> but we re good on secrets injection for now thanks for the help lmk if there is any other information that would be helpful on our side

Dan Rammer (hamersaw) · Answer

< Chris Antenesse> < Eric Hsiao> this is great to hear glad we were able to get this resolved we re going to update the configuration failure policy to fail by default and that should bubble up these issues into the console quicker in the future once we know the webhook is unable to be called it should ease debugging thanks for being so patient with this fix

Dan Rammer (hamersaw) · Answer

Also I know we have some sparse documentation on <https docs flyte org projects cookbook en latest auto deployment deploying workflows html working hosted service|private image registries> we can explore it a bit further in the new thread you re planning on starting it would be great to fill this out a bit more with any questions you have

Chris Antenesse · Answer

nice i ve been through the docs and hitting some speed bumps will surface those today