Hey, did anyone succeed enabling AWS Secrets Manager in

flyte-binary

? I've added this to my values.yaml:

configuration:
  inline:
    webhook:
      secretManagerType: "AWS"

and it seemed to override the proppeler configurtion correctly but no sidecar is being created for any of the tasks. And I get this error on task execution:

Please make sure to add secret_requests=[Secret(group=arn:aws:secretsmanager:***:***:secret:, key=flyte/***)] in @task. Unable to find secret for key flyte/*** in group arn:aws:secretsmanager:***:***:secret: in Env Var:_FSEC_ARN:AWS:SECRETSMANAGER:***:***:SECRET:_FLYTE/*** and FilePath: /etc/secrets/arn:aws:secretsmanager:***:***:secret:/flyte/***

is your flyte-webhook running in the flyte-binary

I've used the default config so it's seems to be enabled, I can see an exposed webhook port and active webhook service. Can I test it somehow?

could you test it by using k8s secret?

I've modified my config to

configuration:
  inline:
    webhook:
      secretManagerType: "K8s"

and run the example from the link. It worked for k8s secret - I was able to retrieve the secret inside the task.

investigating. cc @icy-agent-73298 if you know how to fix it

Can you dump the pod webhook logs and check to see if it tried to add the sidecar. Also does the task pod have right role annotations to load the secrets from aws secrets manager ,

flyte-binary didn't output any logs for the webhook - is there some kind of config value to make it more verbose?

yes, logger level. https://github.com/flyteorg/flyte/blob/568e6860c248eb8d35fb8475893ed4efc130e510/flyte.yaml#L54

Maybe this is it

{"json":{"src":"secrets.go:51"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_FLYTE-FLYTE-BINARY-5BC97CC4CF-R7QPBARN:AWS:SECRETSMANAGER:E***:***:SECRET:_FLYTE/NEPTUNE-ZVYJTX], file [/etc/secrets/flyte-flyte-binary-5bc97cc4cf-r7qpbarn:aws:secretsmanager:***:***:secret:/flyte/neptune-zvyjTX]","ts":"2023-11-22T19:38:23Z"}
{"json":{"src":"secrets.go:45"},"level":"info","msg":"Skipping SecretManager [K8s] since it's not enabled.","ts":"2023-11-22T19:38:23Z"}
{"json":{"src":"aws_secret_manager.go:116"},"level":"error","msg":"unrecognized mount requirement [ENV_VAR] for secret [flyte/neptune-zvyjTX]","ts":"2023-11-22T19:38:23Z"}
{"json":{"src":"secrets.go:51"},"level":"info","msg":"Failed to inject a secret using injector [AWS]. Error: unrecognized mount requirement [ENV_VAR] for secret [flyte/neptune-zvyjTX]","ts":"2023-11-22T19:38:23Z"}
{"json":{"src":"secrets.go:45"},"level":"info","msg":"Skipping SecretManager [GCP] since it's not enabled.","ts":"2023-11-22T19:38:23Z"}
{"json":{"src":"secrets.go:45"},"level":"info","msg":"Skipping SecretManager [Vault] since it's not enabled.","ts":"2023-11-22T19:38:23Z"}
{"json":{"src":"pod.go:122"},"level":"info","msg":"Failed to mutate using [secrets]. Since it's not a required mutator, skipping. Error: unrecognized mount requirement [ENV_VAR] for secret [flyte/neptune-zvyjTX]","ts":"2023-11-22T19:38:23Z"}

i’m not sure there’s a good reason for this. mind putting in a ticket for this?

I had the same question, and I think I understand the answer. Only, where do I specify the AWS credentials?

"AWS_ACCESS_KEY_ID": "some-value"
            "AWS_SECRET_ACCESS_KEY": "some-value",
            "AWS_DEFAULT_REGION": "some-value"

and may I specify different credentials on different project/environment combinations?

can you open a new question plz? i don’t think this is the same.