https://flyte.org
Join Slack
Does anyone have experience with using AWS Secrets Manager with flyte instead of the default k8 secr...
# flyte-support
n
Does anyone have experience with using AWS Secrets Manager with flyte instead of the default k8 secrets? a little confused on how to set up the secret names.
e
If you're using AWS Secrets Manager, a common pattern is using the external secrets operator to expose those as K8s 
Secret
resources inside the cluster. https://external-secrets.io/v0.8.1/
Also IMO this is a desirable pattern since your K8s workloads aren't aware of the cloud provider as an implementation detail
n
e
Oh - I wasn't aware that Flyte had a native secrets manager integration; that could also work, but we use External Secrets in a number of other places in our system & that's how we're mounting secrets in our workflows today
n
Well, I'm not sure how to use the eative flyte AWS one haha so not sure how well supported it is. yea, I'm able to mount some of my secrets directly as env vars in my pod templates through terraform accessing the AWS Secrets without needing to use External Secrets but wanted to try getting the native secret manager working
t
what did you mean @numerous-hamburger-7178?
the 
:
?
n
that's one of the questions
t
did you have a stack trace we could look at?
what do you have it set to currently?
n
lets say I have
Copy code
secret_group = test_group
secret_key = test_username
secret_string = test_password
what is the expected input supposed to be to Secret Manger? 
When using the AWS secret management plugin, secrets need to be specified by naming them in the format <SECRET_GROUP>:<SECRET_KEY>, where the secret string is a plain-text value, *not* key/value json.
t
can you just remove the :?
and make it 
my_secret_name
?
n
then where do I put the secret_group, secret_key args?
t
so basically it doesn’t matter really, but when you concatenate the two together, you need to have the arn.
the downside of this is that your aws account number ends up in the group name.
this is the current structure at least - if you have ideas on ux improvements, please let us know
or just push a pr 🙂 - i know there were requests for domain specific secrets (which happens automatically with k8s secrets but not with the other secret providers)
n
but the ARN can't have 
:
in it right? bc AWS won't let me put a 
:
in the name of the secret? if the output of 
formatAWSSecretArn
has to match with the actual secret's ARN
I'm not really blocked by this because I can just export my aws secrets as k8s secrets using terraform which then can get picked up natively by flyte
t
oh no, you don’t need to put in the : on the aws side
that happens automatically. if you create the secret and then look at the arn
it’ll come with a bunch of `:`s
n
so should the secret plaintext be 
{secret_key:secret_value}
and the name of the secret be the secret_group?
t
oh no, the secret value can be anything.
n
Copy code
When using the AWS secret management plugin, secrets need to be specified by naming them in the format <SECRET_GROUP>:<SECRET_KEY>
so where do I put secret_group and secret_key then? so that flyte can find it properly
oooh, you're saying that it happens automatically. flyte will mount all the secrets that exist and generate a group name that is something like 
account_id:secret_name in aws
and I will have to reference it by that
t
so if i make a secret named 
mysecretname
and I just put in it “hello world”
the arn will be something like
Copy code
arn:aws:secretsmanager:us-east-2:1233456:secret:mysecretname-RJCgGN
n
yep I'm following, and then how will I reference that in
Copy code
@task(
    secret_requests=[
        Secret(key=XXX, group=XXX),
    ]
)
t
so then in the user code side, you can do something like
Copy code
Secret(group="arn:aws:secretsmanager:us-east-2:1233456:secret", key="mysecretname-RJCgGN")
n
hmm... that probably won't work for me then. was hoping that it'll be more generic than that. but appreciate the clarification. That would be a good example to throw on the secrets doc haha.
t
yeah… we’re working on ways to improve the experience.
if you have suggestions for ux improvements?
n
I think in my case, I need to deploy the same workflows in diff deployments so they'll reference diff secrets values but ideally be under the same name.
t
oh you can do that… you can put in a json document i’m pretty sure. let me make an end to end working example
n
but the group name and key will be different in diff deployments? e.g region and account id will be diff and the hash
t
what do you mean by deployment?
i think that only changes when you rotate the secret.
n
e.g. if I have separate deployments in diff AWS accounts
t
oh then you’ll have different secrets entirely so yes
👍 1
171 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by