https://flyte.org
Join Slack
hey team is there a way to pass aws creds to pod using k8s secrets?
# flyte-support
b
hey team is there a way to pass aws creds to pod using k8s secrets?
b
hi @brash-piano-42461 have you checked out the secrets guide? By default Flyte uses k8s secrets: as long as you create secrets like so and use the 
flytekit.Secret
configuration it should work
b
@broad-monitor-993
this is how i created secrets
Copy code
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=your-access-key-id \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret-access-key
f
but why not use service accounts?
b
i am testing in local sandbox cluster
and im not too aware about service account
b
your secret request should look something like:
Copy code
@task(secret_requests=[
    Secret(
        group="aws-credentials",
        key="AWS_ACCESS_KEY_ID",
    ),
    Secret(
        group="aws-credentials",
        key="AWS_SECRET_ACCESS_KEY",
    ),
])
b
oh i need to pass it in task
b
yep! check out the secrets guide
b
ya i saw that wondering there was any other way?
b
not that I know of
b
hey @broad-monitor-993 still getting same error
Copy code
Unable to locate credentials
Copy code
@task(task_config=ray_config,
    requests=Resources(mem="2Gi", cpu="1"),
    container_image=custom_image,
    secret_requests=[
    Secret(
        group="aws-credentials",
        key="AWS_ACCESS_KEY_ID",
    ),
    Secret(
        group="aws-credentials",
        key="AWS_SECRET_ACCESS_KEY",
    ),
])
b
ah, you’ll need to inject it into your env vars in the task function body:
Copy code
@task(...)
def my_task(...):
    os.environ["AWS_ACCESS_KEY_ID"] = secret_manager.get("aws-credentials", "AWS_ACCESS_KEY_ID")
    os.environ["AWS_SECRET_ACCESS_KEY"] = secret_manager.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")
There’s a PR to make this more convenient
b
Copy code
sc = SecretsManager()
    os.environ["AWS_ACCESS_KEY_ID"] = sc.get("aws-credentials", "AWS_ACCESS_KEY_ID")
    os.environ["AWS_SECRET_ACCESS_KEY"] = sc.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")
is this fine @broad-monitor-993?
Copy code
Message:

    Unable to find secret for key AWS_ACCESS_KEY_ID in group aws-credentials in Env Var:_FSEC_AWS-CREDENTIALS_AWS_ACCESS_KEY_ID and FilePath: /etc/secrets/aws-credentials/aws_access_key_id
getting this error
b
Any ideas @glamorous-carpet-83516 @thankful-minister-83577 ?
g
it doesn’t work because flytekit need the credentials before running the task. e.g. download data when pod is running, it starts to 1. download data 2. run the @task. <- you add secret here, but download data will fail first. 3. upload data
#1726 , yes, this can address your issue
b
@brash-piano-42461 can you share the task code exactly? Is the os.environ in the task function body?
b
Yes
Copy code
@task(task_config=ray_config,
    requests=Resources(mem="2Gi", cpu="1"),
    container_image=custom_image)  
def fn():
   sc = SecretsManager()
   os.environ["AWS_ACCESS_KEY_ID"] = sc.get("aws-credentials", "AWS_ACCESS_KEY_ID")
   os.environ["AWS_SECRET_ACCESS_KEY"] = sc.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")
7 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by