https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
a

Anirudh Sridhar

09/21/2023, 6:02 PM
hey team is there a way to pass aws creds to pod using k8s secrets?
n

Niels Bantilan

09/21/2023, 6:05 PM
hi @Anirudh Sridhar have you checked out the secrets guide? By default Flyte uses k8s secrets: as long as you create secrets like so and use the 
flytekit.Secret
configuration it should work
a

Anirudh Sridhar

09/21/2023, 6:08 PM
@Niels Bantilan
this is how i created secrets
Copy code
kubectl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=your-access-key-id \
  --from-literal=AWS_SECRET_ACCESS_KEY=your-secret-access-key
k

Ketan (kumare3)

09/21/2023, 6:10 PM
but why not use service accounts?
a

Anirudh Sridhar

09/21/2023, 6:10 PM
i am testing in local sandbox cluster
and im not too aware about service account
n

Niels Bantilan

09/21/2023, 6:17 PM
your secret request should look something like:
Copy code
@task(secret_requests=[
    Secret(
        group="aws-credentials",
        key="AWS_ACCESS_KEY_ID",
    ),
    Secret(
        group="aws-credentials",
        key="AWS_SECRET_ACCESS_KEY",
    ),
])
a

Anirudh Sridhar

09/21/2023, 6:17 PM
oh i need to pass it in task
n

Niels Bantilan

09/21/2023, 6:18 PM
yep! check out the secrets guide
a

Anirudh Sridhar

09/21/2023, 6:18 PM
ya i saw that wondering there was any other way?
n

Niels Bantilan

09/21/2023, 6:20 PM
not that I know of
a

Anirudh Sridhar

09/21/2023, 6:28 PM
hey @Niels Bantilan still getting same error
Copy code
Unable to locate credentials
Copy code
@task(task_config=ray_config,
    requests=Resources(mem="2Gi", cpu="1"),
    container_image=custom_image,
    secret_requests=[
    Secret(
        group="aws-credentials",
        key="AWS_ACCESS_KEY_ID",
    ),
    Secret(
        group="aws-credentials",
        key="AWS_SECRET_ACCESS_KEY",
    ),
])
n

Niels Bantilan

09/21/2023, 6:36 PM
ah, you’ll need to inject it into your env vars in the task function body:
Copy code
@task(...)
def my_task(...):
    os.environ["AWS_ACCESS_KEY_ID"] = secret_manager.get("aws-credentials", "AWS_ACCESS_KEY_ID")
    os.environ["AWS_SECRET_ACCESS_KEY"] = secret_manager.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")
There’s a PR to make this more convenient
a

Anirudh Sridhar

09/21/2023, 6:38 PM
Copy code
sc = SecretsManager()
    os.environ["AWS_ACCESS_KEY_ID"] = sc.get("aws-credentials", "AWS_ACCESS_KEY_ID")
    os.environ["AWS_SECRET_ACCESS_KEY"] = sc.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")
is this fine @Niels Bantilan?
Copy code
Message:

    Unable to find secret for key AWS_ACCESS_KEY_ID in group aws-credentials in Env Var:_FSEC_AWS-CREDENTIALS_AWS_ACCESS_KEY_ID and FilePath: /etc/secrets/aws-credentials/aws_access_key_id
getting this error
n

Niels Bantilan

09/21/2023, 6:54 PM
Any ideas @Kevin Su @Yee ?
k

Kevin Su

09/21/2023, 8:12 PM
it doesn’t work because flytekit need the credentials before running the task. e.g. download data when pod is running, it starts to 1. download data 2. run the @task. <- you add secret here, but download data will fail first. 3. upload data
#1726 , yes, this can address your issue
n

Niels Bantilan

09/21/2023, 9:52 PM
@Anirudh Sridhar can you share the task code exactly? Is the os.environ in the task function body?
a

Anirudh Sridhar

09/22/2023, 4:30 AM
Yes
Copy code
@task(task_config=ray_config,
    requests=Resources(mem="2Gi", cpu="1"),
    container_image=custom_image)  
def fn():
   sc = SecretsManager()
   os.environ["AWS_ACCESS_KEY_ID"] = sc.get("aws-credentials", "AWS_ACCESS_KEY_ID")
   os.environ["AWS_SECRET_ACCESS_KEY"] = sc.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")
Powered by