hey team is there a way to pass aws creds to pod u...
# flyte-supportb
brash-piano-42461
09/21/2023, 6:02 PMhey team is there a way to pass aws creds to pod using k8s secrets?
b
broad-monitor-993
09/21/2023, 6:05 PMhi @brash-piano-42461 have you checked out the secrets guide? By default Flyte uses k8s secrets: as long as you create secrets like so and use the
flytekit.Secretb
brash-piano-42461
09/21/2023, 6:08 PMso i have created the secrets so how do put my secret value here
https://github.com/flyteorg/flyte/blob/be7c7891df6e117c031d9b1f6b8e298548e709bd/flyte.yaml#L62-L67
brash-piano-42461
09/21/2023, 6:09 PM@broad-monitor-993
brash-piano-42461
09/21/2023, 6:09 PMthis is how i created secrets
Copy code
kubectl create secret generic aws-credentials \
--from-literal=AWS_ACCESS_KEY_ID=your-access-key-id \
--from-literal=AWS_SECRET_ACCESS_KEY=your-secret-access-keyf
freezing-airport-6809
09/21/2023, 6:10 PM
but why not use service accounts?
b
brash-piano-42461
09/21/2023, 6:10 PMi am testing in local sandbox cluster
brash-piano-42461
09/21/2023, 6:11 PMand im not too aware about service account
b
broad-monitor-993
09/21/2023, 6:17 PMyour secret request should look something like:
Copy code
@task(secret_requests=[
Secret(
group="aws-credentials",
key="AWS_ACCESS_KEY_ID",
),
Secret(
group="aws-credentials",
key="AWS_SECRET_ACCESS_KEY",
),
])b
brash-piano-42461
09/21/2023, 6:17 PMoh i need to pass it in task
b
broad-monitor-993
09/21/2023, 6:18 PMyep! check out the secrets guide
b
brash-piano-42461
09/21/2023, 6:18 PMya i saw that wondering there was any other way?
b
broad-monitor-993
09/21/2023, 6:20 PMnot that I know of
b
brash-piano-42461
09/21/2023, 6:28 PMhey @broad-monitor-993 still getting same error
Copy code
Unable to locate credentialsbrash-piano-42461
09/21/2023, 6:29 PM Copy code
@task(task_config=ray_config,
requests=Resources(mem="2Gi", cpu="1"),
container_image=custom_image,
secret_requests=[
Secret(
group="aws-credentials",
key="AWS_ACCESS_KEY_ID",
),
Secret(
group="aws-credentials",
key="AWS_SECRET_ACCESS_KEY",
),
])b
broad-monitor-993
09/21/2023, 6:36 PMah, you’ll need to inject it into your env vars in the task function body:
Copy code
@task(...)
def my_task(...):
os.environ["AWS_ACCESS_KEY_ID"] = secret_manager.get("aws-credentials", "AWS_ACCESS_KEY_ID")
os.environ["AWS_SECRET_ACCESS_KEY"] = secret_manager.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")b
brash-piano-42461
09/21/2023, 6:38 PM Copy code
sc = SecretsManager()
os.environ["AWS_ACCESS_KEY_ID"] = sc.get("aws-credentials", "AWS_ACCESS_KEY_ID")
os.environ["AWS_SECRET_ACCESS_KEY"] = sc.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")brash-piano-42461
09/21/2023, 6:39 PMis this fine @broad-monitor-993?
brash-piano-42461
09/21/2023, 6:42 PM Copy code
Message:
Unable to find secret for key AWS_ACCESS_KEY_ID in group aws-credentials in Env Var:_FSEC_AWS-CREDENTIALS_AWS_ACCESS_KEY_ID and FilePath: /etc/secrets/aws-credentials/aws_access_key_idbrash-piano-42461
09/21/2023, 6:42 PMgetting this error
b
broad-monitor-993
09/21/2023, 6:54 PMAny ideas @glamorous-carpet-83516 @thankful-minister-83577 ?
g
glamorous-carpet-83516
09/21/2023, 8:12 PMit doesn’t work because flytekit need the credentials before running the task. e.g. download data
when pod is running, it starts to
1. download data
2. run the @task. <- you add secret here, but download data will fail first.
3. upload data
glamorous-carpet-83516
09/21/2023, 8:12 PM#1726 , yes, this can address your issue
b
broad-monitor-993
09/21/2023, 9:52 PM@brash-piano-42461 can you share the task code exactly? Is the os.environ in the task function body?
b
brash-piano-42461
09/22/2023, 4:30 AMYes
brash-piano-42461
09/22/2023, 5:11 AM Copy code
@task(task_config=ray_config,
requests=Resources(mem="2Gi", cpu="1"),
container_image=custom_image)
def fn():
sc = SecretsManager()
os.environ["AWS_ACCESS_KEY_ID"] = sc.get("aws-credentials", "AWS_ACCESS_KEY_ID")
os.environ["AWS_SECRET_ACCESS_KEY"] = sc.get("aws-credentials", "AWS_SECRET_ACCESS_KEY")6 Views