Is there a config somewhere I can set aws sse keys for S3 operations on encrypted buckets?

brief-cricket-90988

10/11/2023, 6:03 PM

Is there a config somewhere I can set aws sse keys for S3 operations on encrypted buckets?

glamorous-carpet-83516

10/11/2023, 6:20 PM

IIRC, it not exposed right now. let me create a pr shortly.

glamorous-carpet-83516

10/11/2023, 7:39 PM

could you give it a try? https://github.com/flyteorg/flytekit/pull/1887/files You need to update the default-env. add FLYTE_AWS_SERVER_SIDE_ENCRYPTION to the config

brief-cricket-90988

10/12/2023, 6:03 PM

Thank you, Kevin. I actually have a ticket open with Union right now. We are having issues with a particular account/bucket that was SSE-S3 encrypted at first. Thinking that was problem, I removed the encryption on the bucket and we are still seeing the issue. I will test drive this branch today if we don't land on a solution with the Union ticket though I don't think the SSE is the culprit right now

gorgeous-waitress-5026

11/29/2023, 2:15 PM

Just running into what looks to be the same issue in my environment -- we use SSE and I'm seeing a 403 when using

pyflyte run --remote

. Will try setting the env var and a

helm upgrade

but looks like this PR never landed / shipped

gorgeous-waitress-5026

11/29/2023, 2:21 PM

Left a comment on your PR @glamorous-carpet-83516 -- the setting needs to be more flexible than what you have defined

gorgeous-waitress-5026

11/29/2023, 2:23 PM

MLflow does this in a way that works -- https://mlflow.org/docs/latest/tracking.html#amazon-s3-and-s3-compatible-storage

3 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.