Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
a
Andrew Korzhuev
10/28/2022, 11:45 AMHello! We're deploying Flyte on EKS and prefer to use workflow-specific IAM roles. Could you say if there is a naming schema for writing temporary files to S3 we could depend on to restrict permissions at least on the project level?
For example we see that currently there are multiple different naming, which makes it complicated to create reliable bucket access policies:
s3://.../2y/project-x/development/
s3://.../metadata/propeller/project-x-development-aq9kwx7nbqmxhccp2wgj/y
Yee
10/28/2022, 3:10 PMcan you do an
lstrying to confirm something
just need to see what the names of the files kinda look like.
also do you think you could cut a gh issue for this please?
a
Andrew Korzhuev
10/29/2022, 5:32 PMIt looks like this:
2022-10-12 16:53:28 30720 0p/project-x/development/QMAM74EECKTJ4JDNWMCVXHEAH4======/fast91686145481555fce72a2a47137a26f2.tar.gz
2022-10-06 10:16:26 30720 0y/project-x/development/PB73FGHUK7JKFQRHCTKEKHT4AA======/fast2e6442b4f38ab209cae9f0b3292d1b57.tar.gz
2022-10-21 17:38:27 40960 0z/project-x/development/ZDWLO6645EE73ENUB4I2U4C764======/fastdb6e0c43aec1ae7b60cdd14797da2f54.tar.gz
2022-09-30 14:13:35 30720 1l/project-x/development/TVHEJEB5H3KG2FRKQ43IYHCKJI======/fast0400dadb6f2dac21d77422b62926eb19.tar.gz
2022-09-30 12:58:44 30720 1t/project-x/development/U7OQA2SO75O3VNX572PZ2XZH74======/fast3ebee174a68065b4841c2308f407c2e6.tar.gz
....
2022-09-14 10:08:15 1007 metadata/admin/project-x/development/.flytegen.workflow.say_hello/v1
2022-09-21 10:39:07 1010 metadata/admin/project-x/development/.flytegen.workflow.say_hello/v11
2022-09-21 16:30:01 1110 metadata/admin/project-x/development/.flytegen.workflows.example.say_hello/v10
2022-09-14 15:31:55 950 metadata/admin/project-x/development/flyte.workflows.example.my_wf/v1
2022-10-07 12:50:10 1749 metadata/admin/project-x/development/workflow.geolocation_wf/00d79fa06eb451f6e5f8747efb6bb3fdfce01de1
2022-10-28 15:03:52 1017 metadata/admin/project-x/development/workflow.geolocation_wf/0d6d35f
2022-10-21 17:38:28 7285 metadata/admin/project-x/development/workflow.geolocation_wf/18e0991
2022-10-19 16:49:50 4625 metadata/admin/project-x/development/workflow.geolocation_wf/25772cb
2022-10-11 11:30:55 1929 metadata/admin/project-x/development/workflow.geolocation_wf/6174a5b
....
2022-09-21 13:07:07 0 metadata/project-x/development/a2lg56hwv25rbxbg5v4x/inputs
2022-09-21 13:07:07 0 metadata/project-x/development/a2lg56hwv25rbxbg5v4x/user_inputs
2022-10-06 17:30:02 0 metadata/project-x/development/a2qqvz6lbcdd6pmngzwd/inputs
2022-10-06 17:30:02 0 metadata/project-x/development/a2qqvz6lbcdd6pmngzwd/user_inputs
2022-10-07 15:43:29 0 metadata/project-x/development/a46h5vpgv8zrbb5lf8xv/inputs
2022-10-07 15:43:29 0 metadata/project-x/development/a46h5vpgv8zrbb5lf8xv/user_inputs
2022-10-17 09:24:16 0 metadata/project-x/development/a4bqd77f8zr54vdlhm9d/inputs
2022-10-17 09:24:16 0 metadata/project-x/development/a4bqd77f8zr54vdlhm9d/user_inputs....
2022-09-21 14:46:05 0 metadata/propeller/project-x-development-a2lg56hwv25rbxbg5v4x/start-node/data/0/outputs.pb
2022-10-06 17:30:07 0 metadata/propeller/project-x-development-a2qqvz6lbcdd6pmngzwd/n0/data/inputs.pb
2022-10-06 17:30:07 0 metadata/propeller/project-x-development-a2qqvz6lbcdd6pmngzwd/start-node/data/0/outputs.pb
2022-10-07 15:43:59 2234 metadata/propeller/project-x-development-a46h5vpgv8zrbb5lf8xv/n0/data/0/error.pb
2022-10-07 15:43:34 0 metadata/propeller/project-x-development-a46h5vpgv8zrbb5lf8xv/n0/data/inputs.pb
2022-10-07 15:43:34 0 metadata/propeller/project-x-development-a46h5vpgv8zrbb5lf8xv/start-node/data/0/outputs.pb
2022-10-17 09:26:54 1335 metadata/propeller/project-x-development-a4bqd77f8zr54vdlhm9d/n0/data/0/error.pb
...Wrote up a ticket for it https://github.com/flyteorg/flyte/issues/3039, not sure if it should be housekeeping or a bug
y
Yee
10/31/2022, 4:24 PMsorry can you clarify the ls please?
everything you pasted was under
s3://.../2y/project-x/development/or both of the the prefixes you had listed initially?
a
Andrew Korzhuev
10/31/2022, 4:27 PMUnder this dir I only see one file
2022-10-27 17:05:00 30720 2y/project-x/production/73IT45V7GAQCQ3SJAVSZPT2I5E======/fasta21ee57b163bb75f71419f3be22de564.tar.gzt
Tyler Su
11/22/2022, 10:15 PMHi Andrew, following up on this just in case: the PR was merged and released in flyteadmin v1.1.55 here: https://github.com/flyteorg/flyteadmin/releases/tag/v1.1.55
a
Andrew Korzhuev
11/23/2022, 12:20 PMGreat, thank you!