https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Andrew Korzhuev

10/28/2022, 11:45 AM
Hello! We're deploying Flyte on EKS and prefer to use workflow-specific IAM roles. Could you say if there is a naming schema for writing temporary files to S3 we could depend on to restrict permissions at least on the project level? For example we see that currently there are multiple different naming, which makes it complicated to create reliable bucket access policies:
Copy code
s3://.../2y/project-x/development/
s3://.../metadata/propeller/project-x-development-aq9kwx7nbqmxhccp2wgj/
y

Yee

10/28/2022, 3:10 PM
can you do an
ls
on the contents of the first one please @Andrew Korzhuev
trying to confirm something
just need to see what the names of the files kinda look like.
also do you think you could cut a gh issue for this please?
a

Andrew Korzhuev

10/29/2022, 5:32 PM
It looks like this:
Copy code
2022-10-12 16:53:28      30720 0p/project-x/development/QMAM74EECKTJ4JDNWMCVXHEAH4======/fast91686145481555fce72a2a47137a26f2.tar.gz
2022-10-06 10:16:26      30720 0y/project-x/development/PB73FGHUK7JKFQRHCTKEKHT4AA======/fast2e6442b4f38ab209cae9f0b3292d1b57.tar.gz
2022-10-21 17:38:27      40960 0z/project-x/development/ZDWLO6645EE73ENUB4I2U4C764======/fastdb6e0c43aec1ae7b60cdd14797da2f54.tar.gz
2022-09-30 14:13:35      30720 1l/project-x/development/TVHEJEB5H3KG2FRKQ43IYHCKJI======/fast0400dadb6f2dac21d77422b62926eb19.tar.gz
2022-09-30 12:58:44      30720 1t/project-x/development/U7OQA2SO75O3VNX572PZ2XZH74======/fast3ebee174a68065b4841c2308f407c2e6.tar.gz
....
2022-09-14 10:08:15       1007 metadata/admin/project-x/development/.flytegen.workflow.say_hello/v1
2022-09-21 10:39:07       1010 metadata/admin/project-x/development/.flytegen.workflow.say_hello/v11
2022-09-21 16:30:01       1110 metadata/admin/project-x/development/.flytegen.workflows.example.say_hello/v10
2022-09-14 15:31:55        950 metadata/admin/project-x/development/flyte.workflows.example.my_wf/v1
2022-10-07 12:50:10       1749 metadata/admin/project-x/development/workflow.geolocation_wf/00d79fa06eb451f6e5f8747efb6bb3fdfce01de1
2022-10-28 15:03:52       1017 metadata/admin/project-x/development/workflow.geolocation_wf/0d6d35f
2022-10-21 17:38:28       7285 metadata/admin/project-x/development/workflow.geolocation_wf/18e0991
2022-10-19 16:49:50       4625 metadata/admin/project-x/development/workflow.geolocation_wf/25772cb
2022-10-11 11:30:55       1929 metadata/admin/project-x/development/workflow.geolocation_wf/6174a5b
....
2022-09-21 13:07:07          0 metadata/project-x/development/a2lg56hwv25rbxbg5v4x/inputs
2022-09-21 13:07:07          0 metadata/project-x/development/a2lg56hwv25rbxbg5v4x/user_inputs
2022-10-06 17:30:02          0 metadata/project-x/development/a2qqvz6lbcdd6pmngzwd/inputs
2022-10-06 17:30:02          0 metadata/project-x/development/a2qqvz6lbcdd6pmngzwd/user_inputs
2022-10-07 15:43:29          0 metadata/project-x/development/a46h5vpgv8zrbb5lf8xv/inputs
2022-10-07 15:43:29          0 metadata/project-x/development/a46h5vpgv8zrbb5lf8xv/user_inputs
2022-10-17 09:24:16          0 metadata/project-x/development/a4bqd77f8zr54vdlhm9d/inputs
2022-10-17 09:24:16          0 metadata/project-x/development/a4bqd77f8zr54vdlhm9d/user_inputs....
2022-09-21 14:46:05          0 metadata/propeller/project-x-development-a2lg56hwv25rbxbg5v4x/start-node/data/0/outputs.pb
2022-10-06 17:30:07          0 metadata/propeller/project-x-development-a2qqvz6lbcdd6pmngzwd/n0/data/inputs.pb
2022-10-06 17:30:07          0 metadata/propeller/project-x-development-a2qqvz6lbcdd6pmngzwd/start-node/data/0/outputs.pb
2022-10-07 15:43:59       2234 metadata/propeller/project-x-development-a46h5vpgv8zrbb5lf8xv/n0/data/0/error.pb
2022-10-07 15:43:34          0 metadata/propeller/project-x-development-a46h5vpgv8zrbb5lf8xv/n0/data/inputs.pb
2022-10-07 15:43:34          0 metadata/propeller/project-x-development-a46h5vpgv8zrbb5lf8xv/start-node/data/0/outputs.pb
2022-10-17 09:26:54       1335 metadata/propeller/project-x-development-a4bqd77f8zr54vdlhm9d/n0/data/0/error.pb
...
Wrote up a ticket for it https://github.com/flyteorg/flyte/issues/3039, not sure if it should be housekeeping or a bug
y

Yee

10/31/2022, 4:24 PM
sorry can you clarify the ls please?
everything you pasted was under
s3://.../2y/project-x/development/
?
or both of the the prefixes you had listed initially?
a

Andrew Korzhuev

10/31/2022, 4:27 PM
Under this dir I only see one file
Copy code
2022-10-27 17:05:00      30720 2y/project-x/production/73IT45V7GAQCQ3SJAVSZPT2I5E======/fasta21ee57b163bb75f71419f3be22de564.tar.gz
t

Tyler Su

11/22/2022, 10:15 PM
Hi Andrew, following up on this just in case: the PR was merged and released in flyteadmin v1.1.55 here: https://github.com/flyteorg/flyteadmin/releases/tag/v1.1.55
a

Andrew Korzhuev

11/23/2022, 12:20 PM
Great, thank you!
78 Views