https://flyte.org logo
#flytekit
Title
# flytekit
t

Tim Bauer

07/14/2022, 2:39 PM
Question regarding fast-registration and tar archives: I've noticed you recently added this
tar_strip_file_attributes
functions. What is the purpose of that? When unpacking fast-registered archives it messes up the permissions:
Copy code
d--------- 3 root root 4096 Jul 14 14:04 unpacked
-rw-r--r-- 1 root root  335 Jul 14 13:59 scriptmode.tar.gz
It doesn't matter if you run as root but we deliberately don't in our containers so everything fast-registered with new flytekit currently breaks.
k

Ketan (kumare3)

07/14/2022, 2:41 PM
This is to get the same archive hash, specifically the time component screws up the hash
I think it's ok to keep user etc if you want
t

Tim Bauer

07/14/2022, 2:41 PM
Ah damn makes sense
k

Ketan (kumare3)

07/14/2022, 2:42 PM
Ya it was hard to figure that out
t

Tim Bauer

07/14/2022, 2:42 PM
user is not the problem, if unpacked by user that one owns it. But we need
drwxr-xr-x
k

Ketan (kumare3)

07/14/2022, 2:43 PM
Ya I guess let's make pervasive perms?
I think that is safe, we were too aggressive
t

Tim Bauer

07/14/2022, 2:52 PM
Removing those two lines results in those permissions when unpacked: - they work for non root user
Copy code
drwx------ 7 tim tim   4096 Jan  1  1970 fixed
-rw-rw-r-- 1 tim tim 583680 Jul 14 16:48 fixed.tar.gz
uid and gid seem to overwritten when unpacking
This fixed fast-registration again for us. But
pyflyte run --remote
still doesn't work. I noticed that it sets the
-dest-dir
argument to
/root
And also it seems to strip the workflow path defined in `flyte.config`:
Copy code
[sdk]
# This option specifies the python packages where-in to search for workflows and tasks workflow packages.
workflow_packages=src.workflows
from the workflow files. I'm running a workflow called "my_wf" from
src/workflows/examples/hello_world.py
- with pyflyte run --remote this gets registered as "examples.hello_world.my_wf", the final argument to pyflyte-execute thus also becomes
-- task-module examples.hello_world
but I want it to be
-- task-module src.workflows.examples.hello_world
k

Ketan (kumare3)

07/14/2022, 3:07 PM
You have missing init files
👍 1
We have decided to standardize names using init
t

Tim Bauer

07/14/2022, 3:08 PM
Indeed, thanks for quick spotting
One more thing I now noticed that pyflyte run doesnt pick up on the service account defined in flyte.config:
Copy code
[auth]
kubernetes_service_account=vault
It runs pods with default
Thank you for always being so fast @Ketan (kumare3) btw. I really wouldn't expect it from someone who is suppose is very busy, but I get really thankful every time 🙏
k

Ketan (kumare3)

07/14/2022, 3:12 PM
Ya the confit has been optimized
We wanted to get rid of the config
This might be an oversight but please help with this migration
t

Tim Bauer

07/14/2022, 3:19 PM
Ok so there is no config anymore? Can you point me to where this
run_level_params
object is being put together in the click context?
Ok I figured how to set dest-dir and service account as arguments to pyflyte run. Opened PR to remove the file permission reset: https://github.com/flyteorg/flytekit/pull/1103
k

Ketan (kumare3)

07/14/2022, 7:33 PM
I delayed in responding- cc @Eduardo Apolinario (eapolinario)
g

Greg Gydush

08/16/2022, 4:57 PM
One more thing I now noticed that pyflyte run doesnt pick up on the service account
@Ketan (kumare3) having a similar issue where the service account specified during registration matches the service account used when a workflow is ran from the UI but does not match the service account used when executing from FlyteRemote
It looks like this problem started when we transitioned from flytekit 0.31.0 -> 1.0.3, still digging into it
32 Views