https://flyte.org logo
Join Slack
Powered by
Question regarding fast-registration and tar archi...
# flytekit
t
Question regarding fast-registration and tar archives: I've noticed you recently added this 
tar_strip_file_attributes
functions. What is the purpose of that? When unpacking fast-registered archives it messes up the permissions:
Copy code
d--------- 3 root root 4096 Jul 14 14:04 unpacked
-rw-r--r-- 1 root root  335 Jul 14 13:59 scriptmode.tar.gz
It doesn't matter if you run as root but we deliberately don't in our containers so everything fast-registered with new flytekit currently breaks.
k
This is to get the same archive hash, specifically the time component screws up the hash
I think it's ok to keep user etc if you want
t
Ah damn makes sense
k
Ya it was hard to figure that out
t
user is not the problem, if unpacked by user that one owns it. But we need 
drwxr-xr-x
k
Ya I guess let's make pervasive perms?
I think that is safe, we were too aggressive
t
Removing those two lines results in those permissions when unpacked: - they work for non root user
Copy code
drwx------ 7 tim tim   4096 Jan  1  1970 fixed
-rw-rw-r-- 1 tim tim 583680 Jul 14 16:48 fixed.tar.gz
uid and gid seem to overwritten when unpacking
This fixed fast-registration again for us. But 
pyflyte run --remote
still doesn't work. I noticed that it sets the 
-dest-dir
argument to 
/root
And also it seems to strip the workflow path defined in `flyte.config`:
Copy code
[sdk]
# This option specifies the python packages where-in to search for workflows and tasks workflow packages.
workflow_packages=src.workflows
from the workflow files. I'm running a workflow called "my_wf" from 
src/workflows/examples/hello_world.py
- with pyflyte run --remote this gets registered as "examples.hello_world.my_wf", the final argument to pyflyte-execute thus also becomes 
-- task-module examples.hello_world
but I want it to be 
-- task-module src.workflows.examples.hello_world
k
You have missing init files
👍 1
We have decided to standardize names using init
t
Indeed, thanks for quick spotting
One more thing I now noticed that pyflyte run doesnt pick up on the service account defined in flyte.config:
Copy code
[auth]
kubernetes_service_account=vault
It runs pods with default
Thank you for always being so fast @Ketan (kumare3) btw. I really wouldn't expect it from someone who is suppose is very busy, but I get really thankful every time 🙏
k
Ya the confit has been optimized
We wanted to get rid of the config
This might be an oversight but please help with this migration
t
Ok so there is no config anymore? Can you point me to where this 
run_level_params
object is being put together in the click context?
Ok I figured how to set dest-dir and service account as arguments to pyflyte run. Opened PR to remove the file permission reset: https://github.com/flyteorg/flytekit/pull/1103
k
I delayed in responding- cc @Eduardo Apolinario (eapolinario)
g
One more thing I now noticed that pyflyte run doesnt pick up on the service account
@Ketan (kumare3) having a similar issue where the service account specified during registration matches the service account used when a workflow is ran from the UI but does not match the service account used when executing from FlyteRemote
It looks like this problem started when we transitioned from flytekit 0.31.0 -> 1.0.3, still digging into it
165 Views
Open in Slack
PreviousNext
Powered by