https://flyte.org logo
Title
a

Anatolii Bardukov

05/03/2023, 11:41 AM
is there an easy way to mount some dir from execution space to container running task, or the only option is to fully build image, publish and then pull into k8s? I want fast iterations πŸ™‚
k

Ketan (kumare3)

05/03/2023, 1:42 PM
Are you using pyflyte run
This should allow you to iterate quickly- it just zips your code and skips rebuild
a

Anatolii Bardukov

05/03/2023, 1:50 PM
probably problem is that my code is not inside the top dir, but higher and the code is not packed
pyflyte register works almost as expected, whilst pyflyte run does not mount code from top-most package
k

Ketan (kumare3)

05/03/2023, 11:43 PM
Hmm this is interesting
Cc @Kevin Su would love to understand your setup
k

Kevin Su

05/03/2023, 11:51 PM
do you have
__init___.py
in the folder
could you share the output of
tree .
a

Anatolii Bardukov

05/04/2023, 7:23 AM
mlrobot/mlrobot/workflows$ tree . . β”œβ”€β”€ init.py β”œβ”€β”€ example.py but actually mlrobot (the parent dir) also contains init.py and I use scripts from this module
@Kevin Su any more things I should check? still no luck
k

Kevin Su

05/05/2023, 5:40 AM
which version of flytekit are you using
a

Anatolii Bardukov

05/05/2023, 5:41 AM
I guess it’s 1.5
k

Kevin Su

05/05/2023, 5:43 AM
is the class you import in example.py a flyte task or workflow?
a

Anatolii Bardukov

05/05/2023, 5:46 AM
ah no
k

Kevin Su

05/05/2023, 5:47 AM
if not, flytekit wont copy that module
a

Anatolii Bardukov

05/05/2023, 5:48 AM
it’s just external code, here is the tree of the parent dirs: mlrobot
β”œβ”€β”€ mlrobot
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ extra
β”‚   β”‚   └── __init__.py
β”‚   β”œβ”€β”€ workflows
β”‚   β”‚   β”œβ”€β”€ example.py
but why, any way to force that behaviour? as flyte register works as expected I do not want to pack that code into package on each update
k

Kevin Su

05/05/2023, 5:53 AM
The idea behind this is that we don’t want to copy all the files in
source_root
. some people may have lot of unnecessary files in that folder. btw only flyte task and workflow has metadata container relative path of file
I think add can add a flag like
copy-all
to pyflyte run. wdyt pyflyte run --copy-all --remote example.py wf
cc @Yee
a

Anatolii Bardukov

05/05/2023, 6:02 AM
so is it like Flyte is deigned to work with existing packages/prepared images, not to add workflows to current-project code? it’s a bit misleading; so this --copy-all is yet to be implemented?
k

Kevin Su

05/05/2023, 6:34 AM
not yet, I can add that tomorrow. we encourage people to use non-fast register (image contains python package and code) when running workflow in production because it saves time by not having to copy code to the pod from S3
a

Anatolii Bardukov

05/05/2023, 6:36 AM
it makes sense for production, but for fast experiments it’s a bit overwhelming (we used to have luigi and now would like to have tasks distribution, so we try flyte for that purpose) thanks! would be great to have that feature
k

Kevin Su

05/05/2023, 6:40 AM
I see. pyflyte register is also fast-register (will copy the code to s3) by default. To workaround it, you could use pyflyte register + manually trigger workflow from UI
a

Anatolii Bardukov

05/05/2023, 6:45 AM
yeah, it’s the current solution, but we have really huge configs, usually in such systems it’s possible to trigger from UI with json description of the params, is it also possible with Flyte? (we have ~100 lines of configs for our regular pipelines with ~20 different params)
k

Kevin Su

05/05/2023, 11:46 PM
@Anatolii Bardukov Could you help me test it?
pip install git+<https://github.com/flyteorg/flytekit@769334528e99df774cae86a1f18de55b22130819>