is there an easy way to mount some dir from execut...
# ask-the-community
a
is there an easy way to mount some dir from execution space to container running task, or the only option is to fully build image, publish and then pull into k8s? I want fast iterations πŸ™‚
k
Are you using pyflyte run
This should allow you to iterate quickly- it just zips your code and skips rebuild
a
probably problem is that my code is not inside the top dir, but higher and the code is not packed
pyflyte register works almost as expected, whilst pyflyte run does not mount code from top-most package
k
Hmm this is interesting
Cc @Kevin Su would love to understand your setup
k
do you have
__init___.py
in the folder
could you share the output of
tree .
a
mlrobot/mlrobot/workflows$ tree . . β”œβ”€β”€ init.py β”œβ”€β”€ example.py but actually mlrobot (the parent dir) also contains init.py and I use scripts from this module
@Kevin Su any more things I should check? still no luck
k
which version of flytekit are you using
a
I guess it’s 1.5
k
is the class you import in example.py a flyte task or workflow?
a
ah no
k
if not, flytekit wont copy that module
a
it’s just external code, here is the tree of the parent dirs: mlrobot
Copy code
β”œβ”€β”€ mlrobot
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ extra
β”‚   β”‚   └── __init__.py
β”‚   β”œβ”€β”€ workflows
β”‚   β”‚   β”œβ”€β”€ example.py
but why, any way to force that behaviour? as flyte register works as expected I do not want to pack that code into package on each update
k
The idea behind this is that we don’t want to copy all the files in
source_root
. some people may have lot of unnecessary files in that folder. btw only flyte task and workflow has metadata container relative path of file
I think add can add a flag like
copy-all
to pyflyte run. wdyt pyflyte run --copy-all --remote example.py wf
cc @Yee
a
so is it like Flyte is deigned to work with existing packages/prepared images, not to add workflows to current-project code? it’s a bit misleading; so this --copy-all is yet to be implemented?
k
not yet, I can add that tomorrow. we encourage people to use non-fast register (image contains python package and code) when running workflow in production because it saves time by not having to copy code to the pod from S3
a
it makes sense for production, but for fast experiments it’s a bit overwhelming (we used to have luigi and now would like to have tasks distribution, so we try flyte for that purpose) thanks! would be great to have that feature
k
I see. pyflyte register is also fast-register (will copy the code to s3) by default. To workaround it, you could use pyflyte register + manually trigger workflow from UI
a
yeah, it’s the current solution, but we have really huge configs, usually in such systems it’s possible to trigger from UI with json description of the params, is it also possible with Flyte? (we have ~100 lines of configs for our regular pipelines with ~20 different params)
k
@Anatolii Bardukov Could you help me test it?
Copy code
pip install git+<https://github.com/flyteorg/flytekit@769334528e99df774cae86a1f18de55b22130819>
106 Views