https://flyte.org
Join Slack
is there an easy way to mount some dir from execution space to container running task, or the only o...
# flyte-support
m
is there an easy way to mount some dir from execution space to container running task, or the only option is to fully build image, publish and then pull into k8s? I want fast iterations 🙂
f
Are you using pyflyte run
This should allow you to iterate quickly- it just zips your code and skips rebuild
m
probably problem is that my code is not inside the top dir, but higher and the code is not packed
pyflyte register works almost as expected, whilst pyflyte run does not mount code from top-most package
f
Hmm this is interesting
Cc @glamorous-carpet-83516 would love to understand your setup
g
do you have 
__init___.py
in the folder
could you share the output of 
tree .
m
mlrobot/mlrobot/workflows$ tree . . ├── init.py ├── example.py but actually mlrobot (the parent dir) also contains init.py and I use scripts from this module
@glamorous-carpet-83516 any more things I should check? still no luck
g
which version of flytekit are you using
m
I guess it’s 1.5
g
is the class you import in example.py a flyte task or workflow?
m
ah no
g
if not, flytekit wont copy that module
m
it’s just external code, here is the tree of the parent dirs: mlrobot
Copy code
├── mlrobot
│   ├── __init__.py
│   ├── extra
│   │   └── __init__.py
│   ├── workflows
│   │   ├── example.py
but why, any way to force that behaviour? as flyte register works as expected I do not want to pack that code into package on each update
g
The idea behind this is that we don’t want to copy all the files in 
source_root
. some people may have lot of unnecessary files in that folder. btw only flyte task and workflow has metadata container relative path of file
I think add can add a flag like 
copy-all
to pyflyte run. wdyt pyflyte run --copy-all --remote example.py wf
cc @thankful-minister-83577
m
so is it like Flyte is deigned to work with existing packages/prepared images, not to add workflows to current-project code? it’s a bit misleading; so this --copy-all is yet to be implemented?
g
not yet, I can add that tomorrow. we encourage people to use non-fast register (image contains python package and code) when running workflow in production because it saves time by not having to copy code to the pod from S3
m
it makes sense for production, but for fast experiments it’s a bit overwhelming (we used to have luigi and now would like to have tasks distribution, so we try flyte for that purpose) thanks! would be great to have that feature
g
I see. pyflyte register is also fast-register (will copy the code to s3) by default. To workaround it, you could use pyflyte register + manually trigger workflow from UI
m
yeah, it’s the current solution, but we have really huge configs, usually in such systems it’s possible to trigger from UI with json description of the params, is it also possible with Flyte? (we have ~100 lines of configs for our regular pipelines with ~20 different params)
g
@miniature-zebra-33903 Could you help me test it?
Copy code
pip install git+<https://github.com/flyteorg/flytekit@769334528e99df774cae86a1f18de55b22130819>
156 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by