is there an easy way to mount some dir from execut...
# flyte-supportm
miniature-zebra-33903
05/03/2023, 11:41 AMis there an easy way to mount some dir from execution space to container running task, or the only option is to fully build image, publish and then pull into k8s? I want fast iterations 🙂
f
freezing-airport-6809
05/03/2023, 1:42 PM
Are you using pyflyte run
freezing-airport-6809
05/03/2023, 1:43 PM
This should allow you to iterate quickly- it just zips your code and skips rebuild
m
miniature-zebra-33903
05/03/2023, 1:50 PMprobably problem is that my code is not inside the top dir, but higher and the code is not packed
miniature-zebra-33903
05/03/2023, 11:29 PMpyflyte register works almost as expected, whilst pyflyte run does not mount code from top-most package
f
freezing-airport-6809
05/03/2023, 11:43 PM
Hmm this is interesting
freezing-airport-6809
05/03/2023, 11:43 PM
Cc @glamorous-carpet-83516 would love to understand your setup
g
glamorous-carpet-83516
05/03/2023, 11:51 PMdo you have
__init___.pyglamorous-carpet-83516
05/03/2023, 11:52 PMcould you share the output of
tree .m
miniature-zebra-33903
05/04/2023, 7:23 AMmlrobot/mlrobot/workflows$ tree .
.
├── init.py
├── example.py
but actually mlrobot (the parent dir) also contains init.py and I use scripts from this module
miniature-zebra-33903
05/05/2023, 5:38 AM@glamorous-carpet-83516 any more things I should check? still no luck
g
glamorous-carpet-83516
05/05/2023, 5:40 AMwhich version of flytekit are you using
m
miniature-zebra-33903
05/05/2023, 5:41 AMI guess it’s 1.5
g
glamorous-carpet-83516
05/05/2023, 5:43 AMis the class you import in example.py a flyte task or workflow?
m
miniature-zebra-33903
05/05/2023, 5:46 AMah no
g
glamorous-carpet-83516
05/05/2023, 5:47 AMif not, flytekit wont copy that module
m
miniature-zebra-33903
05/05/2023, 5:48 AMit’s just external code, here is the tree of the parent dirs:
mlrobot
Copy code
├── mlrobot
│ ├── __init__.py
│ ├── extra
│ │ └── __init__.py
│ ├── workflows
│ │ ├── example.pyminiature-zebra-33903
05/05/2023, 5:49 AMbut why, any way to force that behaviour? as flyte register works as expected
I do not want to pack that code into package on each update
g
glamorous-carpet-83516
05/05/2023, 5:53 AMThe idea behind this is that we don’t want to copy all the files in
source_rootglamorous-carpet-83516
05/05/2023, 5:54 AMI think add can add a flag like
copy-allglamorous-carpet-83516
05/05/2023, 5:57 AMcc @thankful-minister-83577
m
miniature-zebra-33903
05/05/2023, 6:02 AMso is it like Flyte is deigned to work with existing packages/prepared images, not to add workflows to current-project code? it’s a bit misleading;
so this --copy-all is yet to be implemented?
g
glamorous-carpet-83516
05/05/2023, 6:34 AMnot yet, I can add that tomorrow. we encourage people to use non-fast register (image contains python package and code) when running workflow in production because it saves time by not having to copy code to the pod from S3
m
miniature-zebra-33903
05/05/2023, 6:36 AMit makes sense for production, but for fast experiments it’s a bit overwhelming (we used to have luigi and now would like to have tasks distribution, so we try flyte for that purpose)
thanks! would be great to have that feature
g
glamorous-carpet-83516
05/05/2023, 6:40 AMI see. pyflyte register is also fast-register (will copy the code to s3) by default. To workaround it, you could use pyflyte register + manually trigger workflow from UI
m
miniature-zebra-33903
05/05/2023, 6:45 AMyeah, it’s the current solution, but we have really huge configs, usually in such systems it’s possible to trigger from UI with json description of the params, is it also possible with Flyte? (we have ~100 lines of configs for our regular pipelines with ~20 different params)
g
glamorous-carpet-83516
05/05/2023, 11:46 PM@miniature-zebra-33903 Could you help me test it?
Copy code
pip install git+<https://github.com/flyteorg/flytekit@769334528e99df774cae86a1f18de55b22130819>154 Views