salmon-stone-89503
02/08/2023, 5:46 PMproject_1
that includes a dynamic workflow. The dynamic workflow starts a launchplan defined in project_2
(see attached sketch).
This is my code structure:
|- project_1 /
| |- pyproject.toml
| |- project_1 /
| | |- __init__.py
| | |- workflows.py
| | |- tasks.py
| | |- models.py
| | |- ...
|- project_2 /
| |- pyproject.toml
| |- project_2 /
| | |- __init__.py
| | |- workflows.py
| | |- tasks.py
| | |- models.py
| | |- ...
|- Dockerfile
|- docker_build.sh
What I'm doing right now:
β’ add project_2
as a dependency to project one
β’ build a docker image with project_2
installed as a dependency to project 1
β’ package + register
β’ run
This approach requires rebuilding the docker image with every execution because fast register will not realise that the launchplan defined in project_2
should have a different version than the rest of the workflow and it fails while trying to fetch the workflow.
Another option would be to do something like:
remote = FlyteRemote(config=Config.auto())
launchplan = remote.fetch_launch_plan(...)
but that would require my pod to be able to reach flyteadmin because it's inside a dynamic workflow,
Is there anything I can do to avoid rebuilding the docker image for every run? Am I doing something wrong? Any advice is much appreciated.
Another issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesn't provide any link to the newly spawned workflows so I can't reliably track the intermediate states or see why something failed (if it did). (see screenshot, the purple task is dynamic). It also leads to all sorts of mixed status reports (see other screenshot).hallowed-mouse-14616
02/09/2023, 12:39 PMmelodic-magician-71351
02/09/2023, 4:20 PMpyflyte register
will register a new version of the imported launch plan, which will then try to reference a workflow version that doesn't exist (because it was defined in the external package and not imported).
That said I'm not sure how you would specify a version for the launch plan when "passing by value" like this, maybe it is not desirable functionality, and if so we'd like to understand what the right pattern is here (i.e. how do you correctly "pass by reference" for a launch plan or workflow).
I think pass by value is a strong preference for us, because it preserves local functionality. Maybe if we keep the launch flow version in sync with the package __version__
attribute that could serve as a hint during registration?freezing-airport-6809
broad-monitor-993
02/10/2023, 5:06 PMpyflyte register project_1 project_2 --image ...
This should fast-register both projects and keep the launchplan code from project_2
consistentbroad-monitor-993
02/10/2023, 7:49 PMpyflyte package
pyflyte --pkgs project_1 --pkgs project_2 package ...
salmon-stone-89503
02/11/2023, 2:00 AMsalmon-stone-89503
02/14/2023, 1:39 PMpyflyte package
seems to work but just if it's ran from the top level directory (the parent to project_1
and project_2
), thanks!
This is what I see with `flytectl register`:
{"json":{"exec_id":"a6znzvsvnkmmnsdl6m5d","node":"n2","ns":"development","res_ver":"336142287","routine":"worker-35","wf":"<project name>:<domain name>:project_1.workflows.<workflow_name>"},"level":"error","msg":"handling producing dynamic workflow definition failed with error: [system] unable to retrieve launchplan information ::<launchplan_name>:<version>}
so basically the workflow picks up the project and domain properly but it doesn't get forwarded to the imported launchplansalmon-stone-89503
02/14/2023, 1:40 PMbroad-monitor-993
02/14/2023, 3:09 PMsalmon-stone-89503
02/14/2023, 3:20 PMpyflyte --pkgs project_1 --pkgs project_2 package --image <image>
flytectl register --project <project> --domain <domain> --version <version> --archive flyte-package.tgz
broad-monitor-993
02/14/2023, 3:34 PMbroad-monitor-993
02/14/2023, 3:50 PMflytectl demo
cluster if thatβs any consolation π
broad-monitor-993
02/14/2023, 4:24 PM.
βββ Dockerfile
βββ LICENSE
βββ README.md
βββ docker_build.sh
βββ flyte-package.tgz
βββ requirements.txt
βββ workflows
β βββ __init__.py # π module
β βββ __pycache__
β βββ workflows
β βββ __init__.py
β βββ __pycache__
β βββ example.py
βββ workflows2
βββ __init__.py # π module
βββ __pycache__
βββ workflows2
βββ __init__.py
βββ __pycache__
βββ example.py
To summarize, basically I needed to make the top-level workflows
and workflows2
a module as well.
workflows2.example
defines a launch_plan
, which I then import in workflows.example
with
from workflows2.workflows2.example import launch_plan
...
@dynamic
def wf_lp_test() -> typing.List[str]:
out = []
for i in range(3):
out.append(launch_plan()[0])
return out
This produces a dynamic workflow that spawns three launch plans.
Iβm not sure if this solution works well for your requirements and current setup, but basically every time you fast register the latest version of workflows2
should be used.broad-monitor-993
02/14/2023, 4:26 PMAnother issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesnβt provide any link to the newly spawned workflows so I canβt reliably track the intermediate states or see why something failed (if it did)This is currently a limitation in the UI, we need to make a ticket to track this. Would you mind opening up an issue here [flyte-ui] π
user
02/14/2023, 4:26 PMbroad-monitor-993
02/14/2023, 4:53 PMflytectl demo
cluster.
3. Use FlyteRemote: as mentioned in the top post of this thread, launchplans can be fetched from FlyteAdmin and executed using FlyteRemote. The limitation here is that the dynamic workflow pod needs access to the cluster.
π‘ For (1), perhaps fast registration in pyflyte package
and pyflyte register
can be make more flexible such that you can reference subdirectories that are packaged up in the fast-registered zip-file like pyflyte --pkgs <directory>:<module>
so that, e.g. the project_1
module can be inside a non-python module directory but will be made available as a top-level directory in the container /root
. Perhaps another idea is to inject a PYTHONPATH
env variable so that Flyte can find all the subprojects in a repo containing multiple Flyte projects.
Any other thoughts would be appreciated here @thankful-minister-83577 @high-accountant-32689