https://flyte.org logo
Join Slack
Powered by
Hi all, building my first more complex flyte workf...
# ask-the-community
e
Hi all, building my first more complex flyte workflow and I have a couple of questions on best practices. I have a workflow defined in 
project_1
that includes a dynamic workflow. The dynamic workflow starts a launchplan defined in 
project_2
(see attached sketch). This is my code structure:
Copy code
|- project_1 /
|  |- pyproject.toml
|  |- project_1 /
|  |  |- __init__.py
|  |  |- workflows.py
|  |  |- tasks.py
|  |  |- models.py
|  |  |- ...
|- project_2 /
|  |- pyproject.toml
|  |- project_2 /
|  |  |- __init__.py
|  |  |- workflows.py
|  |  |- tasks.py
|  |  |- models.py
|  |  |- ...
|- Dockerfile
|- docker_build.sh
What I'm doing right now: • add 
project_2
as a dependency to project one • build a docker image with 
project_2
installed as a dependency to project 1 • package + register • run This approach requires rebuilding the docker image with every execution because fast register will not realise that the launchplan defined in 
project_2
should have a different version than the rest of the workflow and it fails while trying to fetch the workflow. Another option would be to do something like:
Copy code
remote = FlyteRemote(config=Config.auto())
launchplan = remote.fetch_launch_plan(...)
but that would require my pod to be able to reach flyteadmin because it's inside a dynamic workflow, Is there anything I can do to avoid rebuilding the docker image for every run? Am I doing something wrong? Any advice is much appreciated. Another issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesn't provide any link to the newly spawned workflows so I can't reliably track the intermediate states or see why something failed (if it did). (see screenshot, the purple task is dynamic). It also leads to all sorts of mixed status reports (see other screenshot).
d
cc @Eduardo Apolinario (eapolinario)
e
@Ena Škopelja and I are on the same team, to speak to the behavior we expected: We expect to need to rebuild when there are changes to workflows/launch plans outside the package which is being fast registered. That makes sense. What we didn't expect is that 
pyflyte register
will register a new version of the imported launch plan, which will then try to reference a workflow version that doesn't exist (because it was defined in the external package and not imported). That said I'm not sure how you would specify a version for the launch plan when "passing by value" like this, maybe it is not desirable functionality, and if so we'd like to understand what the right pattern is here (i.e. how do you correctly "pass by reference" for a launch plan or workflow). I think pass by value is a strong preference for us, because it preserves local functionality. Maybe if we keep the launch flow version in sync with the package 
__version__
attribute that could serve as a hint during registration?
k
Sorry we missed the message, will tal in a bit
n
hi @Ena Škopelja have you tried registering both projects at the same time?
Copy code
pyflyte register project_1 project_2 --image ...
This should fast-register both projects and keep the launchplan code from 
project_2
consistent
you can also do with with 
pyflyte package
Copy code
pyflyte --pkgs project_1 --pkgs project_2 package ...
e
I remember trying that and getting an error but I'll give it another go. Thanks 🙏
pyflyte package
seems to work but just if it's ran from the top level directory (the parent to 
project_1
and 
project_2
), thanks! This is what I see with `flytectl register`:
Copy code
{"json":{"exec_id":"a6znzvsvnkmmnsdl6m5d","node":"n2","ns":"development","res_ver":"336142287","routine":"worker-35","wf":"<project name>:<domain name>:project_1.workflows.<workflow_name>"},"level":"error","msg":"handling producing dynamic workflow definition failed with error: [system] unable to retrieve launchplan information ::<launchplan_name>:<version>}
so basically the workflow picks up the project and domain properly but it doesn't get forwarded to the imported launchplan
Any thoughts about the status missmatch? The CLI is equally confusing.
n
can you copy-paste the command your running to package?
e
pyflyte --pkgs project_1 --pkgs project_2 package --image <image>
flytectl register --project <project> --domain <domain> --version <version> --archive flyte-package.tgz
n
cool, I suspect that the nested directories might be an issue, but I’m trying to repro locally now
I’m still doing some investigation on support this use case, but another option I’d like to throw out here is using `reference_launch_plan`: https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.reference_launch_plan.html The downside of this is that it isn’t supported locally (in a python runtime), but should work on a 
flytectl demo
cluster if that’s any consolation 😅
hey @Ena Škopelja so I managed to get something working on a flyte demo cluster.
Copy code
.
├── Dockerfile
├── LICENSE
├── README.md
├── docker_build.sh
├── flyte-package.tgz
├── requirements.txt
├── workflows
│   ├── __init__.py  # 👈 module
│   ├── __pycache__
│   └── workflows
│       ├── __init__.py
│       ├── __pycache__
│       └── example.py
└── workflows2
    ├── __init__.py  # 👈 module
    ├── __pycache__
    └── workflows2
        ├── __init__.py
        ├── __pycache__
        └── example.py
To summarize, basically I needed to make the top-level 
workflows
and 
workflows2
a module as well. 
workflows2.example
defines a 
launch_plan
, which I then import in 
workflows.example
with
Copy code
from workflows2.workflows2.example import launch_plan

...

@dynamic
def wf_lp_test() -> typing.List[str]:
    out = []
    for i in range(3):
        out.append(launch_plan()[0])
        return out
This produces a dynamic workflow that spawns three launch plans. I’m not sure if this solution works well for your requirements and current setup, but basically every time you fast register the latest version of 
workflows2
should be used.
Another issue is that the way I run things now makes the UI fail to expand the dynamic workflow and doesn’t provide any link to the newly spawned workflows so I can’t reliably track the intermediate states or see why something failed (if it did)
This is currently a limitation in the UI, we need to make a ticket to track this. Would you mind opening up an issue here [flyte-ui] 👇
s
n
So to summarize, when working with multi-project Flyte repos there are effectively 3 options when using launchplans across projects: 1. fast registering all projects: this requires all projects to be modules defined at the top-level of the repo (basically this solution I posted earlier). 2. use reference_launch_plan: this lets you use launchplans across projects, referenced by project, domain, name, and version. The limitation of this is that you can test locally on a python runtime, but should be testable in a 
flytectl demo
cluster. 3. Use FlyteRemote: as mentioned in the top post of this thread, launchplans can be fetched from FlyteAdmin and executed using FlyteRemote. The limitation here is that the dynamic workflow pod needs access to the cluster. 💡 For (1), perhaps fast registration in 
pyflyte package
and 
pyflyte register
can be make more flexible such that you can reference subdirectories that are packaged up in the fast-registered zip-file like 
pyflyte --pkgs <directory>:<module>
so that, e.g. the 
project_1
module can be inside a non-python module directory but will be made available as a top-level directory in the container 
/root
. Perhaps another idea is to inject a 
PYTHONPATH
env variable so that Flyte can find all the subprojects in a repo containing multiple Flyte projects. Any other thoughts would be appreciated here @Yee @Eduardo Apolinario (eapolinario)
158 Views
Open in Slack
PreviousNext
Powered by