https://flyte.org logo
Join Slack
Powered by
Hello everyone :wave: Two questions appeared agai...
# flyte-support
c
Hello everyone 👋 Two questions appeared again. Question one, I wrote a root_workflow that runs other workflows. Now I have 1830 objects, registration of this root_workflow takes 40 minutes, is there a way to speed this up? the registration command:
Copy code
flytectl register files \
    --project flytesnacks \
    --domain development \
    --archive flyte-package.tgz --version asdaf908
Question two, workflow calls crash with a module import error, the project structure is as follows:
a
Now I have 1830 objects, registration of this root_workflow takes 40 minutes, is there a way to speed this up?
I think fast registration, and ignoring files that don't need to be packaged could help https://docs.flyte.org/en/latest/user_guide/flyte_fundamentals/registering_workflows.html#fast-registration
workflow calls crash with a module import error
The above section has some guidance to improve the chances of modules being discovered via PYTHONPATH
f
Also 1800 workflows registered at once, you need parallelism. Also why register all of them at once, break them up
c
One master workflow is registered for 40 minutes
the code that I gave above
Copy code
wf_t = ImperativeWorkflow(name="root-workflow")  # , failure_policy=failure_policy

node_dic = {}

for parent, child in dependencies:
    if parent!=child:
        if parent not in node_dic:
            parent_module = importlib.import_module(f'workflows.rv_wrk.{parent}')
            parent_wf=getattr(parent_module, 'wf_test')
            parent_launch_plan = LaunchPlan.get_or_create(
                parent_wf, parent
            )
            node_dic[parent] = wf_t.add_launch_plan(parent_launch_plan)

        if child not in node_dic:
            child_module = importlib.import_module(f'workflows.rv_wrk.{child}') #workflows.
            child_wf=getattr(child_module, 'wf_test')
            parent_launch_plan = LaunchPlan.get_or_create(
                child_wf, child
            )
            node_dic[child] = wf_t.add_launch_plan(parent_launch_plan)

                # child_node = child_node.with_overrides(
                #     name=f"b-{child}"
                # )

        node_dic[parent].runs_before(node_dic[child])
But no, I was wrong. Thank you
@freezing-airport-6809 I used my full-scale graph containing 4333 nodes and 210161 edges. the registration of the master workflow took 23 minutes, after which an error appeared
f
Hmm wow, I think you need more memory in the back and more timeout
b
See if you have any heavy files in your scope which arent ignored - for example we had .git folder which weighed 200mb back then
t
add 
-v
right after the 
pyflyte
command to turn on verbose mode, which should print out the list of files included in any tarball that tom mentioned. also, to separate the concerns and maybe to debug, could you first run pyflyte package? that will just run the compilation step and produce the proto files which you can even inspect to see the structure of each workflow if you want to verify.
c
I tried to set -v, but it did not help to understand the reason for the crash. flyte manages to register all LP workflow masters. After that, the program freezes for a few seconds, exits the -v mode and crashes with the same error, there are no logs in flyte admin.
The only thing I see is
Copy code
2025/01/15 16:06:50 /flyteorg/build/flyteadmin/pkg/repositories/gormimpl/workflow_repo.go:53 context canceled
[0.039ms] [rows:0] SELECT * FROM "workflows" WHERE "workflows"."project" = 'flytesnacks' AND "workflows"."domain" = 'development' AND "workflows"."name" = 'root-workflow' AND "workflows"."version" = 'lxvXAH4Gyzqv7UXF4LKgvQ' LIMIT 1
t
so all the entities successfully registered? like you can see them on the flyte ui?
c
yes, everything except the master workflow. it takes about 27 minutes to assemble, and the build script is higher. I have an airflow running the same way now, it has one master dag to run the others, using trigger.
all the images above, it's all about the master workflow
2 Views
Open in Slack
PreviousNext
Powered by