Hi, I have a wrapper workflow that has certain sub workflows which is comprised of tasks (ml and spark) in a multi image setup model. Now I am trying to invoke this wrapper workflow from a dynamic workflow based on certain conditions. But it fails on say for eg:

ModuleNotFoundError: No module named 'matplotlib'

while importing libraries in tasks that requires a tensorflow image (because the default image is spark). And if I pass the tensorflow image as

container_image

on dynamic, then it fails on

ModuleNotFoundError: No module named 'flytekitplugins'

in tasks that requires spark image, basically the reverse. Is there a way I could make this work in a multi image setup ?

could you post a skeleton example of the task decorators/signatures?

Yes it fails at the dynamic level

where are you importing the libraries? in the tasks?

Yes libraries are imported at the task level…this is how the flow is

import main_wrapper_workflow

@dynamic
def dynamic_wrapper_workflow(dates: List[str]):
 for date in dates:
  main_wrapper_workflow(date=date)
 return True

import sub_wf_1, sub_wf_2

@workflow
def main_wrapper_workflow(date: str):
 x = sub_wf_1(date=date)
 y = sub_wf_2(date=date, input=x)

import spark_task

@workflow
def sub_wf_1(date: str):
 a = spark_task(date=date)

import ml_task
@workflow
def sub_wf_2(date: str, input: str):
 a = ml_task(date=date, input=input)

from flytekitplugins.spark import Spark

@task(container_image=spark_image)
def spark_task()

import matplotlib.pyplot as plt
@task(container_image=ml_image)
def ml_task()

This works when we directly invoke the wrapper workflow. I assume this is happening because I am wrapping the wrapper workflow with

dynamic

understood. can you import the libraries within your task?

@task(...)
def spark_task():
    from flytekitplugins.spark import Spark

    ...

make sure you only have libraries available in the default container image of the dynamic workflow at the top-level.

We use Dockerfiles so I will try importing the libraries inside the task and see. Will let you know the results here gratitude thank you