My team has a number of models that train, organiz...
# flyte-support
m
My team has a number of models that train, organized like
myproject.models.alabama
,
myproject.models.california
,
myproject.models.texas
. Each one has the same API. How would you all create tasks/workflows for each of these models, without having to copy/paste the same code for each model? I've tried things like making a function/task factory, but Flyte keeps telling me
ValueError: TaskFunction cannot be a nested/inner or local function.
. Using importlib to import-by-string works okay, but then
pyflyte run --remote
doesn't package the model's code and its dependencies when submitting the workflow to the remote.
g
which version of flytekit you are using? we add an auto copy feature, and it should copy your code dependencies.
t
why the need for distinct tasks? can you add a string as an input/output?
m
1.13.12
Will
--copy auto
pickup modules imported by calling
importlib
inside a task run? I'm not sure it can, since those won't show up in
sys.modules
until the task is run.
My problem is that I want to make say... a single
fetch_training_data
and not
fetch_training_data_texas
and
fetch_training_data_alabama
t
but why can’t the single
fetch_training_data
task take a string input?
m
It certainly can, I'm just struggling to find a way to use that string that works well with both our project's organization and what flyte's expecting.