jolly-florist-66728
11/27/2024, 7:58 PMmyproject.models.alabama
, myproject.models.california
, myproject.models.texas
. Each one has the same API.
How would you all create tasks/workflows for each of these models, without having to copy/paste the same code for each model?
I've tried things like making a function/task factory, but Flyte keeps telling me ValueError: TaskFunction cannot be a nested/inner or local function.
. Using importlib to import-by-string works okay, but then pyflyte run --remote
doesn't package the model's code and its dependencies when submitting the workflow to the remote.
# Answer
It seems likely that you can address this issue by organizing your code into a shared library and using Flyte's ImageSpec
to manage dependencies. According to Flyte GitHub Issue #5492, using pyflyte run --remote
requires ensuring that all source files are present in the container running the workflow. You can achieve this by using the --copy-all
flag to include all necessary modules. Alternatively, you can structure your project to use a setup.py
file and install the package during execution.
Would you like more details on how to set up your project structure or manage dependencies with ImageSpec
?
Sources:
- Flyte GitHub Issue #5492
Slack Link
https://flyte.slack.com/archives/CP2HDHKE1/p1732737489.392219 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.