Hi everyone! Is there any plan to add a way to define task dependencies just using a requiriment.txt file or dependency name, as it is in Airflow's PythonVirtualenvOperator?

Hey, I asked about that before and the suggestion was to contribute a plugin. Meanwhile what I'm doing is using pip from within my python script, which works fine for most packages without restarting.

how would you like to do this @Lucas Rolim

Hey @Yee! I imagine something pretty similar to what we have on BentoML @env annotation (https://docs.bentoml.org/en/0.13-lts/concepts.html#defining-service-environmentdefining-service-environment) or what we have on Aiflow virtual env operator (https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator)

yeah that’s kinda what we were thinking. • Not the direct dependencies in the task decorator, but a requirements file instead. If you were to list out the dependencies you’d have to change them on all the tasks. • You could have the dependencies in the workflow decorator instead, but that makes it tricky in those instances where a workflow has tasks that use different images - maybe you want diffferent dependencies in different images (a spark one and a pytorch one) • We have the Image/ImageConfig object already. pip requirements, or more generally a last minute init script of some kind, could belong there. the benefit of a listing of requirements is that that the system can make sense of it (maybe cache an image in the future, when we have image building support). Images are currently listed in config files and on the command line to pyflyte… how would we introduce these more complicated settings on top of that? • in general, i like the idea of separating runtime environments from code, but requirements is i guess one of those places where you can’t really separate things out. By the way, there’s now a tag in the Flyte issues now for interactive experience. This stuff is peripherally related to that so feel free to peruse/add issues/comment on existing issues there. (for example, in an interactive type experience, we’re thinking of relying on cloudpickle. cloudpickle is finicky though, and you really have to make sure you’re working with the same version at pickle and unpickle time.)

@Lucas Rolim i am interested in this, would you be open to filing an issue

Opened it https://github.com/flyteorg/flyte/issues/2573 Lmk if i should edit something

Thank you Robin