https://flyte.org logo
#announcements
Title
# announcements
l

Lucas Rolim

05/24/2022, 9:41 PM
Hi everyone! Is there any plan to add a way to define task dependencies just using a requiriment.txt file or dependency name, as it is in Airflow's PythonVirtualenvOperator?
r

Robin Kahlow

05/25/2022, 9:30 AM
Hey, I asked about that before and the suggestion was to contribute a plugin. Meanwhile what I'm doing is using pip from within my python script, which works fine for most packages without restarting.
👍 1
y

Yee

05/26/2022, 6:21 AM
how would you like to do this @Lucas Rolim
like what’s the ideal ux?
and this is assuming purely pip dependencies right? never an apt-get dependency
this is something we’re starting to think about actually, so if you have suggestions or want to partner on something like this, please let us know.
l

Lucas Rolim

05/27/2022, 2:11 AM
apt-get dependencies will be nice, but pip dependencies are definitely more important
What do you think?
y

Yee

05/27/2022, 4:43 AM
yeah that’s kinda what we were thinking. • Not the direct dependencies in the task decorator, but a requirements file instead. If you were to list out the dependencies you’d have to change them on all the tasks. • You could have the dependencies in the workflow decorator instead, but that makes it tricky in those instances where a workflow has tasks that use different images - maybe you want diffferent dependencies in different images (a spark one and a pytorch one) • We have the Image/ImageConfig object already. pip requirements, or more generally a last minute init script of some kind, could belong there. the benefit of a listing of requirements is that that the system can make sense of it (maybe cache an image in the future, when we have image building support). Images are currently listed in config files and on the command line to pyflyte… how would we introduce these more complicated settings on top of that? • in general, i like the idea of separating runtime environments from code, but requirements is i guess one of those places where you can’t really separate things out. By the way, there’s now a tag in the Flyte issues now for interactive experience. This stuff is peripherally related to that so feel free to peruse/add issues/comment on existing issues there. (for example, in an interactive type experience, we’re thinking of relying on cloudpickle. cloudpickle is finicky though, and you really have to make sure you’re working with the same version at pickle and unpickle time.)
k

Ketan (kumare3)

06/02/2022, 4:03 AM
@Lucas Rolim i am interested in this, would you be open to filing an issue
@Robin Kahlow you could file an issue too and tag Lucas and I?
r

Robin Kahlow

06/02/2022, 11:52 AM
Opened it https://github.com/flyteorg/flyte/issues/2573 Lmk if i should edit something
❤️ 1
k

Ketan (kumare3)

06/02/2022, 1:53 PM
Thank you Robin
33 Views