apt-get dependencies will be nice, but pip dependencies are definitely more important
What do you think?
05/27/2022, 4:43 AM
yeah that’s kinda what we were thinking.
• Not the direct dependencies in the task decorator, but a requirements file instead. If you were to list out the dependencies you’d have to change them on all the tasks.
• You could have the dependencies in the workflow decorator instead, but that makes it tricky in those instances where a workflow has tasks that use different images - maybe you want diffferent dependencies in different images (a spark one and a pytorch one)
• We have the Image/ImageConfig object already. pip requirements, or more generally a last minute init script of some kind, could belong there. the benefit of a listing of requirements is that that the system can make sense of it (maybe cache an image in the future, when we have image building support). Images are currently listed in config files and on the command line to pyflyte… how would we introduce these more complicated settings on top of that?
• in general, i like the idea of separating runtime environments from code, but requirements is i guess one of those places where you can’t really separate things out.
By the way, there’s now a tag in the Flyte issues now for interactive experience. This stuff is peripherally related to that so feel free to peruse/add issues/comment on existing issues there. (for example, in an interactive type experience, we’re thinking of relying on cloudpickle. cloudpickle is finicky though, and you really have to make sure you’re working with the same version at pickle and unpickle time.)
06/02/2022, 4:03 AM
@Lucas Rolim i am interested in this, would you be open to filing an issue
@Robin Kahlow you could file an issue too and tag Lucas and I?