another question, Are there any examples of repos with many workflows that have overlapping tasks, and/or combined workflows? I'm somewhat confused as to whether a python library with many tasks would be best, or if I should be registering everything and using

fetch_task

... I'm leaning towards a library because then we get type hints which I forsee as solving some input/output headaches... Edit: I take it back... I see the utility of registering repos ( disconnected python env requirements )... lack of python typing does seem sad though....

are there conflicts in dependencies? or just too many dependencies?

working with a very complex (to put it nicely) codebase... maybe wf-A requires torch=XX while wf-B requires torch=YY (except so much worse than that...)

you can use reference tasks which are kinda like fetched tasks, but those are annoying because you have to manually specify the python type signature.

yes I noticed that... not ideal : /

in the past we definitely talked about including some indication of the python type signature in the fetched task but that kinda breaks the type boundary… also no guarantee that things work (type X fetched from a task doesn’t necessarily correspond to type X in your codebase)

@Dan Farrell thank you for connecting and the questions. IMO, large software projects become complex over time. There is no one right answer. Here are the reasons we support a few things 1. Multiple registrations in single repo You want to register many workflows, tasks and manage everything in one place. But separate these across teams. This is why we also have a concept of project and domains 2. Multi image support in one workflow You can build different images and use them within a workflow. Imagespec actually makes this ever simpler 3. Reference tasks and workflows In cases you want to share tasks and workflows across teams or as a common repository, without having to add all dependencies to every project. More like RPC services

So right now

if sklearn_image_spec.is_container():

works if the types are contained to within the task that the container is used in. But what about if I want a type that is specific to that container to be a return type? what would you suggest the type-hinting be?

@Dan Farrell then the best method is to separate out the file. I had an example for this, let me ask @Samhita Alla if she turned it into a flytesnacks example?

I haven't worked on an example yet. @Ketan (kumare3) should the task be defined in the

is_container()

block?

no let me share with you again

@Ketan (kumare3) might you be able to share the example directly with me? When the 'combo' workflow is being parsed

flytekit/core/interface.py

seems to be grabbing all tasks's type hints via

type_hints = get_type_hints(fn, include_extras=True)

but this fails because my

Foo

class isn't available in the

combo-wf

environment.

@Dan Farrell sorry i am looking at your example, is this the multi container example

@Dan Farrell, could you please explain why you used a dynamic workflow? Since you want to call a bunch of workflows, you can just create a parent workflow and call all your sub-workflows from within it.

@Ketan (kumare3) yes, or having multiple subworkflows w/ separate environments + avoiding having to used registered workflows + having type hinting for development. @Samhita Alla because workflows(or subworkflows) just return an error, and dynamic workflows return a stacktrace either way, the errors were the same (name 'Foo' is not defined)

Can you do a relative import instead of an absolute import?

for the

one_off_combo...

script? I'd prefer not to, would like to allow scientists to work outside of the git-repo if they choose to (most do)

Isn't

name 'Foo' is not defined

the error?

yes

Right. In that case, the error has got to do with the imports.