Hello everyone! I work with Ought – we're currently evaluating a couple of frameworks to handle a data pipeline: ingesting academic papers, joining, filtering, enriching, transforming, calculating embeddings, … We're excited to have Flyte in the mix. I'm also passively looking for a replacement for Cortex.dev after they were acquired, and more general MLOps tooling – so UnionML would be something we look at if Flyte works out well for the pipeline.

Welcome @James Brady! Glad to have you here. For orchestrating ML pipelines in production, UnionML should fit the bill! @Niels Bantilan would be the best person to chat with about UnionML!

hi @James Brady, welcome! Yes let’s chat, curious to learn more about what you do at Ought

welcome @James Brady

Hi all! Just working through the User Guide now, and enjoying it.

1. for a code example on gpu-accelerated nodes, see here. Another useful class here is Resources 2. yes, maptasks are the recommended way to parallelize nodes, a more flexible alternative would be dynamic workflows (the tradeoff is that dynamics don’t have a the ability to express

concurrency

and

min_success_ratio

)

Fabulous! Thanks Niels

yes, maptasks are the recommended way to parallelize nodes, a more flexible alternative would be dynamic workflows (the tradeoff is that dynamics don’t have a the ability to express

concurrency

and

min_success_ratio

)

+1. Also, maps tasks do not spin up nodes for every instance whereas dynamic workflow does.

Map tasks are like static parallelized, they will run a new container per unit

Am I understanding this ☝️ correctly: Dynamic workflows create new nodes (like k8s nodes) for each item being mapped over? Whereas

map_task

creates a new pod per item being mapped over? (Side-note: I've not seen where I configured the podspec we need, but I'm sure I will, just haven't read the docs yet)

I think Samhita confused the terms, think about dynamic workflows as a parent graph node that spins off new child nodes that represent a new child graph. So dynamic workflows at runtime receive input and create new workflows. These have graph nodes 😊

Ah ha, I see. yeah "node" is overloaded 😬

Please help improve docs

pod plugin example: https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/pod/pod.html#sphx-glr-auto-integrations-kubernetes-pod-pod-py docs for flytekit pods: https://docs.flyte.org/projects/flytekit/en/latest/plugins/generated/flytekitplugins.pod.Pod.html#flytekitplugins-pod-pod (@Smriti Satyan @Samhita Alla we should probably update the docstrings for this class)