Hi folks - I'm looking to build an index of workflow outputs, but have a couple of questions to reso...

gorgeous-waitress-5026

12/06/2023, 3:39 PM

Hi folks - I'm looking to build an index of workflow outputs, but have a couple of questions to resolve. I need a way of indicating what outputs those are exactly. • By convention,

end_node

is possible.. but it's not flexible enough. • Anything in the type system should be marked as indexable, so using a specific custom type won't fit. • I see that LiteralType has both

metadata

and

annotations

fields. Would either be appropriate? I see that

Annotated

is used to generate custom hash keys for cached outputs, so I thought something like this might work more generically:

Copy code

@workflow
def training_workflow() -> Annotated[LogisticRegression, "IndexMe"]:

However, looking through my S3 bucket protobuf files I don't see anything there. Thanks!

gorgeous-waitress-5026

12/06/2023, 3:42 PM

Follow-on question, assuming what I'm interested in is possible: Is there a pluggable way to extend Flyte to index these files as they're generated / written? Rather than setting up a batch job to process outputs / index them after they've been written to blob storage, it would be great to be able to hook into Flyte to do it in real time. Any thoughts on the right way to do this?

tall-lock-23197

12/07/2023, 10:55 AM

@freezing-airport-6809 would you mind providing your input here?

gorgeous-waitress-5026

12/14/2023, 6:40 PM

@freezing-airport-6809 I just stumbled on https://github.com/unionai-oss/artifact-demo and it looks like you're building out a first class

Artifact

concept already which might do the things I'm looking for. I also see the PR at https://github.com/flyteorg/flyte/pull/4077. Anything further to disclose here yet on direction / roadmap?

freezing-airport-6809

12/15/2023, 5:02 AM

@gorgeous-waitress-5026 yes we are building it out

freezing-airport-6809

12/15/2023, 5:03 AM

this will be ready early next year

freezing-airport-6809

12/15/2023, 5:03 AM

there is a working group in this slack too

freezing-airport-6809

12/15/2023, 5:04 AM

also, what you need is also available as an event stream, but artifacts takes it to the next level

freezing-airport-6809

12/15/2023, 5:04 AM

all built in and working cohesively

gorgeous-waitress-5026

12/15/2023, 9:56 PM

Ah, that's great / exciting news! I will be following this space pretty closely then -- it looks like what you have in mind is pretty well aligned with what I was thinking.

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.