Hi folks I m looking to build an index of workflow outputs b Flyte #ask-the-community

Hi folks - I'm looking to build an index of workfl...

Ethan Brown

12/06/2023, 3:39 PM

Hi folks - I'm looking to build an index of workflow outputs, but have a couple of questions to resolve. I need a way of indicating what outputs those are exactly. • By convention,

end_node

is possible.. but it's not flexible enough. • Anything in the type system should be marked as indexable, so using a specific custom type won't fit. • I see that LiteralType has both

metadata

and

annotations

fields. Would either be appropriate? I see that

Annotated

is used to generate custom hash keys for cached outputs, so I thought something like this might work more generically:

Copy code

@workflow
def training_workflow() -> Annotated[LogisticRegression, "IndexMe"]:

However, looking through my S3 bucket protobuf files I don't see anything there. Thanks!

Ethan Brown

12/06/2023, 3:42 PM

Follow-on question, assuming what I'm interested in is possible: Is there a pluggable way to extend Flyte to index these files as they're generated / written? Rather than setting up a batch job to process outputs / index them after they've been written to blob storage, it would be great to be able to hook into Flyte to do it in real time. Any thoughts on the right way to do this?

Samhita Alla

12/07/2023, 10:55 AM

@Ketan (kumare3) would you mind providing your input here?

Ethan Brown

12/14/2023, 6:40 PM

@Ketan (kumare3) I just stumbled on https://github.com/unionai-oss/artifact-demo and it looks like you're building out a first class

Artifact

concept already which might do the things I'm looking for. I also see the PR at https://github.com/flyteorg/flyte/pull/4077. Anything further to disclose here yet on direction / roadmap?

Ketan (kumare3)

12/15/2023, 5:02 AM

@Ethan Brown yes we are building it out

Ketan (kumare3)

12/15/2023, 5:03 AM

this will be ready early next year

Ketan (kumare3)

12/15/2023, 5:03 AM

there is a working group in this slack too

Ketan (kumare3)

12/15/2023, 5:04 AM

also, what you need is also available as an event stream, but artifacts takes it to the next level

Ketan (kumare3)

12/15/2023, 5:04 AM

all built in and working cohesively

Ethan Brown

12/15/2023, 9:56 PM

Ah, that's great / exciting news! I will be following this space pretty closely then -- it looks like what you have in mind is pretty well aligned with what I was thinking.

Open in Slack

Previous Next