https://flyte.org logo
Title
s

Sohan Shah

04/01/2022, 6:03 PM
hey i have a request that would serve the Forecasting platform we’re building at Spotify on top of Flyte. We currently subscribe to a topic containing execution events to populate our UI with visibility on what forecasts have been run/where their outputs are located etc. That’s worked quite well! What Im wondering if a similar topic exists to capture the *Workflow* information created at registration time For context, our platform works as such: 1. A model developer registers to Flyte a workflow that represents a forecast model that takes in several assumptions as CSVs at runtime 2. A separate financial analyst writes those assumptions and uploads them to our platform 3. the model can be launched from the platform with the appropriate assumptions by anyone My thought is that if we have an understanding of workflow dependencies at registration time, we can have the platform enforce at assumption upload time that all the correct assumptions are uploaded for each model (with the correct schemas) according to the Flyte version selected. Im sure there are other ways to do this (create a hook at registration time that queries the API for all workflows belonging to the registered version), but through a topic is ideal for us so just want to know feasibility here. Thanks!
v

Varun Kulkarni

04/01/2022, 6:41 PM
following
k

Ketan (kumare3)

04/01/2022, 10:45 PM
can you explain a little more. do you mean, you want an event for registration?
s

Sohan Shah

04/01/2022, 11:01 PM
yeah exactly, right after i register a version im able to query Flyte to inspect a workflow associated to that version. For example:
<https://obol.spotify.net/api/v1/workflows/sp-one-model/development/onemodel.models.premium_revenue._flyte.premium_revenue_onemodel/sohans-sohanv12>
The contents of this message allow us to make explicit on our platform what the inputs feeding into the _premium_revenue_onemodel_ workflow on the sohans-sohanv12 version should look like. Was wondering the lift to have this data written to a topic as it’s created (at registration time), similar to how execution events are written right away
cc @Sharon Gong
k

Ketan (kumare3)

04/04/2022, 6:05 PM
so there is no hook today at the point of registration
and tbh, that was never in the original design as well
so the lift might be a decent amount of engineering work
I think since its just workflows, polling is not a bad idea for now
s

Sohan Shah

04/04/2022, 7:26 PM
ah got it, yeah will do it by polling for now. There is a ton of value in understanding registration time lineage for us though so def interested if it ever becomes something that makes it to your roadmap
k

Ketan (kumare3)

04/04/2022, 9:41 PM
ohh
do you want to file an issue
we would love to completely understand the use case
and ofcourse open to collaborating on this
s

Sohan Shah

04/04/2022, 11:17 PM
yeah for sure 🙂 what’s best place to file issue?
k

Ketan (kumare3)

04/04/2022, 11:17 PM
Haha
s

Sohan Shah

04/04/2022, 11:19 PM
lol duh🤦. will do!
k

Ketan (kumare3)

04/08/2022, 2:31 PM
Thank you