https://flyte.org
Join Slack
is there a way to propogate some important output to a flyte workflow without having to propogate it...
# flyte-support
f
is there a way to propogate some important output to a flyte workflow without having to propogate it through all the functions? for example - i would like to have the wandb run url in my flyte outputs in the UI in some way, without obviously passing it around since that feels a bit hacky
maybe its some other concept but basically as the tasks are running sometimes i have something important i want to capture as output of some sort so i dont need to try dig it out of the logs
e
Is there a way that you can construct a wandb link as a pure function of pod-level metadata? https://docs.flyte.org/en/latest/user_guide/productionizing/configuring_logging_links_in_the_ui.html you can try following this (we use it for logging/grafana links)
But yeah may be a little tough at the workflow level, unless you plumb the wandb run URL in as a workflow-level input and know to go to the inputs to find it
f
No I think the url and wandb link is not really predictable since it's config driven based on project name etc .
e
Womp, yeah. You could think of it in a slightly different way and "start" the wandb run and pass a reference into your workflows/expose it as an env var on launch
So your launcher wrapper would be orchestrating 1) kick off a wandb run, 2) launch the flyte workflow
If you want to use flyte for scheduling this gets tricky and breaks down though, but if you're using something like github actions to orchestrate the cron / potentially-manual triggering then I think this would work
f
I wonder if a sort of orphan task to just get these important outputs might work. Not too bad to just grab outputs from the UI from that sort of dummy task
f
we are working on a better integration for wandb - cc @high-accountant-32689 / @flaky-parrot-42438
that will show up in the ui like a log link
🙏 1
also you dont need to pass outputs around, like for example this is completely ok
Copy code
@task
def foo1() -> str:
   ...

@task
def foo2() -> str:
   ...

@task
def foo3() -> str:
   ...

@workflow
def wf() -> str:
   a = foo1()
   b = foo2()
   c = foo3(0
   return a
f
ok so i could just have say a task to get wandb link and then just return once so not needing pass all way though - that might be a solution for me for now, cheers
47 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by