Hey all, anyone here used the neptune.ai plugin? in my workflow all my tasks should report to the same run_id, but I dont understand how can I configure that other than putting the id manually as an argument in the init_neptune_run decorator

cc @flaky-parrot-42438

Do you want to associate a whole workflow execution with a single run in neptune?

exactly

Do you want to set a

custom_run_id

? (According to https://docs.neptune.ai/api/run/ )

custom run id as far as i understand is similar to the regular run_id, it's just allowing free text What Im looking for is a common run_id that can be shared between tasks without explicitly passing it as an argument ( because then the init_neptune_run decorator will be redundant and will create dummy runs)

I'm trying to see how one would do it manually, and then we can figure out the abstraction to make it easier.

manually it would mean that one will create a neptune run, and place that id in all of the task init_neptune_run decorators as __with_id argument_

Hmm, what happens when you use

with_id

but the run is not created yet?

I think it would raise an error

ideally the workflow will create the id

A normal workflow does not actually run any python code, so it can not generate the id. So we'll need a task to generate it in a task and then pass it around somehow. Can you open an issue on this?

Sure thing, could you send me a link? Workflows can run some code as far as I know, isnt that called flyte DSL or something? 🙂

https://github.com/flyteorg/flyte/issues Flyte runs code during registration time. Once it reaches the cluster, the workflow is statically compiled to a DAG and it does not run any python code. The workaround is to use dynamic workflow, which is a mixture between task + workflows.

I see. thanks! Will write that soon