I am trying to run a wandb.sweep as shown in this quickstart. The idea is to run the hyperparameter sweep, on a conditional, based on a boolean flag passed to a workflow and save the best model as an artifact. The main workflow is static, which calls a dynamic sweep_setup function. The wandb.agent calls a callback function hyperparameter_sweep which is also dynamic and which calls the train_model dynamic function. Is there an example of this somewhere or any best way to do this? Because, I am facing too many errors to get this to work. The code is a bit long to share, so first wanted to check if someone has done this successfully.

Cc @Niels Bantilan I have not used w&b for this

@Ketan (kumare3) We tried out Ray for distributed hyper-parameter tuning but hit some blockers there so we had to abandon it. As we are evaluating W&B for experiment tracking, we thought of using their sweep functionality for this. Not being able to run this on Flyte is a blocker for us. Do you have any idea about the ETA for this functionality to be released? And do you have any other tools/packages/plugins for hyper-parameter tuning that work with Flyte that we can use in the mean time, till the functionality is implemented. Thanks!

We would love to know the usecase, would you have a few minutes to jump on a call? Cc @Niels Bantilan

lets talk about the use case! For simple HP-tuning cases you can use vanilla Flyte constructs with weights and biases as the logging layer

@Niels Bantilan - Thanks for the examples! The main requirements for us are ease of setting up the hyper-parameter tuning job with a simple config. If you look at the wandb sweep config, it is super simple, you can configure the method, what metric to track and the parameters. And second is being able to do this in a distributed fashion on k8s, which I believe is also achievable with your example using map_task. What we are observing is that once we set up the sweep using wandb.agent, it should be a blocking call and once all the runs are complete, it should exit the workflow close the sweep, but in none of our runs, it seems to close it, even after all the runs, the status seems to stay as "running" on wandb. We are running some more tests, we can may be connect tomorrow. Do you have a calendar where I can grab time?

@Nandakumar Raghu I do not think we can use weights and biases agent with flyte - as flyte workflows are not python

Ok, are you aware of any other plugins or integrations that make it easy to set up these jobs? If not we might need to go with defining vanilla Flyte constructs as Niels suggested.

i am sorry i want to understand what you want to do, before recommending something

Hi @Nandakumar Raghu just wanted to follow up here: feel free to find a time slot to chat: https://calendly.com/niels-unionai/30min

Hey, sorry was busy with some Q3 planning stuff. We are going to be discussing your examples and other alternatives for HP tuning. Most likely we will try to get KubeRay working on our k8s. I'll let you know how it goes.