Anyone know a good way to do non-grid (eg. some kind of bayesian) hyperparameter optimization with Flyte, with multiple trials in parallel? (ie. is there some library that makes this easy or do I have to implement most of the optimization stuff myself? eg. a library which spits out parameters to try and I give it back the results would be pretty easy to use with Flyte, rather than the library calling an objective function like hyperopt does)

hi @Robin Kahlow there’s currently no canonical way of doing this, although I believe it’s technically possible with Flyte +

<some bayes opt library>

. One would have to use dynamic workflows to collect results and feed it back to the bayesopt sampler for subsequent trials. Do you have a bayesopt library in mind?

i dont have a specific one in mind, i used hyperopt before though is there a way to limit concurrency with dynamic? // i guess that automatically happens since the sampling is sequential

Using BayesianOptimization (basically using the suggest-evaluate-register loop in the advanced guide) something like this might work.

from bayes_opt import BayesianOptimization, UtilityFunction

@task
def black_box_function(points: Dict):
    ...  # inner training loop here

@task
def suggest_points(
    optimizer: BayesianOptimization,
    utility: UtilityFunction,
    concurrency: int,
) -> List[Dict]:
    return [optimizer.suggest(utility) for _ in range(concurrency)]

@task
def register_targets(
    optimizer: BayesianOptimization,
    points: List[Dict],
    targets: List[float],
) -> BayesianOptimization:
    for point, target in zip(points, targets):
        optimizer.register(params=point, target=target)
    return optimizer

@dynamic
def concurrent_trials(points: dict) -> List[float]:
    targets = []
    for _ in points:
        targets.append(black_box_function(**points))
    return targets

@dynamic
def bayesopt(n_iter: int = 5, concurrency: int = 3) -> Dict:
    optimizer = BayesianOptimization(...)
    utility = UtilityFunction(kind="ucb", kappa=2.5, xi=0.0)
    for _ in range(n_iter):
        points = suggest_points(optimizer=optimizer, utility=utility, concurrency=concurrency)
        targets = concurrent_trials(points=points)
        optimizer = register_targets(optimizer, points=points, targets=targets)
    # return point that maximized the target
    return optimizer.max

oh great, that's really helpful Niels thank you 🙂

also, I’m not entirely sure whether the

optimizer = register_targets(optimizer, points=points, targets=targets)

line in

bayesopt

dynamic will work as intended… I do believe this will unroll the dynamic graph correctly, but will have to confirm in practice

ya I'll try it out!

great ! please let me know if this works, would love to work in a canonical example in our tutorials.

cool, got it working too!

great @Robin Kahlow! let me know if you have any other questions on this front… would love to know how this works out for your use case

yea will do! one small issue right now: if the trainings take different amounts of time, we're always waiting for all of them to complete (vs there always being N workers up that just fetch more work when theyre done)

Right, that’s definitely a limitation of this approach.

there always being N workers up that just fetch more work when theyre done

The pure Flyte execution model won’t allow for this, hence the integrations with Spark (and Ray, [coming soon]). With those, you can just wrap everything in a single

@task

and use the underlying Spark/Ray cluster to distribute the computation, while having access to all of the state in the hyperopt routine.

if the trainings take different amounts of time

What’s the min, max, and mean runtime of each trial in your case? i.e. are they in the order of minutes, hours, days, (or 😖 weeks)? The benefit of the pure Flyte approach is all trials are subject to Flyte’s data lineage tracking, cache-able with (

@task(cache=True, …)

) and recoverable under the Flyte system.

yes

@eager

when we build it should allow for this. @Sebastian Schulze is actually working on a prototype in his company

Hi, I'd be happy to have a chat about this. We adopted an approach in which a central "master"-task starts, monitors and terminates trial-workflows as necessary. We use optuna as our hyperopt library but I imagine other choices would work just as well.

@Niels Bantilan the model i was training right now takes about an hour (plus minus a couple minutes), i fixed the number of epochs (keeping only the best tho) so they took all around the same time, but making number of epochs another hyperparameter would be nice, and some of our other models take days to train the caching is indeed very cool, i liked that i could train for 5 hyperparameter optimization iterations, then come back later and see how it did to maybe do more iterations while keeping the progress of the first 5 eager sounds interesting, is that already on a branch somewhere?

Awesome @Sebastian Schulze! Will you be free some time next week (Tue or after)? Would love to learn the approach you describe. @Robin Kahlow would you be interested in joining? We don’t have an implementation of

eager

yet, but it sounds like @Sebastian Schulze’s solution is an early shot at something like it.

yea sure! although I probably won't be working much on this, have a lot of other stuff on my plate that has higher prio unfortunately

@Sebastian Schulze @Robin Kahlow what time zones are y’all at? Does Tue 7/5 11AM EST work for you?

works for me unless something at my job comes up

that should work for me as well

im in the UK, but probably closer to being on US-east timezone 😄

great! just sent invite

Nice

hey @Sebastian Schulze friendly ping: https://meet.google.com/tne-dhjf-nfs