<- Outer Loop Cookbook: All recipes

# Tick-Tock Bayesian Optimization

This is a hyperparameter search algorithm that optimizes both model performance and cost. You specify a desired maximum cost, and the algorithm uses a “tick-tock” approach to find the best hyperparameters that satisfies that constraint.

Each dot indicates the outcome of an experiment for a particular set of hyperparameters. As usual with Bayesian Optimization, these results are fed into a Bayesian regression model, which is used to predict the outcome for other candidate configurations, shown in red. These candidate configurations are then tested, and this process is repeated.

The algorithm alternates between optimizing training time (“tick”) and optimizing model loss (“tock”). On each “tick” stage the algorithm searches for configurations that improve cost without hurting performance. On each “tock” stage it searches for configurations that improve model performance while satisfying the maximum training time.

In practice, this leads to one of two end results:

1. Scenario 1: Maximum model performance is reachable within budget. In this case, the loop will keep exploring downward, searching for a model and training regime that is maximally cheap while still achieving maximal model performance.
2. Scenario 2: Maximum model performance is beyond your budget (think: large AI models). In this case, each “tick” creates some potential energy for improvement, which is then consumed by the “tock”. The loop searches for the best model that fits your budget.

You are free to wrap this search in yet another outer loop, where you increase or decrease the budget over time.

It is noteworthy that optimizing for this second objective can actually decrease your total cost. If you dedicate $$n$$ percent of your experiments to optimizing cost, the whole thing pays for itself as soon as it reduces cost by $$n$$ percent. Usually, adding a second objective to your hyperparameter optimization makes it cost more, but not here.

## Python code

This code demonstrates the core idea. A couple future improvements:

• Create a followup example that adds the ability for the optimized function to return a partial result so that it can estimate the actual cost of a model without fully training it, stopping early if it’s too high. (This means there will be two GP models, one with more observations than the other.)

(Thanks to Rosanne Liu for early feedback on the visualization.)