Parallel evaluations

[nwbhatia]

The flow of the suggest-evaluate-register paradigm implies that one evaluation step should be performed before the optimizer can recommend a new input set. Could bayesian optimization benefit from running multiple evaluation steps in parallel, and could this be implemented using this library?

[fmfn]

It absolutely could. In fact, there's a fully functional example in the examples folder of it doing it just that, check it out: async bayesian optimization.

[nwbhatia]

Sorry for not being clear, I meant to ask about a more simple case without need for parallel machines etc. Rather, I was wondering if the suggest-evaluate-register paradigm could support evaluating multiple "points" in one loop (for example if the black box function is a model that might need to train for a long time but multiple instances can be trained with different suggested parameters). Below is a simple example of what I'm trying to implement:

for _ in range(10):
multiple_inputs = optimizer.suggest(utility)
multiple_outputs = map(pt -> black_box_function(pt), multiple_inputs)
optimizer.register(multiple_inputs, multiple_outputs)

Given optimizer.suggest only returns one point and the examples suggest one point, evaluate that single point, then register that single output, I wasn't sure how to test multiple points in each loop iteration in a way that would actually help the optimizer get to an optimal value faster. Do you have any recommendations for how to do that either using that 4 line example from above or with a specific portion of the async file you linked?

[fmfn]

There's no point in fetching multiple suggestions at a time since, with the exception of the very first suggestion (before anything is registered), they will be exactly the same. Think about it, without registering any new observation, why would the the algorithm, which is deterministic, return a different suggestion?

Nothing, however, stops you from fetching a suggestion while a previous point is being evaluated. But this is only constructive if you register a new observation between the two fetches. Hence the example I linked to you above.

It still seems to me that what you are looking for is equivalent to the concurrent implementation I was talking about. The ability to have multiple instances of the black box function being evaluated concurrently with different suggestions is precisely that. Unfortunately writing concurrent code in Python feels clumsy, so that example is more complex than it should.

Here's another example. It isn't actually concurrent, but I think it illustrates the whole paradigm a little bit more clearly.

optimizer = BayesianOptimization(...
utility = ...
def black_box_function(params...):
    pass

class SER:
    def __init__(self, ...):
        self._last_r = None
        self._last_p = None

    def step(self, optimizer, f):
        if self._last_r is not None:
            optimizer.register(self._last_p, self._last_r) # Changes optimizer in place
        self._last_p = optimizer.suggest(utility)
        self._last_r = f(**p)

s1 = SER()
s2 = SER()
...
for _ in range(10):
    # All the method calls below could be performed concurrently
    s1.step(optimizer, black_box_function)
    s2.step(optimizer, black_box_function)
    ...

[nwbhatia]

Makes perfect sense, thank you for that example