add support for cost aware acquisition function

[yubinhu]

Needs arise when using the repo for our research project to integrate cost-aware acquisition functions into the package. In this change, I included Expected Improvement Per Unit cost (EIpu), as named in Lee et al., 2020.

Please check if you want to merge this change.

[till-m]

Hi @yubinhu, thanks for your contribution.

I personally believe solid cost-aware BO is useful, though one should always carefully consider the threat of feature creep when adding new capabilities.

Moreover, I've quickly scanned the paper you linked and it seems the authors aren't particularly fond of EIpu:

In [their] experiments, EIpu demonstrated underwhelming performance: [...] out of twenty HPO problems, EIpu performs worse than EI on nine.

Considering this, I'm not sure that adding EIpu is the right move.

[yubinhu]

Hi @till-m, thanks for your comments!

The reason the authors observed underwhelming results is that EIpu is not particularly suited to the task. EIpu is best suited for situations where the cost to explore a new parameter setting is not intrinsic to the parameter setting but is related to the distance to go from one setting to another. You may think of this approximately as having the cost function $c(X_{new}, X_{old})$ instead of $c(X_{new})$.

In the paper, the authors used BO for model hyper-parameter tuning and observed that EIpu steers away from high-cost models which typically perform better. In a model hyper-parameter tuning task, higher cost parameters always have higher costs, meaning that EIpu always discourages the exploration of these points.

Consider a situation where you have a physical system and the cost of exploring some parameters is the time it takes to go from the current setting to the new one, then EIpu lets you target the highest improvements per unit of time. It does not discourage exploration of any parameters but only favors going to "closer" points first. We used EIpu for tuning a super-conducting ion source and observed better results than EI in simulations.

[bwheelz36]

Hi @yubinhu - to consider this PR further, could you please fix the code to ensure the tests pass, and also add a notebook documenting how to use this feature?

[till-m]

@yubinhu I know it's been a while, and you probably don't need this feature anymore, but I implemented it recently in the PR linked above (in a notebook called acquisition_functions.ipynb). If you have the time, I'd appreciate you having a look.