Using optimize_acqf

[andrelowky]

Hi everyone, I have 2 questions in my implementation of qNEHVI for optimization.

1. My understanding of optimize_acqf's hyperparameters is that raw_samples determines how many possible solutions there are, and num_restarts determines how many starting solutions to work with. I've tried varying these 2 hyperparameters on the MW7 problem (dim=8), with no clear trend. I am also unsure how does the batchlimit and maxiter arguments affect the process.

2. I have an alternate workflow where I pre-generate the candidate solutions via draw_sobol_samples first, and then directly feed that into the defined qNEHVI acquisition function as such:

candidates = draw_sobol_samples(bounds=standard_bounds, n=raw_samples, q=1).squeeze(1)
acq_value_list = []

for i in range(0, candidates.shape[0]):
    with torch.no_grad():
        acq_value = acq_func(candidates[i].unsqueeze(dim=0))
        acq_value_list.append(acq_value.item())

sorted_x = candidates.cpu().numpy()[np.argsort(acq_value_list)]

This appears to give me very unclear performance, as it strongly outperforms optimize_acqf for MW7 (dim=2) but poorly for MW7 (dim=8). Additionally, I am getting significantly shorter run times compared to optimize_acqf.

Hope this discussion is relevant to the BoTorch community!

[Balandat]

The default optimization procedure isn't exactly straightforward, let me try to explain.

My understanding of optimize_acqf's hyperparameters is that raw_samples determines how many possible solutions there are, and num_restarts determines how many starting solutions to work with.

I am not sure what "how many possible solutions there are" means. We do gradient-based optimization and optimize over the continuous domain (assuming your domain is continuous), so there are potentially infinitely many solutions (well not in floating point precision but you get the idea). In particular we don't evaluate the acquisition function on a discrete set. Here is the basic idea:

1. We draw raw_samples Sobol samples from the domain: X_raw
2. We evaluate the acquisition function on those samples in batch: val_raw = acq(X_raw)
3. We then apply Boltzmann sampling to this - we draw num_restarts samples from the X_raw according to the weights exp(-eta * normalize(val_raw)). The idea here is to bias the samples to the good regions while still ensuring we're drawing samples across the whole domain
4. For each of these num_restarts, we then perform gradient-based optimization by means of the L-BFGS-B optimizer.

I am also unsure how does the batchlimit and maxiter arguments affect the process.
Under the hood, we group multiple of the num_restarts problems together into larger problems each containing up to batch_limit problems (by concatenating the optimization variables and summing the objectives). This allows us to exploit the fast batch evaluation of the acquisition functions due to pytorch's hardware acceleration. The smaller batch_limit the more individual problems we need to solve, but their quality typically improves as compared to the grouped ones. maxiter is the maximum number of iterations for each L-BFGS-B optimization. Ideally you wouldn't have to play around with this much; the algorithm should hopefully be able to converge quickly and compute a stopping condition.

I have an alternate workflow where I pre-generate the candidate solutions via draw_sobol_samples first, and then directly feed that into the defined qNEHVI acquisition function as such:

This looks like you're just evaluating the acquisition function on a discrete set. This will not scale to higher dimensions (as you observed yourself).

This appears to give me very unclear performance, as it strongly outperforms optimize_acqf for MW7 (dim=2) but poorly for MW7 (dim=8). Additionally, I am getting significantly shorter run times compared to optimize_acqf.

For dim=2 you are probably splattering enough points across the domain to get very good coverage. This isn't necessarily the worst thing to do in 1 or two dimensions, but you'd want to not loop through each candidate individually but instead batch-evaluate the acquisition function by passing in the candidates tensor as a raw_samples x 1 x d-shaped tensor, that will be way faster. However, to get equal coverage using this approach in higher dimensions, you'd need to grow the number of discrete elements exponentially with dim, which is not scalable. This is why gradient-based optimization is so powerful in higher-dimensional settings.

as it strongly outperforms optimize_acqf for MW7 (dim=2)

Hmm this is interesting, I wouldn't have expected this to outperform gradient-based optimization. My hunch is that your raw_samples and the range of acquisition values across those samples is so large that the Boltzmann sampling step results in a weight distribution that makes it exceedingly likely to draw samples clustered together rather than providing good coverage of the whole domain. @SebastianAment has observed some similar behaviors in the past. This seems to be a particularly interesting failure mode here.

[andrelowky]

Thank you so much for the prompt response!

If I am understanding you correctly: you still have to evaluate the acq_func on a discrete set of raw_samples first, and then select only num_restart samples that are most optimal based on Boltzmann sampling. These samples are then taken as the starting points for a continous GP that approximates the acquisition function in which you perform gradient descent to eventually get the proposed candidates for true evaluation. Is this correct?

[Balandat]

Not exactly. There is always a continuous GP that the acquisition function is defined on. For the purposes of the optimization you can ignore the specifics of the model, just view the acquisition function as a deterministic function that needs to be optimized. We just first try to be smart about selecting the initial conditions from which to start gradient-based optimization from. This is called multi-start gradient descent, see e.g. section 3 of https://arxiv.org/pdf/1401.3894.pdf

[andrelowky]

Thank you for the clarification! Its a pleasure to learn from you.