Suggestion fails because pareto groups is empty

[jsuarez5341]

Got CARBS working on a toy env. Went to throw it on something larger and it's failing after a few trials because the pareto front is empty. Any idea what would cause this?

[abefetterman]

Did you set num_random_samples in your config to 0? That's the only thing that I imagine could cause it -- we recommend setting it to at least 3 (if you are running without any parallel suggestions, set it to at least the number of dimensions you have)

[jsuarez5341]

Hmm, no, but I was editing the suggestions returned by CARBS. There's a round function for linear spaces but not for exponential ones, so I was postprocessing values to make minibatches powers of 2. Will see if I repro it in the cleaned up version

[jsuarez5341]

@abefetterman Re-opening this because I found the cause of the issue, which appears to be a real bug

When num_random_samples > 1, I was getting duplicate suggestions. This then causes group_observations to produce groups where the parameters are identical, but there are multiple different costs. But the quantile computation uses the min across all observations, not across groups. So there can be no groups for which the mean is less than the quantile.

[kywch]

Hello, I got the same error, during both suggest() and observe().

I did a quick fix and am testing it. If my runs work well, and it looks good to you, I'll make a PR. Thank you!

main...kywch:carbs:main

[b-vm]

I've got the same bug. Been running carbs for a few months with no issues, but getting this today. @kywch I will try your pr

[kywch]

Hi @b-vm can you try https://github.com/kywch/carbs/tree/py310 this branch?

Previously, I only changed get_pareto_groups_conservative but your error comes from get_pareto_groups, due to a similar cause.

BTW, resample_frequency=2 gives you the previous suggestion once every two suggestions. Is that what you intended?