IPOP and BIPOP restart mechanisms

doc/function_types.md

@@ -358,7 +358,10 @@ Each of the implemented methods is allowed to have additional cv-modifiers
 
 The following optimizers can be used with arbitrary separable functions:
 
+ - [ActiveCMAES](#activecmaes)
+ - [BIPOP_CMAES](#bipopcmaes)
  - [CMAES](#cmaes)
+ - [IPOP_CMAES](#ipopcmaes)
 
 Each of these optimizers has an `Optimize()` function that is called as
 `Optimize(f, x)` where `f` is the function to be optimized and `x` holds the

doc/optimizers.md

@@ -86,7 +86,9 @@ approxOptimizer.Optimize(f, coordinates);
 
 #### See also:
 
+ * [BIPOP_CMAES](#bipopcmaes)
  * [CMAES](#cmaes)
+ * [IPOP_CMAES](#ipopcmaes)
  * [Improving Evolution Strategies through Active Covariance Matrix Adaptation](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.114.4239&rep=rep1&type=pdfn)
  * [Evolution strategy in Wikipedia](https://en.wikipedia.org/wiki/Evolution_strategy)
 
@@ -869,6 +871,72 @@ optimizer2.Optimize(f, coordinates);
  * [SGD in Wikipedia](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)
  * [SGD](#standard-sgd)
 
+## BIPOP_CMAES
+
+*An optimizer for [separable functions](#separable-functions).*
+
+BIPOP CMA-ES (Bi-Population CMA-ES) extends the idea of IPOP CMA-ES by using
+two intertwined restart strategies: one with an increasing population size and
+another maintaining a smaller, variable population size. This strategy allows
+BIPOP CMA-ES to adaptively balance exploration and exploitation across the
+fitness landscape, and can outperform IPOP. The larger population restarts aim
+to explore broadly, improving global search capabilities, while the smaller
+populations intensify the search in promising regions.
+
+### Constructors
+
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>()`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy`_`)`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize`_`)`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy`_`)`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize`_`)`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts`_`)`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts, populationFactor`_`)`
+* `BIPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts, populationFactor, maxFunctionEvaluations`_`)`
+
+### Attributes
+
+| **type** | **name** | **description** | **default** |
+|----------|----------|-----------------|-------------|
+| `size_t` | **`lambda`** | The population size (0 uses a default size). | `0` |
+| `TransformationPolicyType` | **`transformationPolicy`** | Instantiated transformation policy used to map the coordinates to the desired domain. | `TransformationPolicyType()` |
+| `size_t` | **`batchSize`** | Batch size to use for the objective calculation. | `32` |
+| `size_t` | **`maxIterations`** | Maximum number of iterations. | `1000` |
+| `double` | **`tolerance`** | Maximum absolute tolerance to terminate the algorithm. | `1e-5` |
+| `SelectionPolicyType` | **`selectionPolicy`** | Instantiated selection policy used to calculate the objective. | `SelectionPolicyType()` |
+| `double` | **`stepSize`** | Initial step size. | `0` |
+| `size_t` | **`maxRestarts`** | Maximum number of restarts. | `9` |
+| `double` | **`populationFactor`** | The factor by which the population increases after each restart. | `2.0` |
+| `size_t` | **`maxFunctionEvaluations`** | Maximum number of function evaluations. | `1e9` |
+
+The _`SelectionPolicyType`_ and _`TransformationPolicyType`_ template parameters are used to control how the selection of points and the transformation of the decision variables are handled during the optimization process. For more information, check the [CMAES](#cmaes) class. The `lambda`, `transformationPolicy`, `batchSize`, `maxIterations`, `tolerance`, `selectionPolicy`, and `stepSize` are all parameters that affect the underlying CMAES process.
+
+#### Examples:
+
+<details open>
+<summary>Click to collapse/expand example code.
+</summary>
+
+```c++
+RosenbrockFunction f;
+arma::mat coordinates = f.GetInitialPoint();
+
+// IPOP Active CMA-ES
+BoundaryBoxConstraint<> b(-1, 1);
+BIPOP_CMAES optimizer(0, b, 32, 200, 1e-4, 5, 2, 9, 1e4);
+Optimizer.Optimize(f, coordinates);
+```
+
+</details>
+
+#### See also:
+
+ * [CMAES](#cmaes)
+ * [ActiveCMAES](#activecmaes)
+ * [IPOP_CMAES](#ipopcmaes)
+ * [Benchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed](https://dl.acm.org/doi/pdf/10.1145/1570256.1570333)
+ * [Evolution strategy in Wikipedia](https://en.wikipedia.org/wiki/Evolution_strategy)
+
 ## Coordinate Descent (CD)
 
 *An optimizer for [partially differentiable functions](#partially-differentiable-functions).*
@@ -1029,6 +1097,9 @@ approxOptimizer.Optimize(f, coordinates);
 
 #### See also:
 
+ * [ActiveCMAES](#activecmaes)
+ * [BIPOP_CMAES](#bipopcmaes)
+ * [IPOP_CMAES](#ipopcmaes)
  * [Completely Derandomized Self-Adaptation in Evolution Strategies](http://www.cmap.polytechnique.fr/~nikolaus.hansen/cmaartic.pdf)
  * [CMA-ES in Wikipedia](https://en.wikipedia.org/wiki/CMA-ES)
  * [Evolution strategy in Wikipedia](https://en.wikipedia.org/wiki/Evolution_strategy)
@@ -1575,6 +1646,73 @@ optimizer.Optimize(f, coordinates);
  * [HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent](https://arxiv.org/abs/1106.5730)
  * [Sparse differentiable separable functions](#sparse-differentiable-separable-functions)
 
+## IPOP_CMAES
+
+*An optimizer for [separable functions](#separable-functions).*
+
+IPOP CMA-ES (Increasing Population Size CMA-ES) is an extension of the
+Covariance Matrix Adaptation Evolution Strategy (CMA-ES). It introduces a 
+restart mechanism that progressively increases the population size. This 
+approach is beneficial for optimizing multi-modal functions,
+characterized by numerous local optima. The restart mechanism is designed to 
+improve the adaptability of CMA-ES by improving the likelihood of escaping
+local optima, thus increasing the chances of discovering the global optimum.
+
+### Constructors
+
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>()`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy`_`)`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize`_`)`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy`_`)`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize`_`)`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts`_`)`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts, populationFactor`_`)`
+* `IPOP_CMAES<`_`SelectionPolicyType, TransformationPolicyType`_`>(`_`lambda, transformationPolicy, batchSize, maxIterations, tolerance, selectionPolicy, stepSize, maxRestarts, populationFactor, maxFunctionEvaluations`_`)`
+
+### Attributes
+
+| **type** | **name** | **description** | **default** |
+|----------|----------|-----------------|-------------|
+| `size_t` | **`lambda`** | The population size (0 uses a default size). | `0` |
+| `TransformationPolicyType` | **`transformationPolicy`** | Instantiated transformation policy used to map the coordinates to the desired domain. | `TransformationPolicyType()` |
+| `size_t` | **`batchSize`** | Batch size to use for the objective calculation. | `32` |
+| `size_t` | **`maxIterations`** | Maximum number of iterations. | `1000` |
+| `double` | **`tolerance`** | Maximum absolute tolerance to terminate the algorithm. | `1e-5` |
+| `SelectionPolicyType` | **`selectionPolicy`** | Instantiated selection policy used to calculate the objective. | `SelectionPolicyType()` |
+| `double` | **`stepSize`** | Initial step size. | `0` |
+| `size_t` | **`maxRestarts`** | Maximum number of restarts. | `9` |
+| `double` | **`populationFactor`** | The factor by which the population increases after each restart. | `2.0` |
+| `size_t` | **`maxFunctionEvaluations`** | Maximum number of function evaluations. | `1e9` |
+
+The _`SelectionPolicyType`_ and _`TransformationPolicyType`_ template parameters are used to control how the selection of points and the transformation of the decision variables are handled during the optimization process. For more information, check the [CMAES](#cmaes) class. The `lambda`, `transformationPolicy`, `batchSize`, `maxIterations`, `tolerance`, `selectionPolicy`, and `stepSize` are all parameters that affect the underlying CMAES process.
+
+#### Examples:
+
+<details open>
+<summary>Click to collapse/expand example code.
+</summary>
+
+```c++
+RosenbrockFunction f;
+arma::mat coordinates = f.GetInitialPoint();
+
+// IPOP Active CMA-ES
+BoundaryBoxConstraint<> b(-1, 1);
+IPOP_CMAES optimizer(0, b, 32, 200, 1e-4, 5, 2, 9, 1e4);
+Optimizer.Optimize(f, coordinates);
+```
+
+</details>
+
+#### See also:
+
+ * [CMAES](#cmaes)
+ * [ActiveCMAES](#activecmaes)
+ * [BIPOP_CMAES](#bipopcmaes)
+ * [A Restart CMA Evolution Strategy With Increasing Population Size](http://www.cmap.polytechnique.fr/~nikolaus.hansen/cec2005ipopcmaes.pdf)
+ * [Evolution strategy in Wikipedia](https://en.wikipedia.org/wiki/Evolution_strategy)
+
+
 ## IQN
 
 *An optimizer for [differentiable separable functions](#differentiable-separable-functions).*

include/ensmallen.hpp

@@ -104,6 +104,7 @@
 #include "ensmallen_bits/bigbatch_sgd/bigbatch_sgd.hpp"
 #include "ensmallen_bits/cmaes/cmaes.hpp"
 #include "ensmallen_bits/cmaes/active_cmaes.hpp"
+#include "ensmallen_bits/cmaes/pop_cmaes.hpp"
 #include "ensmallen_bits/cd/cd.hpp"
 #include "ensmallen_bits/cne/cne.hpp"
 #include "ensmallen_bits/de/de.hpp"

include/ensmallen_bits/cmaes/cmaes.hpp

@@ -172,6 +172,15 @@ class CMAES
   double& StepSize()
   { return stepSize; }
 
+  //! Get the total number of function evaluations.
+  size_t FunctionEvaluations() const 
+  { return functionEvaluations; }
+
+  //! Type of Selection Policy.
+  typedef SelectionPolicyType selectionPolicyType;
+  //! Type of Transformation Policy.
+  typedef TransformationPolicyType transformationPolicyType;
+
  private:
   //! Population size.
   size_t lambda;
@@ -195,6 +204,9 @@ class CMAES
 
   //! The step size.
   double stepSize;
+
+  //! Counter for the number of function evaluations.
+  size_t functionEvaluations = 0;
 };
 
 /**

include/ensmallen_bits/cmaes/cmaes_impl.hpp

@@ -151,6 +151,7 @@ typename MatType::elem_type CMAES<SelectionPolicyType,
     terminate |= Callback::Evaluate(*this, function, transformedIterate,
         objective, callbacks...);
   }
+  functionEvaluations += numFunctions;
 
   ElemType overallObjective = currentObjective;
   ElemType lastObjective = std::numeric_limits<ElemType>::max();
@@ -235,6 +236,8 @@ typename MatType::elem_type CMAES<SelectionPolicyType,
         transformationPolicy.Transform(mPosition[idx1]), terminate,
         callbacks...);
 
+    functionEvaluations += lambda; 
+
     // Update best parameters.
     if (currentObjective < overallObjective)
     {