Interface for ParameterEstimation.jl and Optim.jl or BlackBoxOptim.jl derivative-free optimization

[adelinehillier]

A priority for future development should be to make this package compatible with arbitrary derivative-free Optim.jl or BlackBoxOptim.jl optimization algorithms. These problems are formulated not as Bayesian inverse problems but purely in terms of the objective function that needs to be minimized. However, I think we should still use the InverseProblem, OutputMap and Tranformation constructs to set up the problem and compute the objective; there will just be the open question of what kinds of options to provide for what form the objective function should take (e.g. regularized or unregularized).

Before I implement an API, does anyone have strong opinions about what kind of flexibility should be available? For now, I think a keyword argument regularized indicating which version of the EKI objective is desired might be all we need.

[glwagner]

Can you provide a code example of what you're envisioning?

[glwagner]

Can we just extend optimize and bboptimize so that we can write

inverse_problem = InverseProblem(observations, forward_simulation, free_parameters)
objective = RegularizedObejctiveFunction()

# Optim
res = optimize(inverse_problem, options...; objective)

# BlackBoxOptim
res = bboptimize(inverse_problem; objective, kwargs...)

Using a type rather than a flag for objective will make it easier for future developers to extend the code so I suggest that.

[glwagner]

Also what about GalacticOptim.jl? If we believe what they say: https://github.com/SciML/GalacticOptim.jl then we can just implement one interface into GalacticOptim and get all optimization packages.