O3.7.2 Implement ETKI for high-dimensional output scaling benefits

[eviatarbach]

Tasks

Preview Give feedback

1. Implement and test ensemble transform Kalman inversion (ETKI) #324
   +0
   
2. Examples and documentation for ETKI #330
   +0
   

Probably won't be an issue in the near future, but I believe that EKI as currently implemented is O(p^3), where p is the observation dimension. However, based on the idea of ensemble square-root Kalman filters, this can be reduced to O(p). I think that some of the EKI variants implemented in Huang et al. (2022) have this property.

[eviatarbach]

It seems that this issue has come up for ParameterEstimocean: CliMA/ParameterEstimocean.jl#84

[odunbar]

I was unaware they had been encountering this issue...

Indeed most of these filters etc can avoid building the covariance matrices, although if I recall the scaling might O(p * (m^2)) if we including the ensemble size m?

In any case, I will create a milestone towards this as it is something that has been encountered! Thanks for drawing our attention to this

[eviatarbach]

Yes, I wasn't including the scaling of m. Here is the table of the time complexities of different square-root ensemble filter variants, from Tippett et al. (2003):

Here m is ensemble size, p is observation dimension, and n is the state-space dimension (parameter dimension in our case).

[odunbar]

Chapter 8.5 & 8.9 of "Data Assimilation Fundamentals" Evensen, Vossepoel, Jan Van Leeuwen Might be useful for the linear algebra

[odunbar]

@costachris this thread will track the high-dim output EKI