-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement adaptive localization #4243
Conversation
c4d6da6
to
fe5a485
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just looking at the code it looks good.
- Might be worth it to add some comments to the functions if its not obvious whats going on.
- Is it possible to write a test for this specific improvement? ie:
correlated_parameter_response_pairs
62bd91a
to
6af26ac
Compare
624ce42
to
9ff133e
Compare
04b5241
to
a946297
Compare
6da3aa5
to
3724fd2
Compare
Codecov Report
@@ Coverage Diff @@
## main #4243 +/- ##
==========================================
+ Coverage 59.19% 59.26% +0.07%
==========================================
Files 440 440
Lines 30667 30729 +62
Branches 3135 3135
==========================================
+ Hits 18153 18212 +59
- Misses 11729 11732 +3
Partials 785 785
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
87e8eaa
to
ef0252c
Compare
b470cd7
to
a4eb36e
Compare
3f5e943
to
f164944
Compare
74e63f6
to
1aeeb5e
Compare
14baf1a
to
0007e8e
Compare
b6f14fe
to
20cfcde
Compare
We can speed up the computations by not forming the diagonal matrices. Here's an example showing what I mean, where ensemble_size = 10
S = np.random.randn(1000, ensemble_size)
X_local = np.random.randn(1000, ensemble_size)
# Standard deviations
Sigma_Y = np.std(S, axis=1, ddof=1)
Sigma_A = np.std(X_local, axis=1, ddof=1)
# State-measurement covariance matrix
Y_prime = S - S.mean(axis=1, keepdims=True)
A = X_local - X_local.mean(axis=1, keepdims=True)
C_AY = A @ Y_prime.T / (ensemble_size - 1)
# Absolute values of the correlation matrix
c_AY = np.abs((C_AY / Sigma_Y.reshape(1, -1)) / Sigma_A.reshape(-1, 1)) alternatively, if ensemble_size = 10
S = np.random.randn(1000, ensemble_size)
X_local = np.random.randn(1000, ensemble_size)
# Estimate covariance matrix for outputs
Y_prime = S - S.mean(axis=1, keepdims=True)
C_YY = Y_prime @ Y_prime.T / (ensemble_size - 1)
Sigma_Y = np.sqrt(np.diag(C_YY))
A = X_local - X_local.mean(axis=1, keepdims=True)
C_AA = A @ A.T / (ensemble_size - 1)
Sigma_A = np.sqrt(np.diag(C_AA))
# State-measurement covariance matrix
C_AY = A @ Y_prime.T / (ensemble_size - 1)
# State-measurement correlation matrix
c_AY = np.abs((C_AY / Sigma_Y.reshape(1, -1)) / Sigma_A.reshape(-1, 1)) |
I don't see there being anything in the |
Add option of running adaptive localization that can simply be turned on and does not need any user input. Only parameters that are significantly correlated to responses will be updated. Default value of what constitutes significant correlation is calculated based on theory, but can be set by the user.
Has been merged to main. |
Resolves: #4411
Note that the current implementation is naive in that it loops through all parameters.
My understanding is that this is the most accurate but too computationally expensive in practice.
Perhaps we can use this as reference to compare more efficient methods against.
I've added two new keywords;
LOCALIZATION
andLOCALIZATION_CORRELATION_THRESHOLD
which may be set as for example:Pre review checklist
Adding labels helps the maintainers when writing release notes. This is the list of release note labels.