Implement simultaneous CTMRG scheme and CTMRG gradients without gauge-fix differentiation

[pbrehmer]

Here I will implement the CTMRG scheme where all projector are computed simultaneously and applied from both sides to the corners at the same time. In combination with a gauge-fixed SVD, it makes it possible to differentiate a CTMRG iteration without also differentiating the gauge_fix function itself. This speeds up and stabilizes the fixed-point gradient computation.

Things still to do:

• add a flag for selecting the way how the CTMRG iteration is differentiated (either with or without differentiating gauge_fix)
• implement new index convention & use @autoopt
• implement a good way of passing a unit cell of U, S and Vs (probably with some SVDAdjoint algorithm)
• add multi-threading (since all enlarged corners and projectors can be computed simultaneously now)
• add new tests
• add safeguards for illegal setting configurations

[pbrehmer]

I tried to fix the issue we were talking about earlier, where FixedSVD does not produce an element-wise converged environment if the supplied U, S and V come from a IterSVD, however, it still doesn't work. The U's and V's are normalized in the same way as with TensorKit.SVD and the singular values and the complete decompositions match. (I added a small extra testset for this.) This is really weird and I'm not so sure where to look next.

For now, I have added a safeguard in the constructor of PEPSOptimize which forbids using IterSVD together with the :fixed mode.

Once the tests turn green (I might need to stabilize them first), I would be fine with merging. What do you think? @lkdvos

EDIT: I'll fix the tests tomorrow ;-)

[lkdvos]

I left some small comments, mostly the accessing type parameters I find important. Otherwise, there still seems to be some problems with the tests, so this might still require some stabilizing.
For the rest, I think I am happy with merging once these things are resolved!

[lkdvos]

Was there a problem with the in-place versions? In principle, this should still be compatible with AD, because of how we structured our rrules for functions that destroy their inputs.

[pbrehmer]

This should hopefully stabilize and speed up the test suite a bit. I decided to cut the p-wave superconductor from the gradients.jl test because it was very slow (especially the nothing setting which ADs everything) and also unstable. I don't think we have that example figured out properly yet, the differentiation doesn't seem to be very efficient and stable yet. Maybe I can try to make that run better in the future.

However, I modified the gradparts.jl test and actually added it to the test suite (it was just ignored previously). It now only tests for ctmrg_iter but for different CTMRG schemes (and on fermionic spaces.) That should be the most relevant setting anyway but I guess it would be good to add more functions to that test eventually.

Let me know if the modifications to the tests are fine, but I think these should run relatively stable and hopefully faster now!

[lkdvos]

I have to say that I am not super happy with cutting the pwave tests, as there now is no full test for a fermionic model. In particular, this is very much not ideal, as this is definitely one of those things where we need to continuously check if we didn't mess up any of the twists by making changes, sometimes requiring multiplication and sometimes contraction, which would now not show up at all for example in the expectation_values etc.

[pbrehmer]

I can totally see that; let me check if I can get it to run properly again...

[lkdvos]

I would be completely fine with actually loading a pre-computed near-groundstate from disk if that solves the problem

[pbrehmer]

That would be a good idea in the future, but currently I'm struggling to get converging gradients for the :simultaneous and :fixed schemes. Somehow the finite difference tests for ctmrg_iter work but the adjoint linear problem doesn't converge properly or the contangents are significantly sensitive to the gauge choice. While this occasionally comes up also with the other models with |gauge| ~ 1e-9, here for the fermionic model we have |gauge|~ 1e-2 which really shouldn't happen.

I can't really tell yet why there is such a difference between the :sequential (the old way) and :simultaneous (the new way) schemes. For now, I can disable the :simultaneous tests for the p-wave superconductor and then it should run as before.

[pbrehmer]

@lkdvos Do you have an idea where these InexactErrors in the gradparts.jl test such as Int64(99.99) come from? I don't really understand what is happening under the hood and if this is easy to fix...

Otherwise the test should finally run through !

[lkdvos]

I am not sure if we really want to include the gradpart tests. Quinten added these to debug on a bit finer level, but there is really no reason to check if the individual steps are compatible with finite differencing if a full gradient also works. (Also, they really are hacked together a bit...)

In particular, while I didn't investigate in detail, the InexactError is typically the rrule tests that attempts to generate random tangents for things like the algorithm etc, where somehow it thinks that it should generate maxiter=99.9 type things. You could avoid this by manually passing the tangent to the test_rrule function, arg \vdash NoTangent() might do the trick

[pbrehmer]

Hmm I guess you got a point, that it is a bit hacked together and that the full gradient is already tested elsewhere. I will then disable the test but it might be useful to leave it in the tests folder if needed in the future?

[codecov]

Codecov Report

Attention: Patch coverage is 95.87302% with 13 lines in your changes missing coverage. Please review.

Files	Coverage Δ
src/PEPSKit.jl	100.00% <ø> (ø)
src/algorithms/ctmrg.jl	92.98% <100.00%> (-0.68%)	⬇️
src/operators/models.jl	100.00% <100.00%> (ø)
src/utility/svd.jl	88.33% <100.00%> (+0.61%)	⬆️
src/utility/util.jl	82.55% <100.00%> (+10.28%)	⬆️
src/algorithms/ctmrg_gauge_fix.jl	98.19% <98.19%> (ø)
src/algorithms/peps_opt.jl	95.19% <95.55%> (-0.46%)	⬇️
src/utility/diffset.jl	85.18% <85.18%> (ø)
src/algorithms/ctmrg_all_sides.jl	94.11% <94.11%> (ø)

... and 3 files with indirect coverage changes

[pbrehmer]

I guess all of the requested changes are now in place :-)