Conformance + Load testing infrastructure

[phbnf]

We need to both have a conformance and load test for the SCTFE binary.

Today, we have:

• the certificate-transparency-go/.../ct_hammer tool. It allows to send generate and send add-chain and add-pre-chain requests, which work with the SCTFE with no modification. However, the read paths are not compatible, so it does not work as a conformance tool.
• the trillian-tessera/.../hammer tool, which we use for conformance testing and load testing Tessera. To works with CT, it needs to:
  1. support different add endpoints: add-chain and add-prechain instead
  2. support different read endpoints for leaves, which are stored under /tile/data instead of /tile/entries
  3. parse leaf bundles in a different way, since they are sequences of TileLeaf, and instead of sequences of big-endian uint16 length-prefixed entries
  4. support static-ct api checkpoints, instead of classic checkpoints
  5. support generating certificates chains instead of generating regular entries. Note that these chains need to also be compatible with the nice deduplication tests that the hammer supports. That might be a bit of a challenge for CT given that certs generation might not be idempotent for various reasons.

[phbnf]

A few weeks ago, I wrote two experiment PRs, that handled i, ii, iii, iv and v.

Changes for i, ii, iii, and iv are not that big, but may require further rearchitecturing shall we decide for the hammer tool to optionally supports CT with flags for instance.

I did something a bit special for v. that I think is out of scope of this issue (though it might be a nice-to-have): instead of generating chains locally, it plugged the certificate-transparency-go/.../scanlog tool into trillian-tessera/.../hammer. Since a lot of things have changed on the hammer since I wrote this, and especially that would impact point v. with respect to deduplication, I think that it won't work anymore. But I'll give it a go when I come around to it. For history and inspiration, here are the two PRs that resulted from this, which I have rebased on main/master.

• trillian-tessera/.../hammer changes
• certificate-transparency-go/.../scanlog changes

[phbnf]

I spoke with Al and Roger today - let's start by updating certificate-transparency-go/.../scanlog changes to make sure that we implement v. appropriately and that it works well with the hammer. That will give us a fork of the hammer that supports CT, and we can take it from there.