Add index performance script (#2042)

Add terraform configuration and scripts to set up rekor standalone on
GCP, perform a series of insert and search operations, use Prometheus to
gather metrics, and plot the results with gnuplot.

The scripts added here are for comparing mysql and redis as index
storage backends. Other types of performance measurement scripts could
be added here in the future.

To get a realistic sense of query speed for searches, a large data set
is needed. Rather than using the rekor API to insert real data, fake
data is generated and uploaded directly to the backend before searching
it.

Different types of searches are performed: searches where there should
be many results, searches where there should be few results, and
searches where there should be no results. The goal is not to compare
the latency of these different searches, but to take the overall average
to compare across backends.

Signed-off-by: Colleen Murphy <[email protected]>

scripts/performance/README.md

@@ -0,0 +1,4 @@
+Performance Scripts
+===================
+
+This directory is a collection of scripts for exercising rekor's performance.

scripts/performance/index-storage/README.md

@@ -0,0 +1,40 @@
+Rekor Performance Tester
+========================
+
+Scripts to repeatably gather performance metrics for index storage insertion and
+retrieval in rekor.
+
+Usage
+-----
+
+Use terraform to set up the services in GCP:
+
+```
+cd terraform
+terraform init
+terraform plan
+terraform apply
+```
+
+Copy or clone this repository on to the bastion VM that terraform instantiates.
+Run this script from there:
+
+```
+export INSERT_RUNS=<N> # The number of inserts to perform and measure. This doesn't need to be terribly high.
+export SEARCH_ENTRIES=<M> # The number of entries to upload to the database out of band to search against. This should be sufficiently high to represent a real database.
+export INDEX_BACKEND=<redis|mysql> # The index backend to test against
+export REGION=<region> # The GCP region where the rekor services are deployed
+./index-performance.sh
+```
+
+On the first run, `indices.csv` will be populated with fake search entries,
+which will take a while depending on how big $SEARCH_ENTRIES is. This only
+happens once as long as indices.csv is not removed.
+
+Run `index-performance.sh` against each backend. Then plot the results:
+
+```
+./plot.sh
+```
+
+Copy the resulting `graph.png` back to your local host.