Instructions to launch a test instance from a VM (#348)

* Terraform+Terragrunt+Doc to launch tessera from a VM

* better instructions

* link codelab to it

* address comments

cmd/conformance/README.md

@@ -9,6 +9,7 @@ Implementations are provided that use:
  - [A local filesystem](./posix/)
  - [MySQL](./mysql/)
  - [GCP](./gcp/)
+ - [AWS](deployment/live/aws/codelab/)
 
 Each of these personalities exposes an endpoint that accepts `POST` requests at a `/add` URL.
 The contents of any request body will be appended to the log, and the decimal index assigned to this newly _sequenced_ entry will be returned.

deployment/live/aws/codelab/README.md

@@ -0,0 +1,108 @@
+# AWS codelab deployment
+
+This codelab helps you bring a test Trillian Tessera stack on AWS,
+and to use it running a test personality server on an EC2 VM.
+The Tessera test stack will be comprised of an Aurora RDS MySQL database
+and a private S3 bucket. This codelab will also guide you to connect both
+the RDS instance and the S3 bucket to your VM.
+
+## Prerequisites
+For the remainder of this codelab, you'll need to have an AWS account,
+with a running EC2 Amazon Linux VM, and the following software installed:
+ - [golang](https://go.dev/doc/install), which we'll use to compile and
+   run the test personality on the VM
+ - [terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
+   and [terragrunt](https://terragrunt.gruntwork.io/docs/getting-started/install/)
+   in order to deploy the Trillian Tessera stack from the VM.
+ - `git` to clone the repo
+ - a terminal multiplexer of your choice for convenience
+
+## Instructions
+
+ 1. SSH to your VM
+ 1. Authenticate with a role that has sufficient access to create resources.
+    For the purpose of this codelab, and for ease of demonstration, we'll use
+    the `AdministratorAccess` role, and authenticate with `aws configure sso`.
+    DO NOT use this role to run any production software, or if there are other
+    services running on your AWS account.
+    Here's an example run:
+    ```
+    [ec2-user@ip-172-31-21-186 trillian-tessera]$ aws configure sso
+    SSO session name (Recommended): greenfield-session
+    SSO start URL [None]: https://console.aws.amazon.com/ // unless you use a custom signin console
+    SSO region [None]: us-east-1
+    SSO registration scopes [sso:account:access]:
+    Attempting to automatically open the SSO authorization page in your default browser.
+    If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
+    
+    https://device.sso.us-east-1.amazonaws.com/
+    
+    Then enter the code:
+    
+    <REDACTED>
+    There are 4 AWS accounts available to you.
+    Using the account ID <REDACTED>
+    The only role available to you is: AdministratorAccess
+    Using the role name "AdministratorAccess"
+    CLI default client Region [None]: us-east-1
+    CLI default output format [None]:
+    CLI profile name [AdministratorAccess-<REDACTED>]:
+    
+    To use this profile, specify the profile name using --profile, as shown:
+    
+    aws s3 ls --profile AdministratorAccess-<REDACTED>
+    ```
+ 1. Set these environment variables according to the ones you chose when configuring
+    your AWS profile:
+    ```
+    export AWS_REGION=us-east-1
+    export AWS_PROFILE=AdministratorAccess-<REDACTED>
+    ```
+ 1. Fetch the Tessera repo:
+    ```
+    git clone https://github.com/transparency-dev/trillian-tessera
+    ```
+ 1. From the root of the trillian-tessera repo, init terragrunt:
+    ```
+    terragrunt init --terragrunt-working-dir=deployment/live/aws/codelab/
+    ```
+ 1. Apply everything:
+    ```
+    terragrunt apply --terragrunt-working-dir=deployment/live/aws/codelab/
+    ```
+    This brings up the Terraform infrastructure (S3 bucket + DynamoDB table
+    for terraform state locking only) and the Trillian Tessera stack: an RDS
+    Aurora instance, a private S3 bucket, and connects this bucket to the
+    default VPC.
+ 1. Save the RDS instance URI and S3 bucket name for later:
+    ```
+    export LOG_RDS_DB=$(terragrunt output --terragrunt-working-dir=deployment/live/aws/codelab/ --raw log_rds_db)
+    export LOG_BUCKET=$(terragrunt output --terragrunt-working-dir=deployment/live/aws/codelab/ --raw log_bucket_id)
+    export LOG_NAME=$(terragrunt output --terragrunt-working-dir=deployment/live/aws/codelab/ --raw log_name)
+    ```
+ 1. Configure the VM and RDS instance to be able to speak to one another following
+    [these instructions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/tutorial-ec2-rds-option3.html),
+    it takes a few clicks in the UI.
+ 1. Generate the key pair used to sign and verify checkpoints:
+    ```
+    mkdir -p /home/ec2-user/tessera-keys
+    go run github.com/transparency-dev/serverless-log/cmd/generate_keys@80334bc9dc573e8f6c5b3694efad6358da50abd4 \
+              --key_name=$LOG_NAME \
+              --out_priv=/home/ec2-user/tessera-keys/$LOG_NAME.sec \
+              --out_pub=/home/ec2-user/tessera-keys/$LOG_NAME.pub
+    ```
+ 1. Run the conformance binary in `trillian-tessera/cmd/conformance/aws`.
+    This binary is a small personality that accepts `add/` requests,
+    and stores the data in the Trillian Tessera infrastructure you've
+    just brought up:
+    ```
+    go run ./cmd/conformance/aws --bucket=$LOG_BUCKET --db_user=root --db_password=password --db_name=tessera --db_host=$LOG_RDS_DB --signer=$(cat /home/ec2-user/tessera-keys/$LOG_NAME.sec)  -v=3
+    ```
+ 1. Congratulations, you've now successfully brought up a Trillian Tessera
+    stack on AWS, and started a personality server that can add entries to it.
+    This server accepts `add/` requests at `WRITE_URL=http://localhost:2024/`.
+    Log entries can be read directly from S3 without going through the server,
+    at `READ_URL=https://$LOG_BUCKET.s3.$AWS_REGION.amazonaws.com/`
+ 1. Head over to the [remainder of this codelab](https://github.com/transparency-dev/trillian-tessera/tree/main/cmd/conformance#codelab)
+    to add leaves to the log.
+ 

deployment/live/aws/codelab/terragrunt.hcl

@@ -0,0 +1,26 @@
+terraform {
+  source = "${get_repo_root()}/deployment/modules/aws//codelab"
+}
+
+locals {
+  region      = get_env("AWS_REGION", "us-east-1")
+  base_name   = "trillian-tessera"
+  prefix_name = "codelab-${get_aws_account_id()}"
+  ephemeral   = true
+}
+
+remote_state {
+  backend = "s3"
+
+  config = {
+    region         = local.region
+    bucket         = "${local.prefix_name}-${local.base_name}-terraform-state"
+    key            = "terraform.tfstate"
+    dynamodb_table = "${local.prefix_name}-${local.base_name}-terraform-lock"
+    s3_bucket_tags = {
+      name = "terraform_state_storage"
+    }
+  }
+}
+
+inputs = local

deployment/modules/aws/codelab/main.tf

@@ -0,0 +1,79 @@
+# Header ######################################################################
+terraform {
+  backend "s3" {}
+  required_providers {
+    aws = {
+      source = "hashicorp/aws"
+      version = "5.76.0"
+    }
+  }
+}
+
+locals {
+  name = "${var.prefix_name}-${var.base_name}"
+  port = 2024
+}
+
+provider "aws" {
+  region = var.region
+}
+
+module "storage" {
+  source = "../storage"
+
+  prefix_name = var.prefix_name
+  base_name   = var.base_name
+  region      = var.region
+  ephemeral   = true
+}
+
+# Resources ####################################################################
+## Virtual private network #####################################################
+# This will be used for the containers to communicate between themselves, and
+# the S3 bucket.
+resource "aws_default_vpc" "default" {
+   tags = {
+    Name = "Default VPC"
+  }
+}
+
+## Connect S3 bucket to VPC ####################################################
+# This allows the hammer to talk to a non public S3 bucket over HTTP.
+resource "aws_vpc_endpoint" "s3" {
+  vpc_id       = aws_default_vpc.default.id
+  service_name = "com.amazonaws.${var.region}.s3"
+}
+
+resource "aws_vpc_endpoint_route_table_association" "private_s3" {
+  vpc_endpoint_id = aws_vpc_endpoint.s3.id
+  route_table_id  = aws_default_vpc.default.default_route_table_id
+}
+
+resource "aws_s3_bucket_policy" "allow_access_from_vpce" {
+  bucket = module.storage.log_bucket.id
+  policy = data.aws_iam_policy_document.allow_access_from_vpce.json
+}
+
+data "aws_iam_policy_document" "allow_access_from_vpce" {
+  statement {
+    principals {
+      type        = "*"
+      identifiers = ["*"]
+    }
+
+    actions = [
+      "s3:GetObject",
+    ]
+
+    resources = [
+      "${module.storage.log_bucket.arn}/*",
+    ]
+
+    condition {
+     test = "StringEquals"
+     variable = "aws:sourceVpce" 
+     values = [aws_vpc_endpoint.s3.id]
+    }
+  }
+  depends_on = [aws_vpc_endpoint.s3]
+}

deployment/modules/aws/codelab/outputs.tf

@@ -0,0 +1,14 @@
+output "log_bucket_id" {
+  description = "Log S3 bucket name"
+  value       = module.storage.log_bucket.id
+}
+
+output "log_rds_db" {
+  description = "Log RDS database endpoint"
+  value       = module.storage.log_rds_db.endpoint
+}
+
+output "log_name" {
+  description = "Log name"
+  value       = local.name
+}

deployment/modules/aws/codelab/variables.tf

@@ -0,0 +1,19 @@
+variable "prefix_name" {
+  description = "Common prefix to use when naming resources, ensures unicity of the s3 bucket name."
+  type        = string
+}
+
+variable "base_name" {
+  description = "Common name to use when naming resources."
+  type        = string
+}
+
+variable "region" {
+  description = "Region in which to create resources."
+  type        = string
+}
+
+variable "ephemeral" {
+  description = "Set to true if this is a throwaway/temporary log instance. Will set attributes on created resources to allow them to be disabled/deleted more easily."
+  type = bool
+}

deployment/modules/aws/storage/main.tf

@@ -1,15 +1,3 @@
-terraform {
-  backend "s3" {}
-  required_providers {
-    aws = {
-      source = "hashicorp/aws"
-      version = "5.76.0"
-    }
-  }
-}
-
-data "aws_caller_identity" "current" {}
-
 locals {
   name = "${var.prefix_name}-${var.base_name}"
 }
@@ -39,7 +27,7 @@ resource "aws_rds_cluster" "log_rds" {
   # TODO(phboneff): move to either random strings / Secret Manager / IAM
   master_password         = "password"
   skip_final_snapshot     = true
-  backup_retention_period = 0
+  backup_retention_period = 1
 }
 
 resource "aws_rds_cluster_instance" "cluster_instances" {

deployment/modules/aws/storage/outputs.tf

@@ -0,0 +1,9 @@
+output "log_bucket" {
+  description = "Log S3 bucket"
+  value       = aws_s3_bucket.log_bucket
+}
+
+output "log_rds_db" {
+  description = "Log RDS database"
+  value       = aws_rds_cluster.log_rds
+}