Add documentation project

.github/workflows/docs.yml

@@ -0,0 +1,22 @@
+name: docs
+on:
+  push:
+    branches:
+      - main
+      - mkdocs
+permissions:
+  contents: write
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - uses: actions/cache@v2
+        with:
+          key: ${{ github.ref }}
+          path: .cache
+      - run: pip install mkdocs-material
+      - run: mkdocs gh-deploy --force

.pre-commit-config.yaml

@@ -3,6 +3,8 @@ repos:
     rev: v4.4.0
     hooks:
       - id: check-yaml
+        args:
+          -   --unsafe
       - id: check-json
       - id: end-of-file-fixer
       - id: trailing-whitespace
@@ -18,6 +20,8 @@ repos:
   - repo: https://github.com/terraform-docs/terraform-docs
     rev: v0.16.0
     hooks:
+      - id: terraform-docs-go
+        args: ["./terraform", "--output-file", "../docs/cloud-resources.md"]
       - id: terraform-docs-go
         args: ["./terraform"]
       - id: terraform-docs-go

README.md

@@ -0,0 +1,6 @@
+# CalData Data Services and Engineering Infrastructure
+
+![deploy](https://github.com/cagov/data-infrastructure/actions/workflows/deploy.yml/badge.svg?branch=main)
+![docs](https://github.com/cagov/data-infrastructure/actions/workflows/docs.yml/badge.svg?branch=main)
+
+Documentation for this project can be found [here](https://cagov.github.io/data-infrastructure/).

docs/cloud-infrastructure.md

@@ -0,0 +1,207 @@
+# Cloud Infrastructure
+
+The DSE team uses Terraform to manage cloud infrastructure.
+Our stack includes:
+
+* An [AWS Batch](https://aws.amazon.com/batch/) environment for running arbitrary containerized jobs
+* A [Managed Workflows on Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) environment for orchestrating jobs
+* A VPC and subnets for the above
+* An ECR repository for hosting Docker images storing code and libraries for jobs
+* A bot user for running AWS operations in GitHub Actions
+* An S3 scratch bucket
+
+## Architecture
+
+```mermaid
+flowchart TD
+  subgraph AWS
+    J[GitHub CD\nbot user]
+    G[Artifact in S3]
+    subgraph VPC
+      subgraph Managed Airflow
+        K1[Scheduler]
+        K2[Worker]
+        K3[Webserver]
+      end
+      F[AWS Batch Job\n on Fargate]
+    end
+    E[AWS ECR Docker\nRepository]
+  end
+  subgraph GitHub
+    A[Code Repository]
+  end
+  E --> F
+  A -- Code quality check\n GitHub action --> A
+  A -- Job submission\nvia GitHub Action --> F
+  A -- Docker build \nGitHub Action --> E
+  A --> H[CalData\nadministrative\nuser]
+  H -- Terraform -----> AWS
+  K2 -- Job submission\nvia Airflow --> F
+  K1 <--> K2
+  K3 <--> K1
+  K3 <--> K2
+  F --> G
+  J -- Bot Credentials --> A
+```
+
+## Setup
+
+### Installation
+
+This project requires Terraform to run.
+You might use a different package manager to install it depending on your system.
+
+For Macs, you can use `brew`:
+
+```bash
+brew install terraform
+```
+
+Anaconda users on any architecture should be able to use `conda` or `mamba`:
+
+```bash
+conda install -c conda-forge terraform
+```
+
+We also use `tflint` for linting, and `terraform-docs` to help with documentation of resources.
+These can be installed in the same manner, e.g.:
+
+```bash
+conda install -c conda-forge tflint go-terraform-docs
+```
+
+There are a number of pre-commit checks that run on committing as well as in CI.
+To install the checks, run the following from the repository root:
+
+```bash
+pre-commit install
+```
+
+You can manually run the pre-commit checks using:
+
+```bash
+pre-commit run --all-files
+```
+
+### Bootstrapping remote state
+
+When deploying a new version of your infrastrucutre, Terraform diffs the current state
+against what you have specified in your infrastructure-as-code.
+The current state is tracked in a JSON document,
+which can be stored in any of a number of locations (including local files).
+
+This project stores remote state using the [S3 backend](https://developer.hashicorp.com/terraform/language/settings/backends/s3).
+Before you can stand up the main infrastructure, you must first prep the remote state backend:
+
+```bash
+cd remote-state
+terraform init
+terraform apply
+```
+
+With the remote state infrastructure deployed, you should be able to initialize the main project.
+From this directory, run:
+
+```bash
+terraform init -backend-config=./remote-state/dse-infra-dev.tfbackend
+```
+
+## Deploying Infrastructure
+
+When you are ready to deploy a new version of the infrastructure, run
+
+```bash
+terraform apply
+```
+
+This will output the changes to the infrastructure that will be made, and prompt you for confirmation.
+
+
+<!-- BEGIN_TF_DOCS -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | 4.56.0 |
+| <a name="requirement_random"></a> [random](#requirement\_random) | 3.4.3 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_aws"></a> [aws](#provider\_aws) | 4.56.0 |
+| <a name="provider_random"></a> [random](#provider\_random) | 3.4.3 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [aws_batch_compute_environment.default](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/batch_compute_environment) | resource |
+| [aws_batch_job_definition.default](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/batch_job_definition) | resource |
+| [aws_batch_job_queue.default](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/batch_job_queue) | resource |
+| [aws_ecr_repository.default](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/ecr_repository) | resource |
+| [aws_eip.this](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/eip) | resource |
+| [aws_iam_policy.batch_submit_policy](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_policy) | resource |
+| [aws_iam_policy.default_ecr_policy](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_policy) | resource |
+| [aws_iam_policy.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_policy) | resource |
+| [aws_iam_policy.s3_scratch_policy](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_policy) | resource |
+| [aws_iam_role.aws_batch_service_role](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role) | resource |
+| [aws_iam_role.batch_job_role](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role) | resource |
+| [aws_iam_role.ecs_task_execution_role](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role) | resource |
+| [aws_iam_role.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role) | resource |
+| [aws_iam_role_policy_attachment.aws_batch_service_role](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.ecs_task_execution_role_policy](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.mwaa_batch_submit_role](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.mwaa_execution_role](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_role_policy_attachment.s3_scratch_policy_role_attachment](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_user.cd_bot](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_user) | resource |
+| [aws_iam_user_policy_attachment.batch_cd_bot_policy_attachment](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_user_policy_attachment) | resource |
+| [aws_iam_user_policy_attachment.ecr_cd_bot_policy_attachment](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/iam_user_policy_attachment) | resource |
+| [aws_internet_gateway.this](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/internet_gateway) | resource |
+| [aws_mwaa_environment.this](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/mwaa_environment) | resource |
+| [aws_nat_gateway.this](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/nat_gateway) | resource |
+| [aws_route_table.private](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/route_table) | resource |
+| [aws_route_table.public](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/route_table) | resource |
+| [aws_route_table_association.private](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/route_table_association) | resource |
+| [aws_route_table_association.public](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/route_table_association) | resource |
+| [aws_s3_bucket.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/s3_bucket) | resource |
+| [aws_s3_bucket.scratch](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/s3_bucket) | resource |
+| [aws_s3_bucket_public_access_block.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/s3_bucket_public_access_block) | resource |
+| [aws_s3_bucket_versioning.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/s3_bucket_versioning) | resource |
+| [aws_security_group.batch](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/security_group) | resource |
+| [aws_security_group.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/security_group) | resource |
+| [aws_subnet.private](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/subnet) | resource |
+| [aws_subnet.public](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/subnet) | resource |
+| [aws_vpc.this](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/resources/vpc) | resource |
+| [random_id.private_subnet](https://registry.terraform.io/providers/hashicorp/random/3.4.3/docs/resources/id) | resource |
+| [random_id.public_subnet](https://registry.terraform.io/providers/hashicorp/random/3.4.3/docs/resources/id) | resource |
+| [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/availability_zones) | data source |
+| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/caller_identity) | data source |
+| [aws_iam_policy_document.assume](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.assume_role_policy](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.aws_batch_service_policy](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.batch_submit_policy_document](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.default_ecr_policy_document](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.mwaa](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_policy_document.s3_scratch_policy_document](https://registry.terraform.io/providers/hashicorp/aws/4.56.0/docs/data-sources/iam_policy_document) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_environment"></a> [environment](#input\_environment) | Deployment environment of the resource | `string` | `"dev"` | no |
+| <a name="input_owner"></a> [owner](#input\_owner) | Owner of the resource | `string` | `"dse"` | no |
+| <a name="input_project"></a> [project](#input\_project) | Name of the project the resource is serving | `string` | `"infra"` | no |
+| <a name="input_region"></a> [region](#input\_region) | Region for AWS resources | `string` | `"us-west-2"` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_state"></a> [state](#output\_state) | Resources from terraform-state |
+<!-- END_TF_DOCS -->

code-review.md → docs/code-review.md

@@ -1,6 +1,8 @@
 # Conventions for Code Review and GitHub
 
-## The Process
+This page documents the Data Services and Engineering Team's practices around GitHub-based development and code review.
+
+## Creating a Pull Request
 
 The process for GitHub-based development is:
 
@@ -20,7 +22,7 @@ The process for GitHub-based development is:
 1. In general, the author of a PR should not approve and merge their own pull request.
 1. Delete your feature branch, it's in `main` now.
 
-## Authoring a pull request
+## Considerations when authoring a pull request
 
 #### Have Empathy
 

docs/index.md

@@ -0,0 +1,5 @@
+# CalData Data Services and Engineering Infrastructure
+
+This is the technical documentation for CalData's
+Data Services and Engineering (DSE) projects.
+It consists of processes, conventions, instructions, and architecture diagrams.