#207 - Simplified docker-compose and added support for testnet (#278)

* Simplified docker-compose and added support for testnet

* Additional docker documentation

* updated docker-compose curl download command

* Updated docker README.md

* referenced new Sidtree build that core DB retry logic

* minor edits

docker/README.md

@@ -1,66 +1,131 @@
 # ION using Docker 
 
-This directory contains the automated Docker deployment of ION.  This will create 5 docker containers that contain the core components to run ION.
+This directory contains the automated Docker deployment of ION. 
 
-- bitcoin-core-mainnet - This container will run the bitcoin-core client.  `NOTE: This is config is running on mainnet as the source chain`
-- ion-bitcoin-mainnet - This is the bitcoin sidetree node for ION.
-- ipfs - This is the local IPFS node used by the sidetree.
-- mongo - This is the local MongoDB used by the sidetree.
-- ion-core-mainnet - This is the core sidetree node.
 
-The automation will create the docker containers in order (and will wait for bitcoin sync).
-After ion-bitcoin is done synchronizing the blocks you need to hit enter to start the ion-core container. It cannot start if ion-bitcoin is still scanning files.
+# Running an ION node using Docker
+Running a ION node using docker is as simple as executing a docker command once docker engine is installed. This will create 5 docker containers that contain the core components to run an ION node.
+
+- `mongo` - This is the local MongoDB used by the ION node.
+- `bitcoin-core` - This container will run the Bitcoin Core client.
+- `ipfs` - This is the local IPFS node used by the ION node.
+- `ion-bitcoin` - This is the ION service interacting with Bitcoin Core.
+- `ion-core` - This is the ION Core service.
+
+> NOTE: currently the docker image only supports running a read-only ION node.
 
 ## Prerequisites
 
-This implementation is designed to be run with very little dependencies, on a Linux host.
+This implementation is designed to be run with very little dependencies beyond beyond a docker engine.
+
+### Linux Host
 
-- Linux Ubuntu 18.04LTS  `NOTE: You will need about 350 GB of free space for bitcoin mainnet.`
+- Linux Ubuntu 18.04LTS
 
 - Docker engine (latest) (from https://docs.docker.com/engine/install/ubuntu/)
   ```
+  sudo apt-get update
+
   sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
 
   curl -fsSL --max-time 10 --retry 3 --retry-delay 3 --retry-max-time 60 https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
 
   sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
 
-  sudo apt-get update
   sudo apt-get install -y docker-ce
   sudo systemctl enable docker
   ```
 
 - Docker compose (latest)
   ```
-  sudo curl -L --max-time 10 --retry 3 --retry-delay 3 --retry-max-time 60 "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
+  sudo curl -L --max-time 60 --retry 3 --retry-delay 3 --retry-max-time 100 "https://github.com/docker/compose/releases/download/v2.6.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
 
   sudo chmod +x /usr/local/bin/docker-compose
   ```
 
-- NodeJS - version 14.x   `NOTE: This has a requirement for version 14 of nodejs only`
-  ```
-  curl -sl https://deb.nodesource.com/setup_14.x | sudo bash -
-  ```
-
-- make - Build tool used for library this depends on.  `NOTE: You can get this as part of the build-essential package for debian based os`
-  ```
-  sudo apt-get install build-essential -y
-  ```
+### MacOS Host
+- Install [Docker Desktop](https://www.docker.com/products/docker-desktop/).
 
 ## Run
-Before you start the setup you need to install the packages from the directory with the ION repo.
+
+You will first need to download the `docker-compose.*` files in this directory before you can run the docker command to start an ION node. You can either clone the entire repo or just copy the few files locally manually. To clone the entire repo:
+
 ```
 git clone https://github.com/decentralized-identity/ion.git
-npm install
-npm run build
 ```
-After those ran succesfully, navigate to the docker directory and run the configuration script [configuration script](deploy-docker-mainnet.sh). The script will prompt for data directories to use. As noted above, you will need about 350GB of free space to run this.
+
+Once you've downloaded the docker files locally, `cd` into the directory containing the `docker-compose` files, then:
+
+- To run a `mainnet` ION node:
+```sh
+docker-compose up -d
+```
+
+- To run a `testnet` ION node:
+
+```sh
+docker-compose -f docker-compose.yml -f docker-compose.testnet-override.yml up -d
+```
+
+That's it! An ION node should be starting up.
+
+The initial synchronization will take at least 24 hours to complete for `mainnet` (quicker for `testnet`), based on your machine and internet connectivity speed. The long initialization time is primarily because ION requires a full bitcoin node for its trustless setup, and downloading the entire bitcoin blockchain takes a very long time. Sync progress is logged by the `ion-bitcoin` service.
+
+> NOTE: For bitcoin `mainnet`, you will be downloading at least _450 GB_ worth of blockchain data, thus you will also need at least that must local storage space.
+
+> NOTE: the dockers containers expose their service ports so that requests can be sent from the host machine for debugging purposes.
+
+## Customize ION Data Directory
+You can customize the location where all ION data generated is stored by specifying it in the `ION_DATA_VOLUME` variable.
+
+This is especially useful if you already have a copy of the fully synchronized bitcoin data, allowing you to avoid downloading the entire bitcoin blockchain again. Just pass the custom path to `docker-compose` like so:
+
+
+```sh
+ION_DATA_VOLUME=<custom_data_path> docker-compose up -d
+```
+
+The `docker-compose` files assumes the following directory structure:
 
 ```
-chmod +x deploy-docker-mainnet.sh
-./deploy-docker-mainnet.sh
+<custom_data_path>
+  |
+  |-- bitcoin
+  |
+  |-- ipfs
+  |
+  |-- mongo
 ```
 
-The setup will take around 24-30 hours, based on your machine and internet connectivity speed.  This is primarily because the bitcoin mainnet full node will be required to be synced before starting ION.  A simple progress indicator will be shown to give status on the sync operation. The sync of mainnet can take around 24 hours.
+# Building a Multi-Platform ION Image
+For developers only. Ignore this section if you looking to just run an ION node using docker. If you don't care about multi-platform support, ignore rest of this section, just run:
+
+  `docker build -t <repository>:<tag> -f dockerfile ../`
+
+  e.g. `docker build -t thehenrytsai/ion:1.0.4 -f dockerfile ../`
+
+## Prerequisites
+
+- You must use `docker buildx` instead of `docker build`, this should come installed with Docker Desktop.
+
+- Default build driver does not support building multi-platform images. You will have to create a separate `BuildKit` build driver instance by running:
+
+  `docker buildx create --name <custom_name> --use`
+  - `--name` - if not specified a name will be generated for you
+  - `--use` - sets this instance as the default build instance
+  - By default `buildx create` creates a new build instance using `BuildKit` container driver, which is what we want. This is equivalent to specifying the parameter explicitly: `--driver docker-container`
+
+- You can use `docker buildx ls` to list the build driver instances to verify you have created the `BuildKit` instance.
+
+## Building the Docker Image
+- `npm i` followed by `npm run build` in root directory.
+- `cd` into the `docker` directory, run:
+
+- Run the `buildx` command to build the multi-platform image:
+
+`docker buildx build --platform=linux/arm64,linux/amd64 --push -t <repository>:<tag> -f dockerfile ../`
+
+  e.g. `docker buildx build --platform=linux/arm64,linux/amd64 --push -t thehenrytsai/ion:1.0.4 -f dockerfile ../`
 
-`NOTE: This configuration is using a test key for bitcoin.  You can find this in the json file mainnet-bitcoin-docker-config.json file with the parameter, bitcoinWalletOrImportString.  This should be changed to a real key that is appropriately secured for production workloads if you want to be able to write to the network. If you want to only resolve did's the keys works fine!`
+  - `--platform` - specifies the platforms this images will be built for
+  - `--push` - pushes the image to the specified repository

docker/docker-compose.testnet-override.yml

@@ -0,0 +1,18 @@
+services:
+  bitcoin-core:
+    container_name: testnet-bitcoin-core
+    command: -printtoconsole -txindex -server -rpcuser=user -rpcpassword=password -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0 -testnet
+    ports:
+      - "18332:18332"
+
+  ion-bitcoin:
+    container_name: testnet-ion-bitcoin
+    environment:
+      - ION_BITCOIN_CONFIG_FILE_PATH=/json/docker-testnet-bitcoin-config.json
+      - ION_BITCOIN_VERSIONING_CONFIG_FILE_PATH=/json/testnet-bitcoin-versioning.json
+
+  ion-core:
+    container_name: testnet-ion-core
+    environment:
+      - ION_CORE_CONFIG_FILE_PATH=/json/docker-testnet-core-config.json
+      - ION_CORE_VERSIONING_CONFIG_FILE_PATH=/json/testnet-core-versioning.json