Docker local-testnet scripts #5998

cristure · 2024-02-27T15:58:52Z

Reasoning behind the pull request

Scipt to setup a local testnet with docker.

Proposed changes

Added scripts to setup local testnet.

Testing procedure

Run scripts/docker-testnet/start.sh
Inspect the created containers.

Pre-requisites

Docker installed.

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

was the PR targeted to the correct branch?
if this is a larger feature that probably needs more than one PR, is there a feat branch created?
if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

codecov · 2024-02-27T16:06:25Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.17%. Comparing base (5e1569b) to head (71d48d4).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5998      +/-   ##
==========================================
- Coverage   80.21%   80.17%   -0.04%     
==========================================
  Files         709      709              
  Lines       94166    94166              
==========================================
- Hits        75535    75501      -34     
- Misses      13284    13316      +32     
- Partials     5347     5349       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

scripts/docker-testnet/helpers.sh

iulianpascalau · 2024-03-14T08:21:51Z

scripts/docker-testnet/start.sh

+
+copyConfig
+
+copySeednodeConfig


interesting, so you actually use a lot of the existing testnet scripts configuration functions 🤔

yep, it's mostly the same configuration. The only difference in configuration are the ports and the ip addresses.

right, I've noticed that: the first question was: how was possible to function if all nodes run on the same ports. But then I've saw the IPs.

…stnet.

iulianpascalau

will test this ASAP

ssd04 · 2024-03-16T19:26:14Z

scripts/docker-testnet/build.sh

@@ -0,0 +1,6 @@
+pushd ../..


add #!/usr/bin/env bash here also? i think pushd is bash specific

is this script intended to be run separately?

yes, it is. I could have added the build step inside the ./start.sh script, but it was easier for me to debug, since it didn't build the images at every run. Don't have a strong opinion on this, I can add it to the wrapper script instead. I was planning on adding a README.md nevertheless.

I think it is better that we keep the build separate from the run script. Not a strong opinion, thus.

ssd04 · 2024-03-16T19:39:46Z

scripts/docker-testnet/functions.sh

+  --network ${DOCKER_NETWORK_NAME} \
+  seednode:dev \
+  --rest-api-interface=0.0.0.0:10000


indentation here?

ssd04 · 2024-03-16T19:40:43Z

scripts/docker-testnet/functions.sh

+        -v $TESTNETDIR/node/config:/go/mx-chain-go/cmd/node/config \
+        --network ${DOCKER_NETWORK_NAME} \
+        node:dev \


please use same indentation pattern for the entire script and remove unnecessary empty lines

I hope it's fixed.

ssd04 · 2024-03-16T19:44:25Z

scripts/docker-testnet/functions.sh

+}
+
+createDockerNetwork() {
+  docker network create -d bridge --subnet=${DOCKER_NETWORK_SUBNET} ${DOCKER_NETWORK_NAME}


maybe check if network DOCKER_NETWOR_NAME already exists

added set -u. Unbound variables will throw an error now.

here I was referring to the fact that if the network is already created, it will fail because it is not able to create docker network again

ssd04 · 2024-03-16T19:46:39Z

scripts/docker-testnet/clean.sh

+echo "Stopping all containers..."
+docker stop $(docker ps -a -q)
+
+echo "Removing all containers..."
+docker container prune -f


the scripts will most probably be used in development environment where there might be also other local containers running, i don't think it's fine to stop and prune all containers, we should stop only the intended containers

iulianpascalau · 2024-03-18T08:46:00Z

scripts/docker-testnet/variables.sh

-if [ "$TESTNETMODE" == "trace" ]; then
-  LOGLEVEL="*:TRACE"
-fi
+## Use tmux or not. If set to 1, only 2 terminal windows will be opened, and


commented code. Can use a local.sh file instead. We have exceptions for it in the .gitignore

removed altogether. not needed in our context.

iulianpascalau · 2024-03-18T08:46:55Z

scripts/docker-testnet/functions.sh

-    git clone [email protected]:multiversx/mx-chain-deploy-go.git || true
-    git clone [email protected]:multiversx/mx-chain-proxy-go.git || true
-  fi
+  cd $(dirname $MULTIVERSXDIR)


Why this change? Does this function will always clone the proxy and deploy repos?

yes, it will. But if the repository already exists, it won't clone them again.

confirmed that is working properly 👍

git clone [email protected]:multiversx/mx-chain-deploy-go.git || true fatal: destination path 'mx-chain-deploy-go' already exists and is not an empty directory.

ssd04 · 2024-03-18T08:55:37Z

scripts/docker-testnet/README.md

+./build.sh  # (Optional) Can be ignored if you already have the images stored in the local registry.
+./start.sh  # Will start the local-testnet.
+./clean.sh  # Will stop and remove the containers related to the local-testnet.


i think that a good flow here is to have also a stop step, in order to be able to have the following development flow:

initially: build

multiple times: start , stop

and finally: clean

i think is not very good to have to clean everything on each run, sometimes we just wan't to stop the squad and start it again, with the same/setup configuration, later on

this is also related to the docker network create comment

This intended workflow adds some complexity. The start.sh uses the docker run command. If you stop all containers, they will only start again withdocker start instead, otherwise it will throw an error. I don't see much of a benefit to this behaviour, as I need to persists all the containerIds/containerNames between stopping and restarting. One can achieve this manually by leveraging already available commands in the clean.sh script.

iulianpascalau

tested locally with system-tests-go and worked perfectly

iulianpascalau · 2024-03-20T15:07:15Z

scripts/docker-testnet/start.sh

+    docker start $line
+done < "$file_path"
+
+rmdir ./tmp


add empty line?

iulianpascalau · 2024-03-20T15:07:39Z

scripts/docker-testnet/stop.sh

+for CONTAINER_ID in $CONTAINER_IDS; do
+    docker stop "$CONTAINER_ID"
+    echo "$CONTAINER_ID" >> ./tmp/stopped_containers
+done


add empty line?

cristure marked this pull request as draft February 27, 2024 16:04

cristure marked this pull request as ready for review February 28, 2024 10:10

added script to create local-testnet with docker.

41d8908

cristure force-pushed the local-testnet-docker branch from 90b5a51 to 41d8908 Compare March 14, 2024 07:21

remove CI to build local testnet.

29367f3

iulianpascalau self-requested a review March 14, 2024 08:16

iulianpascalau reviewed Mar 14, 2024

View reviewed changes

AdoAdoAdo self-requested a review March 14, 2024 08:27

cristure added 3 commits March 14, 2024 10:49

cosmetic changes.

d13ca14

added build.sh

42d2b78

fixed typo.

a77aef7

iulianpascalau previously approved these changes Mar 14, 2024

View reviewed changes

added execution permissions on build.sh

b8cf372

cristure dismissed iulianpascalau’s stale review via b8cf372 March 14, 2024 09:10

iulianpascalau previously approved these changes Mar 14, 2024

View reviewed changes

AdoAdoAdo previously approved these changes Mar 14, 2024

View reviewed changes

changed containers name for better visibility.

4f53303

cristure dismissed stale reviews from AdoAdoAdo and iulianpascalau via 4f53303 March 14, 2024 11:13

cristure changed the title ~~Docker compose CI test.~~ Docker local-testnet scripts Mar 14, 2024

cristure and others added 3 commits March 14, 2024 14:12

Merge branch 'master' into local-testnet-docker

c6caf6c

removed hardcoded ips and added dedicated network for docker local-te…

092ef55

…stnet.

Removed executable permissions for variables.sh

01e02fd

andreibancioiu assigned cristure Mar 15, 2024

iulianpascalau previously approved these changes Mar 15, 2024

View reviewed changes

ssd04 requested changes Mar 16, 2024

View reviewed changes

fixes after review.

24c1ac4

cristure dismissed iulianpascalau’s stale review via 24c1ac4 March 18, 2024 08:33

Added README.md

ecd2a34

iulianpascalau reviewed Mar 18, 2024

View reviewed changes

ssd04 reviewed Mar 18, 2024

View reviewed changes

cosmetic changes.

9bf2d7c

iulianpascalau previously approved these changes Mar 18, 2024

View reviewed changes

Added publish ports argument and formatting changes.

96adb1f

cristure dismissed iulianpascalau’s stale review via 96adb1f March 20, 2024 14:07

Added start/stop mechanism along with comsetic changes.

e3cd364

iulianpascalau previously approved these changes Mar 20, 2024

View reviewed changes

Fixed paths.

d3cf4f8

cristure dismissed iulianpascalau’s stale review via d3cf4f8 March 20, 2024 15:18

fixed ports for nodes.

71d48d4

ssd04 approved these changes Mar 21, 2024

View reviewed changes

iulianpascalau approved these changes Mar 21, 2024

View reviewed changes

AdoAdoAdo approved these changes Mar 21, 2024

View reviewed changes

cristure merged commit 69d5f59 into master Mar 22, 2024
7 of 8 checks passed

cristure deleted the local-testnet-docker branch March 22, 2024 12:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker local-testnet scripts #5998

Docker local-testnet scripts #5998

cristure commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading

iulianpascalau Mar 14, 2024

cristure Mar 14, 2024

iulianpascalau Mar 14, 2024

iulianpascalau left a comment

ssd04 Mar 16, 2024

ssd04 Mar 16, 2024

cristure Mar 18, 2024

iulianpascalau Mar 18, 2024

ssd04 Mar 16, 2024

cristure Mar 18, 2024

ssd04 Mar 16, 2024

cristure Mar 18, 2024

ssd04 Mar 16, 2024 •

edited

Loading

cristure Mar 18, 2024

ssd04 Mar 18, 2024

ssd04 Mar 16, 2024

cristure Mar 18, 2024

iulianpascalau Mar 18, 2024

cristure Mar 18, 2024

iulianpascalau Mar 18, 2024

cristure Mar 18, 2024

iulianpascalau Mar 18, 2024

ssd04 Mar 18, 2024 •

edited

Loading

ssd04 Mar 18, 2024 •

edited

Loading

cristure Mar 18, 2024

iulianpascalau left a comment

iulianpascalau Mar 20, 2024

iulianpascalau Mar 20, 2024


		copyConfig

		copySeednodeConfig

Docker local-testnet scripts #5998

Docker local-testnet scripts #5998

Conversation

cristure commented Feb 27, 2024 • edited Loading

Reasoning behind the pull request

Proposed changes

Testing procedure

Pre-requisites

codecov bot commented Feb 27, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iulianpascalau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssd04 Mar 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssd04 Mar 18, 2024 • edited Loading

Choose a reason for hiding this comment

ssd04 Mar 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iulianpascalau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cristure commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading

ssd04 Mar 16, 2024 •

edited

Loading

ssd04 Mar 18, 2024 •

edited

Loading

ssd04 Mar 18, 2024 •

edited

Loading