Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker local-testnet scripts #5998

Merged
merged 17 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions docker/node/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,10 @@ RUN go build -v -ldflags="-X main.appVersion=$(git describe --tags --long --dirt
RUN cp /go/pkg/mod/github.com/multiversx/$(cat /go/mx-chain-go/go.mod | grep mx-chain-vm-v | sort -n | tail -n -1| awk -F '/' '{print$3}'| sed 's/ /@/g')/wasmer/libwasmer_linux_amd64.so /lib/libwasmer_linux_amd64.so
RUN cp /go/pkg/mod/github.com/multiversx/$(cat /go/mx-chain-go/go.mod | grep mx-chain-vm-go | sort -n | tail -n -1| awk -F '/' '{print$3}'| sed 's/ /@/g')/wasmer2/libvmexeccapi.so /lib/libvmexeccapi.so

WORKDIR /go/mx-chain-go/cmd/node

# ===== SECOND STAGE ======
FROM ubuntu:22.04
RUN apt-get update && apt-get upgrade -y
COPY --from=builder "/go/mx-chain-go/cmd/node" "/go/mx-chain-go/cmd/node/"
COPY --from=builder "/go/mx-chain-go/cmd/node/node" "/go/mx-chain-go/cmd/node/"
COPY --from=builder "/lib/libwasmer_linux_amd64.so" "/lib/libwasmer_linux_amd64.so"
COPY --from=builder "/lib/libvmexeccapi.so" "/lib/libvmexeccapi.so"
WORKDIR /go/mx-chain-go/cmd/node/
Expand Down
6 changes: 6 additions & 0 deletions scripts/docker-testnet/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
pushd ../..
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add #!/usr/bin/env bash here also? i think pushd is bash specific

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this script intended to be run separately?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it is. I could have added the build step inside the ./start.sh script, but it was easier for me to debug, since it didn't build the images at every run. Don't have a strong opinion on this, I can add it to the wrapper script instead. I was planning on adding a README.md nevertheless.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better that we keep the build separate from the run script. Not a strong opinion, thus.


docker build -f docker/seednode/Dockerfile . -t seednode:dev

docker build -f docker/node/Dockerfile . -t node:dev

19 changes: 19 additions & 0 deletions scripts/docker-testnet/clean.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/usr/bin/env bash

# Delete the entire testnet folder, which includes configuration, executables and logs.

export MULTIVERSXTESTNETSCRIPTSDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

source "$MULTIVERSXTESTNETSCRIPTSDIR/variables.sh"

echo "Stopping all containers..."
docker stop $(docker ps -a -q)

echo "Removing all containers..."
docker container prune -f
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the scripts will most probably be used in development environment where there might be also other local containers running, i don't think it's fine to stop and prune all containers, we should stop only the intended containers

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.


echo "Removing network..."
docker network rm ${DOCKER_NETWORK_NAME}

echo "Removing $TESTNETDIR..."
rm -rf $TESTNETDIR
196 changes: 196 additions & 0 deletions scripts/docker-testnet/functions.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
#!/usr/bin/env bash

# Starts from 3, if the DOCKER_NETWORK_SUBNET ends with a 0. The first IP address is reserved for the gateway and the
# second one is allocated to the seednode. Therefore the counting starts from 3. If you modify the DOCKER_NETWORK_SUBNET
# variable, make sure to change this one accordingly too.
IP_HOST_BYTE=3

cloneRepositories() {
if [[ -n $CI_RUN ]]; then
echo "Repositories have been cloned in the CI"
else
cd $(dirname $MULTIVERSXDIR)

git clone [email protected]:multiversx/mx-chain-deploy-go.git || true
git clone [email protected]:multiversx/mx-chain-proxy-go.git || true
fi
}

createDockerNetwork() {
docker network create -d bridge --subnet=${DOCKER_NETWORK_SUBNET} ${DOCKER_NETWORK_NAME}
Copy link
Contributor

@ssd04 ssd04 Mar 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe check if network DOCKER_NETWOR_NAME already exists

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added set -u. Unbound variables will throw an error now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here I was referring to the fact that if the network is already created, it will fail because it is not able to create docker network again


# this variable is used to keep track of the allocated IP addresses in the network, by removing the last byte
# of the DOCKER_NETWORK_SUBNET. One can consider this the host network address without the last byte at the end.
export NETWORK_ADDRESS=$(echo "$DOCKER_NETWORK_SUBNET" | rev | cut -d. -f2- | rev)
}

startSeedNode() {
docker run -d --name seednode -v ${TESTNETDIR}/seednode/config:/go/mx-chain-go/cmd/seednode/config \
--network ${DOCKER_NETWORK_NAME} \
seednode:dev \
--rest-api-interface=0.0.0.0:10000
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indentation here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

}

startObservers() {
local observerIdx=0
# Example for loop with injected variables in Bash
for ((i = 0; i < SHARDCOUNT; i++)); do
for ((j = 0; j < SHARD_OBSERVERCOUNT; j++)); do
# Your commands or code to be executed in each iteration
KEY_INDEX=$((TOTAL_NODECOUNT - observerIdx - 1))

docker run -d --name "observer${observerIdx}-${NETWORK_ADDRESS}.${IP_HOST_BYTE}-10200-shard${i}" \
-v $TESTNETDIR/node/config:/go/mx-chain-go/cmd/node/config \
--network ${DOCKER_NETWORK_NAME} \
node:dev \
--destination-shard-as-observer $i \
--rest-api-interface=0.0.0.0:10200 \
--config ./config/config_observer.toml \
--sk-index=${KEY_INDEX} \
$EXTRA_OBSERVERS_FLAGS


(( IP_HOST_BYTE++ ))
((observerIdx++)) || true
done
done

for ((i = 0; i < META_OBSERVERCOUNT; i++)); do
KEY_INDEX=$((TOTAL_NODECOUNT - observerIdx - 1))

docker run -d --name "observer${observerIdx}-${NETWORK_ADDRESS}.${IP_HOST_BYTE}-10200-metachain" \
-v $TESTNETDIR/node/config:/go/mx-chain-go/cmd/node/config \
--network ${DOCKER_NETWORK_NAME} \
node:dev \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please use same indentation pattern for the entire script and remove unnecessary empty lines

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope it's fixed.

--destination-shard-as-observer "metachain" \
--rest-api-interface=0.0.0.0:10200 \
--config ./config/config_observer.toml \
--sk-index=${KEY_INDEX} \
$EXTRA_OBSERVERS_FLAGS

(( IP_HOST_BYTE++ ))
((observerIdx++)) || true
done
}

startValidators() {
validatorIdx=0
# Example for loop with injected variables in Bash
for ((i = 0; i < SHARDCOUNT; i++)); do
for ((j = 0; j < SHARD_VALIDATORCOUNT; j++)); do

docker run -d --name "validator${validatorIdx}-${NETWORK_ADDRESS}.${IP_HOST_BYTE}-10200-shard${i}" \
-v $TESTNETDIR/node/config:/go/mx-chain-go/cmd/node/config \
--network ${DOCKER_NETWORK_NAME} \
node:dev \
--rest-api-interface=0.0.0.0:10200 \
--config ./config/config_validator.toml \
--sk-index=${validatorIdx} \

(( IP_HOST_BYTE++ ))
((validatorIdx++)) || true
done
done

for ((i = 0; i < META_VALIDATORCOUNT; i++)); do
docker run -d --name "validator${validatorIdx}-${NETWORK_ADDRESS}.${IP_HOST_BYTE}-10200-metachain" \
-v $TESTNETDIR/node/config:/go/mx-chain-go/cmd/node/config \
--network ${DOCKER_NETWORK_NAME} \
node:dev \
--rest-api-interface=0.0.0.0:10200 \
--config ./config/config_observer.toml \
--sk-index=${validatorIdx} \

(( IP_HOST_BYTE++ ))
((validatorIdx++)) || true
done
}

updateProxyConfigDocker() {
pushd $TESTNETDIR/proxy/config
cp config.toml config_edit.toml

# Truncate config.toml before the [[Observers]] list
sed -i -n '/\[\[Observers\]\]/q;p' config_edit.toml

if [ "$SHARD_OBSERVERCOUNT" -le 0 ]; then
generateProxyValidatorListDocker config_edit.toml
else
generateProxyObserverListDocker config_edit.toml
fi

mv config_edit.toml config.toml

echo "Updated configuration for the Proxy."
popd
}

generateProxyObserverListDocker() {
local ipByte=3
OUTPUTFILE=$!


for ((i = 0; i < SHARDCOUNT; i++)); do
for ((j = 0; j < SHARD_OBSERVERCOUNT; j++)); do

echo "[[Observers]]" >> config_edit.toml
echo " ShardId = $i" >> config_edit.toml
echo " Address = \"http://${NETWORK_ADDRESS}.${ipByte}:10200\"" >> config_edit.toml
echo ""$'\n' >> config_edit.toml

(( ipByte++ )) || true
done
done

for META_OBSERVER in $(seq $META_OBSERVERCOUNT); do
echo "[[Observers]]" >> config_edit.toml
echo " ShardId = $METASHARD_ID" >> config_edit.toml
echo " Address = \"http://${NETWORK_ADDRESS}.${ipByte}:10200\"" >> config_edit.toml
echo ""$'\n' >> config_edit.toml

(( ipByte++ )) || true
done
}

generateProxyValidatorListDocker() {
local ipByte=3
OUTPUTFILE=$!


for ((i = 0; i < SHARDCOUNT; i++)); do
for ((j = 0; j < SHARD_VALIDATORCOUNT; j++)); do

echo "[[Observers]]" >> config_edit.toml
echo " ShardId = $i" >> config_edit.toml
echo " Address = \"http://${NETWORK_ADDRESS}.${ipByte}:10200\"" >> config_edit.toml
echo " Type = \"Validator\"" >> config_edit.toml
echo ""$'\n' >> config_edit.toml

(( ipByte++ )) || true
done
done

for META_OBSERVER in $(seq $META_VALIDATORCOUNT); do
echo "[[Observers]]" >> config_edit.toml
echo " ShardId = $METASHARD_ID" >> config_edit.toml
echo " Address = \"http://${NETWORK_ADDRESS}.${ipByte}:10200\"" >> config_edit.toml
echo " Type = \"Validator\"" >> config_edit.toml
echo ""$'\n' >> config_edit.toml

(( ipByte++ )) || true
done
}

buildProxyImage() {
pushd ${PROXYDIR}
cd ../..
docker build -f docker/Dockerfile . -t proxy:dev
}

startProxyDocker() {
docker run -d --name "proxy" \
-p $PORT_PROXY:8080 \
-v $TESTNETDIR/proxy/config:/mx-chain-proxy-go/cmd/proxy/config \
--network ${DOCKER_NETWORK_NAME} \
proxy:dev
}
43 changes: 43 additions & 0 deletions scripts/docker-testnet/start.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#!/usr/bin/env bash

set -e

export DOCKERTESTNETDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

MULTIVERSXTESTNETSCRIPTSDIR="$(dirname "$DOCKERTESTNETDIR")/testnet"

source "$DOCKERTESTNETDIR/variables.sh"
source "$DOCKERTESTNETDIR/functions.sh"
source "$MULTIVERSXTESTNETSCRIPTSDIR/include/config.sh"
source "$MULTIVERSXTESTNETSCRIPTSDIR/include/build.sh"

cloneRepositories

prepareFolders

buildConfigGenerator

generateConfig

copyConfig

copySeednodeConfig
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting, so you actually use a lot of the existing testnet scripts configuration functions 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, it's mostly the same configuration. The only difference in configuration are the ports and the ip addresses.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, I've noticed that: the first question was: how was possible to function if all nodes run on the same ports. But then I've saw the IPs.

updateSeednodeConfig

copyNodeConfig
updateNodeConfig

createDockerNetwork

startSeedNode
startObservers
startValidators

if [ $USE_PROXY -eq 1 ]; then
buildProxyImage
prepareFolders_Proxy
copyProxyConfig
updateProxyConfigDocker
startProxyDocker
fi

Loading
Loading