Skip to content

Commit

Permalink
Avoid using double the amount of space when downloading a blockstore (#…
Browse files Browse the repository at this point in the history 
…80)
  • Loading branch information
@bakhtin
bakhtin authored May 25, 2022
1 parent d60d6a4 commit 858e50a
Show file tree
Hide file tree
Showing 4 changed files with 33 additions and 33 deletions.
2 changes: 1 addition & 1 deletion charts/node/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ apiVersion: v2
name: node
description: A Helm chart to deploy Substrate/Polkadot nodes
type: application
version: 2.0.0
version: 2.1.0
14 changes: 7 additions & 7 deletions charts/node/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ This will deploy a single Polkadot node with the default configuration.

Polkadot:
```console
helm install polkadot-node parity/node --set node.chainDataSnapshotUrl=https://dot-rocksdb.polkashots.io/snapshot --set node.chainDataSnapshotFormat=7z
helm install polkadot-node parity/node --set node.chainDataSnapshotUrl=https://dot-rocksdb.polkashots.io/snapshot --set node.chainDataSnapshotFormat=lz4
```

Kusama:
```console
helm install kusama-node parity/node --set node.chainDataSnapshotUrl=https://ksm-rocksdb.polkashots.io/snapshot --set node.chainDataSnapshotFormat=7z --set node.chainPath=ksmcc3
helm install kusama-node parity/node --set node.chainDataSnapshotUrl=https://ksm-rocksdb.polkashots.io/snapshot --set node.chainDataSnapshotFormat=lz4 --set node.chainPath=ksmcc3
```
⚠️ For some chains where the local directory name is different from the chain ID, `node.chainPath` needs to be set to a custom value.

Expand Down Expand Up @@ -86,7 +86,7 @@ node:
| `node.replica` | Number of replica in the node StatefulSet | `1` |
| `node.role` | Set the role of the node: `full`, `authority`/`validator`, `collator` or `light` | `full` |
| `node.chainDataSnapshotUrl` | Download and load chain data from a snapshot archive http URL | `` |
| `node.chainDataSnapshotFormat` | The snapshot archive format (`tar` or `7z`) | `tar` |
| `node.chainDataSnapshotFormat` | The snapshot archive format (`tar` or `lz4`) | `tar` |
| `node.chainDataGcsBucketUrl` | Sync chain data files from a GCS bucket (eg. `gs://bucket-name/folder-name`) | `` |
| `node.chainPath` | Path at which the chain database files are located (`/data/chains/${CHAIN_PATH}`) | `nil` (if undefined, fallbacks to the value in `node.chain`) |
| `node.chainDataKubernetesVolumeSnapshot` | Initialize the chain data volume from a Kubernetes VolumeSnapshot | `` |
Expand All @@ -95,7 +95,7 @@ node:
| `node.collator.isParachain` | If true, configure the node as a parachain (set the relay-chain flags after `--`) | `nil` |
| `node.collator.relayChainCustomChainspecUrl` | Download and use a custom relay-chain chainspec file from a URL | `nil` |
| `node.collator.relayChainDataSnapshotUrl` | Download and load relay-chain data from a snapshot archive http URL | `nil` |
| `node.collator.relayChainDataSnapshotFormat` | The relay-chain snapshot archive format (`tar` or `7z`) | `nil` |
| `node.collator.relayChainDataSnapshotFormat` | The relay-chain snapshot archive format (`tar` or `lz4`) | `nil` |
| `node.collator.relayChainPath` | Path at which the chain database files are located (`/data/polkadot/chains/${RELAY_CHAIN_PATH}`) | `nil` |
| `node.collator.relayChainDataKubernetesVolumeSnapshot` | Initialize the relay-chain data volume from a Kubernetes VolumeSnapshot | `nil` |
| `node.collator.relayChainDataGcsBucketUrl` | Sync relay-chain data files from a GCS bucket (eg. `gs://bucket-name/folder-name`) | `nil` |
Expand Down Expand Up @@ -144,7 +144,7 @@ node:
| `image.repository` | Node image name | `parity/polkadot` |
| `image.tag` | Node image tag | `latest` |
| `image.pullPolicy` | Node image pull policy | `Always` |
| `initContainer.image.repository` | Download-chain-snapshot init container image name | `crazymax/7zip` |
| `initContainer.image.repository` | Download-chain-snapshot init container image name | `alpine` |
| `initContainer.image.tag` | Download-chain-snapshot init container image tag | `latest` |
| `googleCloudSdk.image.repository` | Sync-chain-gcs init container image name | `google/cloud-sdk` |
| `googleCloudSdk.image.tag` | Sync-chain-gcs init container image tag | `slim` |
Expand All @@ -160,6 +160,6 @@ node:
| `jaegerAgent.ports.binaryPort` | Port to use for jaeger.thrift over binary thrift protocol | `6832` |
| `jaegerAgent.ports.samplingPort` | Port for HTTP sampling strategies | `5778` |
| `jaegerAgent.collector.url` | The URL which jaeger agent sends data | `nil` |
| `jaegerAgent.collector.port ` | The port which jaeger agent sends data | `14250` |
| `extraContainers ` | Sidecar containers to add to the node | `[]` |
| `jaegerAgent.collector.port ` | The port which jaeger agent sends data | `14250` |
| `extraContainers ` | Sidecar containers to add to the node | `[]` |
| `serviceAccount` | ServiceAccount used in init containers | `{create: true, createRoleBinding: true, annotations: {}}` |
44 changes: 22 additions & 22 deletions charts/node/templates/statefulset.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,23 +72,18 @@ spec:
args:
- -c
- |
set -eu -o pipefail
if [ -d "/data/chains/${CHAIN_PATH}/db" ]; then
echo "Database directory already exists, skipping chain snapshot download"
else
echo "Downloading chain snapshot"
SNAPSHOT_URL="{{ .Values.node.chainDataSnapshotUrl }}"
wget -O /data/snapshot ${SNAPSHOT_URL}
if [ ! -f /data/snapshot ]; then
echo "Failed to download chain snapshot"
exit 1
fi
mkdir -p /data/chains/${CHAIN_PATH}/
if [ "${SNAPSHOT_FORMAT}" == "7z" ]; then
7z x /data/snapshot -o/data/chains/${CHAIN_PATH}/
mkdir -p /data/chains/${CHAIN_PATH}/db/full
if [ "${SNAPSHOT_FORMAT}" == "lz4" ]; then
wget -O - ${SNAPSHOT_URL} | lz4 -c -d - | tar -x -C /data/chains/${CHAIN_PATH}/
else
tar xvf /data/snapshot --directory=/data/chains/${CHAIN_PATH}/db/full/
wget -O - ${SNAPSHOT_URL} | tar -x -C /data/chains/${CHAIN_PATH}/db/full/
fi
rm /data/snapshot
fi
env:
- name: CHAIN_PATH
Expand All @@ -106,23 +101,18 @@ spec:
args:
- -c
- |
set -eu -o pipefail
if [ -d "/data/relay/chains/${RELAY_CHAIN_PATH}/db" ]; then
echo "Database directory already exists, skipping relay-chain snapshot download"
else
echo "Downloading relay-chain snapshot"
RELAY_SNAPSHOT_URL="{{ .Values.node.collator.relayChainDataSnapshotUrl }}"
wget -O /data/relay-snapshot ${RELAY_SNAPSHOT_URL}
if [ ! -f /data/relay-snapshot ]; then
echo "Failed to download relay-chain snapshot"
exit 1
fi
mkdir -p /data/relay/chains/${RELAY_CHAIN_PATH}/
if [ "${RELAY_SNAPSHOT_FORMAT}" == "7z" ]; then
7z x /data/relay-snapshot -o/data/relay/chains/${RELAY_CHAIN_PATH}/
mkdir -p /data/relay/chains/${RELAY_CHAIN_PATH}/db/full
if [ "${RELAY_SNAPSHOT_FORMAT}" == "lz4" ]; then
wget -O - ${RELAY_SNAPSHOT_URL} | lz4 -c -d - | tar -x -C /data/chains/${RELAY_CHAIN_PATH}/
else
tar xvf /data/relay-snapshot --directory=/data/relay/chains/${RELAY_CHAIN_PATH}/db/full/
wget -O - ${RELAY_SNAPSHOT_URL} | tar -x -C /data/chains/${RELAY_CHAIN_PATH}/db/full/
fi
rm /data/relay-snapshot
fi
env:
- name: RELAY_SNAPSHOT_FORMAT
Expand All @@ -140,6 +130,7 @@ spec:
args:
- -c
- |
set -eu
{{- if .Values.googleCloudSdk.serviceAccountKey }}
gcloud auth activate-service-account --key-file /tmp/service-account-key.json
{{- end }}
Expand Down Expand Up @@ -174,9 +165,10 @@ spec:
args:
- -c
- |
{{- if .Values.googleCloudSdk.serviceAccountKey }}
set -eu
{{- if .Values.googleCloudSdk.serviceAccountKey }}
gcloud auth activate-service-account --key-file /tmp/service-account-key.json
{{- end }}
{{- end }}
if [ -d "/data/relay/chains/${RELAY_CHAIN_PATH}/db" ]; then
echo "Relay-chain database directory already exists, skipping GCS sync"
else
Expand Down Expand Up @@ -208,6 +200,8 @@ spec:
args:
- -c
- |
set -eu -o pipefail
apk add --no-cache wget
{{- if .Values.node.customChainspecUrl }}
{{- if not .Values.node.forceDownloadChainspec }}
if [ ! -f {{ .Values.node.customChainspecPath }} ]; then
Expand All @@ -229,6 +223,8 @@ spec:
volumeMounts:
- mountPath: /data
name: chain-data
securityContext:
runAsUser: 0
{{- end }}
{{- if .Values.node.keys }}
- name: inject-keys
Expand All @@ -237,6 +233,7 @@ spec:
args:
- -c
- |
set -eu
{{- range $keys := .Values.node.keys }}
{{ $.Values.node.command }} key insert --base-path /data \
--chain {{ if $.Values.node.customChainspecUrl }}{{ $.Values.node.customChainspecPath }}{{ else }}${CHAIN}{{ end }} \
Expand All @@ -263,6 +260,7 @@ spec:
args:
- -c
- |
set -eu
{{- range $keys := .Values.node.vault.keys }}
{{ $.Values.node.command }} key insert --base-path /data \
--chain {{ if $.Values.node.customChainspecUrl }}{{ $.Values.node.customChainspecPath }}{{ else }}${CHAIN}{{ end }} \
Expand Down Expand Up @@ -290,6 +288,7 @@ spec:
args:
- -c
- |
set -eu
POD_INDEX="${HOSTNAME##*-}"
RELAY_CHAIN_P2P_PORT="$(kubectl --namespace {{ .Release.Namespace }} get service {{ $fullname }}-${POD_INDEX}-relay-chain-p2p -o jsonpath='{.spec.ports[*].nodePort}')"
echo "${RELAY_CHAIN_P2P_PORT}" > /data/relay_chain_p2p_port
Expand All @@ -316,6 +315,7 @@ spec:
args:
- -c
- |
set -eu
POD_INDEX="${HOSTNAME##*-}"
{{- if or $.Values.node.perNodeServices.relayP2pService.enabled $.Values.node.perNodeServices.paraP2pService.enabled }}
{{- if .Values.node.perNodeServices.setPublicAddressToExternalIp.enabled }}
Expand Down
6 changes: 3 additions & 3 deletions charts/node/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ image:

initContainer:
image:
repository: crazymax/7zip
repository: paritytech/lz4
tag: latest

kubectl:
Expand Down Expand Up @@ -70,7 +70,7 @@ node:
customChainspecPath: "/data/chainspec.json"
forceDownloadChainspec: false
#chainDataSnapshotUrl: "https://dot-rocksdb.polkashots.io/snapshot"
#chainDataSnapshotFormat: 7z
#chainDataSnapshotFormat: lz4
#chainPath: ""
#chainDataKubernetesVolumeSnapshot: ""
#chainDataGcsBucketUrl: ""
Expand All @@ -80,7 +80,7 @@ node:
# relayChainCustomChainspecUrl: ""
relayChainCustomChainspecPath: "/data/relay_chain_chainspec.json"
# relayChainDataSnapshotUrl: "https://dot-rocksdb.polkashots.io/snapshot"
# relayChainDataSnapshotFormat: 7z
# relayChainDataSnapshotFormat: lz4
# relayChainPath: ""
# relayChainDataKubernetesVolumeSnapshot: ""
# relayChainDataGcsBucketUrl: ""
Expand Down

0 comments on commit 858e50a

Please sign in to comment.