diff --git a/docker/fullnode/README.md b/docker/fullnode/README.md index eea6c38606f..6d1ff10bd1e 100644 --- a/docker/fullnode/README.md +++ b/docker/fullnode/README.md @@ -1,6 +1,6 @@ -# Use Docker to Run a Iota Full node Locally +# Use Docker to Run an IOTA Full Node Locally -Follow the steps in this Readme to install and configure a Iota Full node for testing locally using Docker. The instructions were validated on the following operating system/processor combinations: +Use this guide to install and configure an IOTA Full Node locally using Docker for testing. The instructions apply to the following operating system and processor combinations: - Linux/AMD64 - Darwin/AMD64 @@ -8,91 +8,125 @@ Follow the steps in this Readme to install and configure a Iota Full node for te ## Prerequisites -- [Install Docker](https://docs.docker.com/get-docker/) -- [Install Docker Compose](https://docs.docker.com/compose/install/) -- Download the Full node [docker-compose.yaml](https://github.com/iotaledger/iota/blob/develop/docker/fullnode/docker-compose.yaml) file. +Before you begin, ensure you have the following: -## Configure Iota Full node +- [Docker](https://docs.docker.com/get-docker/) installed. +- [Docker Compose](https://docs.docker.com/compose/install/) installed. +- Download the full node [docker-compose.yaml](https://github.com/iotaledger/iota/blob/develop/docker/fullnode/docker-compose.yaml) file. -Download the latest version of the Iota Full node configuration file [fullnode-template.yaml](https://github.com/iotaledger/iota/raw/develop/crates/iota-config/data/fullnode-template.yaml). Use the following command to download the file: +## Configure the IOTA Full Node + +Download the latest version of the IOTA Full Node configuration file [fullnode-template.yaml](https://github.com/iotaledger/iota/raw/develop/crates/iota-config/data/fullnode-template.yaml). You can use the following command to download the file: ```shell wget https://github.com/iotaledger/iota/raw/develop/crates/iota-config/data/fullnode-template.yaml ``` -To ensure you add the appropriate peers for your full node, please refer to the instructions provided at the following link: -https://docs.iota.org/operator/iota-full-node#set-up-from-source and proceed with step 4. - -### Download the Iota genesis blob +### Add Peers -The genesis blob contains the information that defined the Iota network configuration. Before you can start the Full node, you need to download the most recent file to ensure compatibility with the version of Iota you use. +For `Testnet` or `Devnet` nodes, edit the `fullnode.yaml` file to include peer nodes for state synchronization. Append +the following to the end of the current configuration: -To download the appropriate genesis.blob file, please refer to the instructions provided at the following link: -https://docs.iota.org/operator/iota-full-node#set-up-from-source. and proceed with step 3. +#### Devnet -## Start your Iota Full node +```yaml +p2p-config: + seed-peers: + - address: /dns/access-0.r.devnet.iota.cafe/udp/8084 + peer-id: 01589ac910a5993f80fbc34a6e0c8b2041ddc5526a951c838df3037e11ab0188 + - address: /dns/access-1.r.devnet.iota.cafe/udp/8084 + peer-id: 32875c547ea3b44fa08a711646cedb70fa0c97959d236578131505da09723add +``` -Run the following command to start the Iota fullnode in Docker: +#### Testnet -```shell -docker compose up +```yaml +p2p-config: + seed-peers: + - address: /dns/access-0.r.testnet.iota.cafe/udp/8084 + peer-id: 46064108d0b689ed89d1f44153e532bb101ce8f8ca3a3d01ab991d4dea122cfc + - address: /dns/access-1.r.testnet.iota.cafe/udp/8084 + peer-id: 8ffd25fa4e86c30c3f8da7092695e8a103462d7a213b815d77d6da7f0a2a52f5 ``` -**Important:** The commands in this document assume you use Docker Compose V2. The `docker compose` command uses a dash (`docker-compose`) in Docker Compose V1. If you use Docker Compose V1, replace the space in each `docker compose` command with a dash (`docker-compose`). For more information, see [Docker Compose V2](https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command). +### Download the IOTA Genesis Blob + +The genesis blob defines the IOTA network configuration. Before starting the Full Node, download the latest +`genesis.blob` file to ensure compatibility with your version of IOTA. Keep in mind that the [IOTA Mainnet](https://wiki.iota.org/build/networks-endpoints/#iota) runs the +[Stardust Protocol](https://wiki.iota.org/learn/protocols/stardust/introduction/) at the moment, so only Testnet and Devnet blobs are available. -## Test the Iota Full node +- [Testnet genesis blob](https://dbfiles.testnet.iota.cafe/genesis.blob): + `curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob` +- [Devnet genesis blob](https://dbfiles.devnet.iota.cafe/genesis.blob): + `curl -fLJO https://dbfiles.devnet.iota.cafe/genesis.blob` +- [Devnet migration blob](https://dbfiles.devnet.iota.cafe/migration.blob): + `curl -fLJO https://dbfiles.devnet.iota.cafe/migration.blob` -After the Full node starts you can test the JSON-RPC interfaces. +## Start Your IOTA Full Node -## View activity on your local Full node with Iota Explorer +Run the following command to start the IOTA Full Node in Docker: -Iota Explorer supports connecting to a local network. To view activity on your local Full node, open the URL: [https://explorer.iota.org/?network=local](https://explorer.iota.org/?network=local). +```shell +docker compose up +``` -You can also change the network that Iota Explorer connects to by select it in the Iota Explorer interface. +**Important:** These commands assume you are using Docker Compose V2. In Docker Compose V1, the `docker compose` command uses a dash (`docker-compose`). If you use Docker Compose V1, replace the space in each `docker compose` command with a dash (`docker-compose`). For more information, see [Docker Compose V2](https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command). -### Stop the Full node +## Stop the Full Node -Run the following command to stop the Full node when you finish using it: +Run the following command to stop the Full Node when you finish using it: ```shell docker compose stop ``` +## Test the IOTA Full Node + +After the Full Node starts, you can test the [JSON-RPC interfaces](https://docs.iota.org/iota-api-ref). + ## Troubleshooting -If you encounter errors or your Full node stops working, run the commands in the following section to resolve the issue. +If you encounter errors or your Full Node stops working, run the commands in the following section to resolve the issue. -### Start the Full node in detached mode +### Start the Full Node in Detached Mode -First, try starting the Full node in detached mode: +First, try starting the Full Node in detached mode: ```shell docker compose up -d ``` -### Reset the environment +### Reset the Environment -If you continue to see issues, stop the Full node (`docker compose stop`) and delete the Docker container and volume. Then run the following command to start a new instance of the Full node using the same genesis blob. +If you continue to see issues, stop the Full Node (`docker compose stop`) and delete the Docker container and volume. Then run the following command to start a new instance of the Full Node using the same genesis blob: ```shell docker compose down --volumes ``` -### Stats (CPU/MEM USAGE %) +## Monitoring + +### View Activity on Your Local Full Node with IOTA Explorer -To view usage details for the Full node running in Docker, run the following command: +The IOTA Explorer supports connecting to any network as long as it has `https` enabled. To view activity on your local +Full Node, open the URL: [https://explorer.rebased.iota.org/](https://explorer.rebased.iota.org/), and select Custom +RPC URL from the network dropdown in the top right. + +### View Resource Usage (CPU and Memory) + +To view resource usage details for the Full Node running in Docker, run the following command: ```shell docker stats ``` -This command shows a live data stream of the Docker container resource usage, such as CPU and memory. To view data for all containers, use the following command: +This command shows a live data stream of the Docker container's resource usage, such as CPU and memory. To view data for all containers, use the following command: ```shell docker stats -a ``` -### Inspect the state of a running Full node +### Inspect the State of a Running Full Node Get the running container ID: @@ -112,9 +146,9 @@ Inspect the database: ls -la iotadb/ ``` -### Investigate local RPC connectivity issues +### Investigate Local RPC Connectivity Issues -Update the `json-rpc-address` in the Full node config to listen on all addresses: +Update the `json-rpc-address` in the Full Node config to listen on all addresses: ```shell sed -i 's/127.0.0.1/0.0.0.0/' fullnode-template.yaml @@ -124,21 +158,3 @@ sed -i 's/127.0.0.1/0.0.0.0/' fullnode-template.yaml -json-rpc-address: "127.0.0.1:9000" +json-rpc-address: "0.0.0.0:9000" ``` - -### Install wget and curl - -Download each package. For example, on macOS use [homebrew](https://brew.sh/): - -`brew install wget curl` - -### Learn more about Iota - -- https://docs.iota.io/learn - -### Learn more about building and running a Full node from source code - -- https://docs.iota.io/build/fullnode - -### Learn more about Docker Compose - -- https://docs.docker.com/compose/gettingstarted/ diff --git a/docs/content/_snippets/node-hardware-requirements.mdx b/docs/content/_snippets/node-hardware-requirements.mdx new file mode 100644 index 00000000000..efcb34aa220 --- /dev/null +++ b/docs/content/_snippets/node-hardware-requirements.mdx @@ -0,0 +1,7 @@ +:::info Minimum Hardware Requirements + +- CPUs: 8 physical cores / 16 vCPUs +- RAM: 128 GB +- Storage (SSD): 4 TB NVMe drive + +::: \ No newline at end of file diff --git a/docs/content/_snippets/warning-advanced-instructions-node-setup.mdx b/docs/content/_snippets/warning-advanced-instructions-node-setup.mdx new file mode 100644 index 00000000000..cc42330d620 --- /dev/null +++ b/docs/content/_snippets/warning-advanced-instructions-node-setup.mdx @@ -0,0 +1,6 @@ +:::warning Advanced + +These instructions are for setting up a Full node for network participation. If you just need a local development environment, +you should instead follow the instructions in [Create a Local IOTA Network](/developer/getting-started/local-network) to create a local Full node, validators, and faucet. + +::: \ No newline at end of file diff --git a/docs/content/about-iota/iota-architecture/iota-architecture.mdx b/docs/content/about-iota/iota-architecture/iota-architecture.mdx index f6632242f75..0b8885cf916 100644 --- a/docs/content/about-iota/iota-architecture/iota-architecture.mdx +++ b/docs/content/about-iota/iota-architecture/iota-architecture.mdx @@ -37,7 +37,7 @@ The following diagram describes the architectural structure for IOTA's solution. The core components are: - [Execution Layer](../../references/execution-architecture/execution-layer.mdx) -- [IOTA Node](../../operator/iota-full-node.mdx) +- [IOTA Node](../../operator/iota-full-node/overview.mdx) - [IOTA RPC](../../references/iota-api) - [IOTA CLI](../../references/cli) diff --git a/docs/content/developer/advanced/custom-indexer.mdx b/docs/content/developer/advanced/custom-indexer.mdx index 3dcabc17a54..8dc697fd66f 100644 --- a/docs/content/developer/advanced/custom-indexer.mdx +++ b/docs/content/developer/advanced/custom-indexer.mdx @@ -63,7 +63,7 @@ This is suitable for setups with a single ingestion pipeline where progress trac Colocate the data ingestion daemon with a Full node and enable checkpoint dumping on the latter to set up a local stream source. After enabling, the Full node starts dumping executed checkpoints as files to a local directory, and the data ingestion daemon subscribes to changes in the directory through an inotify-like mechanism. This approach allows minimizing ingestion latency (checkpoint are processed immediately after a checkpoint executor on a Full node) and getting rid of dependency on an externally managed bucket. -To enable, add the following to your [Full node configuration](../../operator/iota-full-node.mdx) file: +To enable, add the following to your [Full node configuration](../../operator/iota-full-node/overview.mdx) file: ```yaml checkpoint-executor-config: diff --git a/docs/content/developer/exchange-integration/exchange-integration.mdx b/docs/content/developer/exchange-integration/exchange-integration.mdx index 25c2a2363a2..e3232b4d09c 100644 --- a/docs/content/developer/exchange-integration/exchange-integration.mdx +++ b/docs/content/developer/exchange-integration/exchange-integration.mdx @@ -25,7 +25,7 @@ You can develop and test your integration using a local development server. It w ## Configure a IOTA Full node -You can set up and configure a [IOTA Full node](../../operator/iota-full-node.mdx) using Docker or directly from source code in the IOTA GitHub repository. +You can set up and configure a [IOTA Full node](../../operator/iota-full-node/overview.mdx) using Docker or directly from source code in the IOTA GitHub repository. ## Set up IOTA addresses diff --git a/docs/content/operator/data-management.mdx b/docs/content/operator/data-management.mdx index 2ce78c19488..9cab1da78c9 100644 --- a/docs/content/operator/data-management.mdx +++ b/docs/content/operator/data-management.mdx @@ -6,7 +6,7 @@ description: A high-level description of data management on the IOTA network tha import Quiz from '@site/src/components/Quiz'; import questions from '/json/node-operators/data-management.json'; -Managing the data on your IOTA Full node is critical to ensuring a healthy IOTA network. This topic provides a high-level description of data management on IOTA Full nodes that you can use to optimize your Full node configuration. For more information about IOTA Full nodes, such as pruning policies and archival settings, see [Run an IOTA Full Node](iota-full-node.mdx). +Managing the data on your IOTA Full node is critical to ensuring a healthy IOTA network. This topic provides a high-level description of data management on IOTA Full nodes that you can use to optimize your Full node configuration. For more information about IOTA Full nodes, such as pruning policies and archival settings, see [Run an IOTA Full Node](iota-full-node/overview.mdx). ## Basic IOTA Full node functionality @@ -75,7 +75,7 @@ As described previously, sustainable disk usage requires IOTA Full nodes to prun Both transaction and object pruners run in the background. The logical deletion of entries from RocksDB ultimately triggers the physical compaction of data on disk, which is governed by RocksDB background jobs: the pruning effect on disk usage is not immediate and might take multiple days. -To learn more about object pruning policies, see [Object pruning](iota-full-node.mdx#object-pruning). You can configure the pruner in two modes: +To learn more about object pruning policies, see [Object pruning](iota-full-node/monitoring-and-pruning.mdx#object-pruning). You can configure the pruner in two modes: * **aggressive pruning** (`num-epochs-to-retain: 0`): Preferred option. IOTA prunes old object versions as soon as possible. * **epoch-based pruning** (`num-epochs-to-retain: X`): IOTA prunes old object versions after X epochs. @@ -85,7 +85,7 @@ Testing indicates that aggressive pruning results in more efficient Full Node op ::: -To learn more about transaction pruning policies, see [Transaction pruning](iota-full-node.mdx#transaction-pruning). To configure transaction pruning, specify the `num-epochs-to-retain-for-checkpoints: X` config option. The checkpoints, including their transactions, effects and events are pruned up to X epochs ago. We suggest setting transaction pruning to 2 epochs. +To learn more about transaction pruning policies, see [Transaction pruning](iota-full-node/monitoring-and-pruning.mdx#transaction-pruning). To configure transaction pruning, specify the `num-epochs-to-retain-for-checkpoints: X` config option. The checkpoints, including their transactions, effects and events are pruned up to X epochs ago. We suggest setting transaction pruning to 2 epochs. ### Set an archiving watermark diff --git a/docs/content/operator/genesis.mdx b/docs/content/operator/genesis.mdx index c1239cd27c8..cd5d30a6146 100644 --- a/docs/content/operator/genesis.mdx +++ b/docs/content/operator/genesis.mdx @@ -13,4 +13,4 @@ The genesis.blob files for each network are in: - [Devnet genesis blob](https://dbfiles.devnet.iota.cafe/genesis.blob): `curl -fLJO https://dbfiles.devnet.iota.cafe/genesis.blob` -See [IOTA Full Node](iota-full-node.mdx#setting-up-a-full-node-from-source) for how to get the genesis.blob file for each network. +See [IOTA Full Node](iota-full-node/overview.mdx) for how to get the genesis.blob file for each network. diff --git a/docs/content/operator/iota-full-node.mdx b/docs/content/operator/iota-full-node.mdx deleted file mode 100644 index c66bae14fd7..00000000000 --- a/docs/content/operator/iota-full-node.mdx +++ /dev/null @@ -1,333 +0,0 @@ ---- -title: IOTA Full Node Configuration -description: Operate an IOTA Full node to validate blockchain activities, like transactions, checkpoints, and epoch changes. ---- - -import Quiz from '@site/src/components/Quiz'; -import questions from '/json/node-operators/iota-full-node.json'; - -:::info - -These instructions are for advanced users. If you just need a local development environment, you should instead follow the instructions in [Create a Local IOTA Network](../developer/getting-started/local-network.mdx) to create a local Full node, validators, and faucet. - -::: - -IOTA Full nodes validate blockchain activities, including transactions, checkpoints, and epoch changes. Each Full node stores and services the queries for the blockchain state and history. - -This role enables validators to focus on servicing and processing transactions. When a validator commits a new set of transactions (or a block of transactions), the validator pushes that block to all connected Full nodes that then service the queries from clients. - -## Features - -IOTA Full nodes: -- Track and verify the state of the blockchain, independently and locally. -- Serve read requests from clients. - -## State synchronization -IOTA Full nodes sync with validators to receive new transactions on the network. - -A transaction requires a few round trips to 2f+1 validators to form a transaction certificate (TxCert). - -This synchronization process includes: - -1. Following 2f+1 validators and listening for newly committed transactions. -1. Making sure that 2f+1 validators recognize the transaction and that it reaches finality. -1. Executing the transaction locally and updating the local DB. - -This synchronization process requires listening to at a minimum 2f+1 validators to ensure that a Full node has properly processed all new transactions. IOTA will improve the synchronization process with the introduction of checkpoints and the ability to synchronize with other Full nodes. - -## Architecture - -An IOTA Full node is essentially a read-only view of the network state. Unlike validator nodes, Full nodes cannot sign transactions, although they can validate the integrity of the chain by re-executing transactions that a quorum of validators previously committed. - -Today, an IOTA Full node maintains the full history of the chain. - -Validator nodes store only the latest transactions on the frontier of the object graph (for example, transactions with >0 unspent output objects). - -## Full node setup - -Follow the instructions here to run your own IOTA Full node. - -### Hardware requirements - -Suggested minimum hardware to run an IOTA Full node: - -- CPUs: 8 physical cores / 16 vCPUs -- RAM: 128 GB -- Storage (SSD): 4 TB NVMe drive - -### Software requirements - -IOTA recommends running IOTA Full nodes on Linux. IOTA supports the Ubuntu and Debian distributions. You can also run an IOTA Full node on macOS. - -Make sure to [update Rust](https://doc.rust-lang.org/book/ch01-01-installation.html#updating-and-uninstalling). - -Use the following command to install additional Linux dependencies. - -```shell -sudo apt-get update \ -&& sudo apt-get install -y --no-install-recommends \ -tzdata \ -libprotobuf-dev \ -ca-certificates \ -build-essential \ -libssl-dev \ -libclang-dev \ -libpq-dev \ -pkg-config \ -openssl \ -protobuf-compiler \ -git \ -clang \ -cmake -``` - -## Configure a Full node - -You can configure an IOTA Full node either using Docker or by building from source. - -### Using Docker Compose - -Follow the instructions in the [Full node Docker Readme](https://github.com/iotaledger/iota/tree/develop/docker/fullnode#readme) to run an IOTA Full node using Docker, including [resetting the environment](https://github.com/iotaledger/iota/tree/develop/docker/fullnode#reset-the-environment). - -### Setting up a local IOTA repository - -You must get the latest source files from the IOTA GitHub repository. -1. Set up your fork of the IOTA repository: - 1. Go to the [IOTA repository](https://github.com/iotaledger/iota) on GitHub and click the Fork button in the top right-hand corner of the screen. - 1. Clone your personal fork of the IOTA repository to your local machine (ensure that you insert your GitHub username into the URL): - `git clone https://github.com//iota.git` -1. `cd` into your `iota` repository: - `cd iota` -1. Set up the IOTA repository as a git remote: - `git remote add upstream https://github.com/iotaledger/iota` -1. Sync your fork: - `git fetch upstream` -1. Check out the branch associated with the network version you want to run (for example, `devnet` to run a Devnet Full node): - `git checkout --track upstream/` - -### Setting up a Full node from source - -Open a terminal or console to the `iota` directory you downloaded in the previous steps to complete the following: -1. Install the required prerequisites. -1. Make a copy of the [Full node YAML template](https://github.com/iotaledger/iota/blob/develop/crates/iota-config/data/fullnode-template.yaml): - `cp crates/iota-config/data/fullnode-template.yaml fullnode.yaml` - -1. Download the genesis and migration blob for the network to use: - - [Testnet genesis blob](https://dbfiles.testnet.iota.cafe/genesis.blob): - `curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob` - - [Devnet genesis blob](https://dbfiles.devnet.iota.cafe/genesis.blob): - `curl -fLJO https://dbfiles.devnet.iota.cafe/genesis.blob` - - [Devnet migration blob](https://dbfiles.devnet.iota.cafe/migration.blob): - `curl -fLJO https://dbfiles.devnet.iota.cafe/migration.blob` - - [Mainnet genesis blob](https://github.com/iotaledger/iota/TODO): - `curl -fLJO TODO` - - [Mainnet migration blob](https://github.com/iotaledger/iota/TODO): - `curl -fLJO TODO` -1. For Testnet or Devnet: Edit the `fullnode.yaml` file to include peer nodes for state synchronization. Append the following to the end of the current configuration: - - - - - - ```yaml - p2p-config: - seed-peers: - - address: /dns/access-0.r.testnet.iota.cafe/udp/8084 - peer-id: 46064108d0b689ed89d1f44153e532bb101ce8f8ca3a3d01ab991d4dea122cfc - - address: /dns/access-1.r.testnet.iota.cafe/udp/8084 - peer-id: 8ffd25fa4e86c30c3f8da7092695e8a103462d7a213b815d77d6da7f0a2a52f5 - ``` - - - - - ```yaml - p2p-config: - seed-peers: - - address: /dns/access-0.r.devnet.iota.cafe/udp/8084 - peer-id: 01589ac910a5993f80fbc34a6e0c8b2041ddc5526a951c838df3037e11ab0188 - - address: /dns/access-1.r.devnet.iota.cafe/udp/8084 - peer-id: 32875c547ea3b44fa08a711646cedb70fa0c97959d236578131505da09723add - ``` - - - - - -1. Optional: Skip this step to accept the default paths to resources. Edit the fullnode.yaml file to use custom paths. -1. Update the `db-path` field with the path to the Full node database. - `db-path: "/db-files/iota-fullnode"` -1. Update the `genesis-file-location` with the path to genesis.blob. - ```yaml - genesis: - genesis-file-location: "/iota-fullnode/genesis.blob" - ``` -1. Update the `migration-tx-data-path` with the path to the migration.blob - ```yaml - migration-tx-data-path: "/opt/iota/config/migration.blob" - ``` -### Compiling the node - -Run the following command to compile the `iota-node`. - -```shell -cargo run --release --bin iota-node -``` - -### Starting services - -At this point, your IOTA Full node is ready to connect to the IOTA network. - -1. Open a terminal or console to the iota directory. -1. Start the IOTA Full node: - `./target/release/iota-node --config-path fullnode.yaml` -1. Optional: Publish/subscribe to notifications using JSON-RPC via websocket. - -If your setup is successful, your IOTA Full node is now connected to the appropriate network. - -Your Full node serves the read endpoints of the IOTA JSON-RPC API at: `http://127.0.0.1:9000`. - -### Troubleshooting -If, during the compilation step, you receive a `cannot find -lpq` error, you are missing the `libpq` library. Use `sudo apt-get install libpq-dev` to install on Linux, or `brew install libpq` on MacOS. After you install on MacOS, create a Homebrew link using `brew link --force libpq`. For further context, reference the [issue on Stack Overflow](https://stackoverflow.com/questions/70313347/ld-library-not-found-for-lpq-when-build-rust-in-macos?rq=1). - -If you receive the following error: - -```shell -panicked at error binding to 0.0.0.0:9184: error creating server listener: Address already in use (os error 98) -``` - -Then update the metrics address in your fullnode.yaml file to use port `9180`. - -```yaml -metrics-address: "0.0.0.0:9180" -``` - -## Monitoring - -Monitor your Full node using the instructions in the _Node Monitoring and Metrics_ section. - -The default metrics port is `9184`. To change the port, edit your `fullnode.yaml` file. - -## Update your Full node - -Whenever IOTA releases a new version, you must update your Full node with the release to ensure compatibility with the network it connects to. For example, if you use IOTA Testnet you should install the version of IOTA running on IOTA Testnet. - -### Update with Docker Compose - -Follow the instructions to [reset the environment](https://github.com/iotaledger/iota/tree/develop/docker/fullnode#reset-the-environment), namely by running the command: - -```shell -docker-compose down --volumes -``` - -### Update from source - -If you followed the instructions for Building from Source, use the following steps to update your Full node: - -1. Shut down your running Full node. -1. `cd` into your local IOTA repository: - ```shell - cd iota - ``` -1. Remove the database, 'genesis.blob' and 'migration.blob' file: - ```shell - rm -r iotadb genesis.blob migration.blob - ``` -1. Fetch the source from the latest release: - ```shell - git fetch upstream - ``` -1. Reset your branch: - ```shell - git checkout -B --track upstream/ - ``` -1. Download the latest genesis and migration blob: - - [Testnet genesis blob](https://dbfiles.testnet.iota.cafe/genesis.blob): - `curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob` - - [Devnet genesis blob](https://dbfiles.devnet.iota.cafe/genesis.blob): - `curl -fLJO https://dbfiles.devnet.iota.cafe/genesis.blob` - - [Devnet migration blob](https://dbfiles.devnet.iota.cafe/migration.blob): - `curl -fLJO https://dbfiles.devnet.iota.cafe/migration.blob` -1. Update your `fullnode.yaml` configuration file, if needed. -1. Recompile your IOTA Full node with the same command as in [Compiling the node](#compiling-the-node): -1. Restart your IOTA Full node: - ```shell - ./target/release/iota-node --config-path fullnode.yaml - ``` - -Your Full node starts on: http://127.0.0.1:9000. - -## Object pruning - -IOTA adds new object versions to the database as part of transaction execution. This makes previous versions ready for -garbage collection. However, without pruning, this can result in database performance degradation and requires large -amounts of storage space. IOTA identifies the objects that are eligible for pruning in each checkpoint, and then performs -the pruning in the background. - -You can enable pruning for an IOTA node by adding the `authority-store-pruning-config` config to `fullnode.yaml` file: - -```yaml -authority-store-pruning-config: - # Number of epoch dbs to keep - # Not relevant for object pruning - num-latest-epoch-dbs-to-retain: 3 - # The amount of time, in seconds, between running the object pruning task. - # Not relevant for object pruning - epoch-db-pruning-period-secs: 3600 - # Number of epochs to wait before performing object pruning. - # When set to 0, IOTA prunes old object versions as soon - # as possible. This is also called *aggressive pruning*, and results in the most effective - # garbage collection method with the lowest disk usage possible. - # This is the recommended setting for IOTA Validator nodes since older object versions aren't - # necessary to execute transactions. - # When set to 1, IOTA prunes only object versions from transaction checkpoints - # previous to the current epoch. In general, when set to N (where N >= 1), IOTA prunes - # only object versions from checkpoints up to `current - N` epoch. - # It is therefore possible to have multiple versions of an object present - # in the database. This setting is recommended for IOTA Full nodes as they might need to serve - # RPC requests that require looking up objects by ID and Version (rather than just the latest - # version). However, if your Full node does not serve RPC requests you should then also enable - # aggressive pruning. - num-epochs-to-retain: 0 - # Advanced setting: Maximum number of checkpoints to prune in a batch. The default - # settings are appropriate for most use cases. - max-checkpoints-in-batch: 10 - # Advanced setting: Maximum number of transactions in one batch of pruning run. The default - # settings are appropriate for most use cases. - max-transactions-in-batch: 1000 -``` -## Transaction pruning - -Transaction pruning removes previous transactions and effects from the database. -IOTA periodically creates checkpoints. Each checkpoint contains the transactions that occurred during the checkpoint and their associated effects. - -IOTA performs transaction pruning in the background after checkpoints complete. - -You can enable transaction pruning for your Full node or Validator node by adding `num-epochs-to-retain-for-checkpoints` -to the `authority-store-pruning-config` config for the node: - -```yaml -authority-store-pruning-config: - num-latest-epoch-dbs-to-retain: 3 - epoch-db-pruning-period-secs: 3600 - num-epochs-to-retain: 0 - max-checkpoints-in-batch: 10 - max-transactions-in-batch: 1000 - # Number of epochs to wait before performing transaction pruning. - # When this is N (where N >= 2), IOTA prunes transactions and effects from - # checkpoints up to the `current - N` epoch. IOTA never prunes transactions and effects from the current and - # immediately prior epoch. N = 2 is a recommended setting for IOTA Validator nodes and IOTA Full nodes that don't - # serve RPC requests. - num-epochs-to-retain-for-checkpoints: 2 - # Ensures that individual database files periodically go through the compaction process. - # This helps reclaim disk space and avoid fragmentation issues - periodic-compaction-threshold-days: 1 -``` - -:::info - -If you prune transactions, Archival nodes can help ensure lagging peer nodes don't lose any information. For more information, see [IOTA Archives](archives.mdx). - -::: - - \ No newline at end of file diff --git a/docs/content/operator/iota-full-node/docker.mdx b/docs/content/operator/iota-full-node/docker.mdx new file mode 100644 index 00000000000..da8706fedaa --- /dev/null +++ b/docs/content/operator/iota-full-node/docker.mdx @@ -0,0 +1,18 @@ +--- +description: Learn how to set up an IOTA Full Node using docker. +--- +import Quiz from '@site/src/components/Quiz'; +import questions from '/json/node-operators/iota-full-node/node-setup.json'; +import Docker from './../../../../docker/fullnode/README.md'; +import WarningAdvanced from './../../_snippets/warning-advanced-instructions-node-setup.mdx' +import NodeHardwareRequirements from './../../_snippets/node-hardware-requirements.mdx' + +# Set Up a Full Node With Docker + + + + + + + + \ No newline at end of file diff --git a/docs/content/operator/iota-full-node/monitoring-and-pruning.mdx b/docs/content/operator/iota-full-node/monitoring-and-pruning.mdx new file mode 100644 index 00000000000..0dd0a5d15f8 --- /dev/null +++ b/docs/content/operator/iota-full-node/monitoring-and-pruning.mdx @@ -0,0 +1,82 @@ +--- +description: How to monitor and Prune your IOTA Full Node. +--- +import Quiz from '@site/src/components/Quiz'; +import questions from '/json/node-operators/iota-full-node/pruning.json'; + +# Monitoring and Pruning + +## Monitoring + +Monitor your Full node using the instructions in the [Node Monitoring and Metrics section](../telemetry/iota-metrics.mdx). + +The default metrics port is `9184`. To change the port, edit your `fullnode.yaml` file. + +## Object Pruning + +IOTA adds new object versions to the database as part of transaction execution. This makes previous versions ready for +garbage collection. However, without pruning, this can result in database performance degradation and requires large +amounts of storage space. IOTA identifies the objects that are eligible for pruning in each checkpoint, and then performs +the pruning in the background. + +You can enable pruning for an IOTA node by adding the `authority-store-pruning-config` config to `fullnode.yaml` file: + +```yaml +authority-store-pruning-config: + # Number of epoch dbs to keep + # Not relevant for object pruning + num-latest-epoch-dbs-to-retain: 3 + # The amount of time, in seconds, between running the object pruning task. + # Not relevant for object pruning + epoch-db-pruning-period-secs: 3600 + # Number of epochs to wait before performing object pruning. + # When set to 0, IOTA prunes old object versions as soon + # as possible. This is also called *aggressive pruning*, and results in the most effective + # garbage collection method with the lowest disk usage possible. + # This is the recommended setting for IOTA Validator nodes since older object versions aren't + # necessary to execute transactions. + # When set to 1, IOTA prunes only object versions from transaction checkpoints + # previous to the current epoch. In general, when set to N (where N >= 1), IOTA prunes + # only object versions from checkpoints up to `current - N` epoch. + # It is therefore possible to have multiple versions of an object present + # in the database. This setting is recommended for IOTA Full nodes as they might need to serve + # RPC requests that require looking up objects by ID and Version (rather than just the latest + # version). However, if your Full node does not serve RPC requests you should then also enable + # aggressive pruning. + num-epochs-to-retain: 0 + # Advanced setting: Maximum number of checkpoints to prune in a batch. The default + # settings are appropriate for most use cases. + max-checkpoints-in-batch: 10 + # Advanced setting: Maximum number of transactions in one batch of pruning run. The default + # settings are appropriate for most use cases. + max-transactions-in-batch: 1000 +``` +## Transaction pruning + +Transaction pruning removes previous transactions and effects from the database. +IOTA periodically creates checkpoints. Each checkpoint contains the transactions that occurred during the checkpoint and their associated effects. + +IOTA performs transaction pruning in the background after checkpoints complete. + +You can enable transaction pruning for your Full node or Validator node by adding `num-epochs-to-retain-for-checkpoints` +to the `authority-store-pruning-config` config for the node: + +```yaml +authority-store-pruning-config: + num-latest-epoch-dbs-to-retain: 3 + epoch-db-pruning-period-secs: 3600 + num-epochs-to-retain: 0 + max-checkpoints-in-batch: 10 + max-transactions-in-batch: 1000 + # Number of epochs to wait before performing transaction pruning. + # When this is N (where N >= 2), IOTA prunes transactions and effects from + # checkpoints up to the `current - N` epoch. IOTA never prunes transactions and effects from the current and + # immediately prior epoch. N = 2 is a recommended setting for IOTA Validator nodes and IOTA Full nodes that don't + # serve RPC requests. + num-epochs-to-retain-for-checkpoints: 2 + # Ensures that individual database files periodically go through the compaction process. + # This helps reclaim disk space and avoid fragmentation issues + periodic-compaction-threshold-days: 1 +``` + + \ No newline at end of file diff --git a/docs/content/operator/iota-full-node/overview.mdx b/docs/content/operator/iota-full-node/overview.mdx new file mode 100644 index 00000000000..00f91922651 --- /dev/null +++ b/docs/content/operator/iota-full-node/overview.mdx @@ -0,0 +1,49 @@ +--- +title: Overview +description: Operate an IOTA Full Node to validate blockchain activities, like transactions, checkpoints, and epoch changes. +--- +import Quiz from '@site/src/components/Quiz'; +import questions from '/json/node-operators/iota-full-node/overview.json'; +import WarningAdvanced from './../../_snippets/warning-advanced-instructions-node-setup.mdx' + +# Overview + + + +IOTA Full nodes validate blockchain activities, including transactions, checkpoints, and epoch changes. Each Full node stores and services the queries for the blockchain state and history. + +This role enables validators to focus on servicing and processing transactions. When a validator commits a new set of transactions (or a block of transactions), the validator pushes that block to all connected Full nodes that then service the queries from clients. + +## Features + +IOTA Full nodes: +- Track and verify the state of the blockchain, independently and locally. +- Serve read requests from clients. + +## State Synchronization + +IOTA Full nodes sync with validators to receive new transactions on the network. + +A transaction requires a few round trips to 2f+1 validators to form a transaction certificate (TxCert). + +This synchronization process includes: + +1. Following 2f+1 validators and listening for newly committed transactions. +1. Making sure that 2f+1 validators recognize the transaction and that it reaches finality. +1. Executing the transaction locally and updating the local DB. + +This synchronization process requires listening to at a minimum 2f+1 validators to ensure that a Full node has properly processed all new transactions. IOTA will improve the synchronization process with the introduction of checkpoints and the ability to synchronize with other Full nodes. + +## Architecture + +An IOTA Full node is essentially a read-only view of the network state. Unlike validator nodes, Full nodes cannot sign transactions, although they can validate the integrity of the chain by re-executing transactions that a quorum of validators previously committed. + +Today, an IOTA Full node maintains the full history of the chain. + +Validator nodes store only the latest transactions on the frontier of the object graph (for example, transactions with >0 unspent output objects). + +## Set Up a Full Node + +You can set up your IOTA Full Node [using Docker](docker.mdx) or [build it from source](source.mdx). + + \ No newline at end of file diff --git a/docs/content/operator/iota-full-node/source.mdx b/docs/content/operator/iota-full-node/source.mdx new file mode 100644 index 00000000000..fc8f03947a2 --- /dev/null +++ b/docs/content/operator/iota-full-node/source.mdx @@ -0,0 +1,226 @@ +--- +description: Learn how to set up an IOTA Full node from source. +title: Build From Source +--- +import Quiz from '@site/src/components/Quiz'; +import questions from '/json/node-operators/iota-full-node/node-setup.json'; +import WarningAdvanced from './../../_snippets/warning-advanced-instructions-node-setup.mdx' +import NodeHardwareRequirements from './../../_snippets/node-hardware-requirements.mdx' + + + + + +## Software requirements + +IOTA recommends running IOTA Full nodes on Linux. IOTA supports the Ubuntu and Debian distributions. You can also run an IOTA Full node on macOS. + +Make sure to [update Rust](https://www.rust-lang.org/tools/install). + +Use the following command to install additional Linux dependencies. + +```shell +sudo apt-get update \ +&& sudo apt-get install -y --no-install-recommends \ +tzdata \ +libprotobuf-dev \ +ca-certificates \ +build-essential \ +libssl-dev \ +libclang-dev \ +libpq-dev \ +pkg-config \ +openssl \ +protobuf-compiler \ +git \ +clang \ +cmake +``` + +## 1. Clone the IOTA repository + +1. The first thing you need to do to set up your Full Node is clone the IOTA repository, you can use the following command +or download it from [GitHub](https://github.com/iotaledger/iota): + +```bash +git clone https://github.com/iotaledger/iota.git +``` + +2. Once you have download the source code, move into the newly generated `iota` directory and check out the branch + associated to the [network](../../developer/getting-started/connect.mdx#iota-networks) you want to run, for example, `testnet`: + +```bash +git checkout testnet +``` + +## 2. Configure the IOTA Full Node + +Open a terminal or console in the `iota` directory you downloaded in the previous steps to complete the following: + +1. Install the required dependencies by running the following command: + +```bash +cargo install +``` + +2. Make a copy of the [Full node YAML template](https://github.com/iotaledger/iota/blob/develop/crates/iota-config/data/fullnode-template.yaml): + +```bash +cp crates/iota-config/data/fullnode-template.yaml fullnode.yaml +``` + +3. Download the genesis and migration blob for the network to use: + + - [Testnet genesis blob](https://dbfiles.testnet.iota.cafe/genesis.blob): + `curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob` + - [Devnet genesis blob](https://dbfiles.devnet.iota.cafe/genesis.blob): + `curl -fLJO https://dbfiles.devnet.iota.cafe/genesis.blob` + - [Devnet migration blob](https://dbfiles.devnet.iota.cafe/migration.blob): + `curl -fLJO https://dbfiles.devnet.iota.cafe/migration.blob` + - [Mainnet genesis blob](https://github.com/iotaledger/iota/TODO): + `curl -fLJO TODO` + - [Mainnet migration blob](https://github.com/iotaledger/iota/TODO): + `curl -fLJO TODO` + +4. (For Testnet or Devnet) Edit the `fullnode.yaml` file to include peer nodes for state synchronization. Append the following to the end of the current configuration: + + + + + + ```yaml + p2p-config: + seed-peers: + - address: /dns/access-0.r.testnet.iota.cafe/udp/8084 + peer-id: 46064108d0b689ed89d1f44153e532bb101ce8f8ca3a3d01ab991d4dea122cfc + - address: /dns/access-1.r.testnet.iota.cafe/udp/8084 + peer-id: 8ffd25fa4e86c30c3f8da7092695e8a103462d7a213b815d77d6da7f0a2a52f5 + ``` + + + + + ```yaml + p2p-config: + seed-peers: + - address: /dns/access-0.r.devnet.iota.cafe/udp/8084 + peer-id: 01589ac910a5993f80fbc34a6e0c8b2041ddc5526a951c838df3037e11ab0188 + - address: /dns/access-1.r.devnet.iota.cafe/udp/8084 + peer-id: 32875c547ea3b44fa08a711646cedb70fa0c97959d236578131505da09723add + ``` + + + + + +5. (Optional, skip this step to accept the default paths to resources). Edit the `fullnode.yaml` file to use custom paths. + 1. Update the `db-path` field with the path to the Full node database. + ```yaml + db-path: "/db-files/iota-fullnode"` + ``` + 2. Update the `genesis-file-location` with the path to genesis.blob. + ```yaml + genesis: + genesis-file-location: "/iota-fullnode/genesis.blob" + ``` + 3. Update the `migration-tx-data-path` with the path to the migration.blob + ```yaml + migration-tx-data-path: "/opt/iota/config/migration.blob" + ``` + +## 3. Compile the Node + +Run the following command to compile the `iota-node`. + +```shell +cargo run --release --bin iota-node +``` + +## 4. Start Services + +At this point, your IOTA Full node is ready to connect to the IOTA network. + +1. Open a terminal or console to the `iota` directory and run the following command to start your node: + +```bash +./target/release/iota-node --config-path fullnode.yaml +``` + +2. (Optional) Publish/subscribe to notifications using JSON-RPC via websocket. + +If your setup is successful, your IOTA Full node is now connected to the appropriate network. + +Your Full node serves the read endpoints of the IOTA JSON-RPC API at: `http://127.0.0.1:9000`. + +## Troubleshooting + +### `cannot find -lpq` + +If, during the compilation step, you receive a `cannot find -lpq` error, you are missing the `libpq` library. +Use `sudo apt-get install libpq-dev` to install on Linux, or `brew install libpq` on MacOS. +After you install on MacOS, create a Homebrew link using `brew link --force libpq`. +For further context, reference the [issue on Stack Overflow](https://stackoverflow.com/questions/70313347/ld-library-not-found-for-lpq-when-build-rust-in-macos?rq=1). + +### Listener: Address Already In Use + +If you receive the following error: + +```shell +panicked at error binding to 0.0.0.0:9184: error creating server listener: Address already in use (os error 98) +``` + +Update the metrics address in your fullnode.yaml file to use port `9180`. + +```yaml +metrics-address: "0.0.0.0:9180" +``` + +## Update Your Full node + +Whenever IOTA releases a new version, you must update your Full node with the release to ensure compatibility with the +network it connects to. +For example, if you use IOTA Testnet you should install the version of IOTA running on IOTA Testnet. + +Use the following steps to update your Full node: + +1. Shut down your running Full node. +2. Move into the directory for your local `iota` repository and remove the database, 'genesis.blob' and 'migration.blob' + files: + + ```shell + rm -r iotadb genesis.blob migration.blob + ``` +3. Fetch the latest changes by pulling from the [IOTA repository](https://github.com/iotaledger/iota): + + ```shell + git pull + ``` + +4. Download the latest genesis and migration blob: + - [Testnet genesis blob](https://dbfiles.testnet.iota.cafe/genesis.blob): + `curl -fLJO https://dbfiles.testnet.iota.cafe/genesis.blob` + - [Devnet genesis blob](https://dbfiles.devnet.iota.cafe/genesis.blob): + `curl -fLJO https://dbfiles.devnet.iota.cafe/genesis.blob` + - [Devnet migration blob](https://dbfiles.devnet.iota.cafe/migration.blob): + `curl -fLJO https://dbfiles.devnet.iota.cafe/migration.blob` +5. (Optional) Update your `fullnode.yaml` configuration file, if needed. +6. Recompile your IOTA Full node: + + ```bash + cargo run --release --bin iota-node + ``` + +7. Restart your IOTA Full node: + + ```shell + ./target/release/iota-node --config-path fullnode.yaml + ``` + +Your Full node restarts on: http://127.0.0.1:9000. + +:::info + +If you prune transactions, Archival nodes can help ensure lagging peer nodes don't lose any information. For more information, see [IOTA Archives](../archives.mdx). + +::: + \ No newline at end of file diff --git a/docs/content/operator/operator.mdx b/docs/content/operator/operator.mdx index f5166dd92aa..31bf74390f1 100644 --- a/docs/content/operator/operator.mdx +++ b/docs/content/operator/operator.mdx @@ -9,7 +9,7 @@ Operator guides demonstrate how to run Full nodes on IOTA, whether as a validato Guides that inform Full node operators include: -- [IOTA Full Node Configuration](iota-full-node.mdx) +- [IOTA Full Node Configuration](iota-full-node/overview.mdx) ## IOTA validators diff --git a/docs/content/operator/validator-operation/ansible/README.mdx b/docs/content/operator/validator-operation/ansible/README.mdx index cf820fbeb6e..29c76e118c2 100644 --- a/docs/content/operator/validator-operation/ansible/README.mdx +++ b/docs/content/operator/validator-operation/ansible/README.mdx @@ -2,9 +2,9 @@ sidebar_label: Ansible Configuration --- - import Configuration from './../../../../../nre/ansible/README.md'; + # Ansible Configuration diff --git a/docs/content/sidebars/operator.js b/docs/content/sidebars/operator.js index a244da04ff4..8b785c58e68 100644 --- a/docs/content/sidebars/operator.js +++ b/docs/content/sidebars/operator.js @@ -4,7 +4,16 @@ const operator = [ 'operator/operator', - 'operator/iota-full-node', + { + type: 'category', + label: 'IOTA Full Node Configuration', + items: [ + 'operator/iota-full-node/overview', + 'operator/iota-full-node/docker', + 'operator/iota-full-node/source', + 'operator/iota-full-node/monitoring-and-pruning', + ], + }, 'operator/validator-config', 'operator/data-management', 'operator/snapshots', diff --git a/docs/site/src/pages/index.js b/docs/site/src/pages/index.js index 7cfe41a3912..1919f7e9b1a 100644 --- a/docs/site/src/pages/index.js +++ b/docs/site/src/pages/index.js @@ -116,7 +116,7 @@ export default function Home() { Run a IOTA Full node diff --git a/docs/site/static/json/node-operators/iota-full-node.json b/docs/site/static/json/node-operators/iota-full-node/node-setup.json similarity index 100% rename from docs/site/static/json/node-operators/iota-full-node.json rename to docs/site/static/json/node-operators/iota-full-node/node-setup.json diff --git a/docs/site/static/json/node-operators/iota-full-node/overview.json b/docs/site/static/json/node-operators/iota-full-node/overview.json new file mode 100644 index 00000000000..e7ca0283c59 --- /dev/null +++ b/docs/site/static/json/node-operators/iota-full-node/overview.json @@ -0,0 +1,11 @@ +[ + { + "questionText": "In the context of IOTA Full nodes, what is the significance of the 2f+1 validators during transaction processing?", + "answerOptions": [ + { "answerText": "They ensure that transactions are signed before being sent to the network.", "isCorrect": false }, + { "answerText": "They help in achieving consensus and finality for a transaction.", "isCorrect": true }, + { "answerText": "They manage the storage of blockchain data across nodes.", "isCorrect": false }, + { "answerText": "They are responsible for creating new epochs in the IOTA network.", "isCorrect": false } + ] + } +] diff --git a/docs/site/static/json/node-operators/iota-full-node/pruning.json b/docs/site/static/json/node-operators/iota-full-node/pruning.json new file mode 100644 index 00000000000..5cc66342e02 --- /dev/null +++ b/docs/site/static/json/node-operators/iota-full-node/pruning.json @@ -0,0 +1,11 @@ +[ + { + "questionText": "In the pruning configuration for an IOTA Full node, why would you set `num-epochs-to-retain` to 0?", + "answerOptions": [ + { "answerText": "To retain all object versions indefinitely.", "isCorrect": false }, + { "answerText": "To enable aggressive pruning of old object versions as soon as possible.", "isCorrect": true }, + { "answerText": "To keep only the latest transaction effects.", "isCorrect": false }, + { "answerText": "To disable pruning entirely.", "isCorrect": false } + ] + } +]