Execute cardano-cli and cardano-node to verify output as below (the exact version and git rev should depend on your checkout tag on github repository):
cardano-cli version
-# cardano-cli 9.x.x - linux-x86_64 - ghc-8.10
+# cardano-cli 10.x.x - linux-x86_64 - ghc-8.10
# git rev <...>
cardano-node version
-# cardano-node 9.x.x - linux-x86_64 - ghc-8.10
+# cardano-node 10.x.x - linux-x86_64 - ghc-8.10
# git rev <...>
The format is based on Keep a Changelog, and this adheres to Semantic Versioning.
+test_koios call from cntools.library to cntools.shdialog by default, it is an optional component - and no longer installed by default.--whole-utxo flag, as it returns all address and will not accept --address--cold-verification-key-file instead of --verification-key-filepool >> show stake distribution showing up as always 0.This documentation site (rather the repository itself) is created by some of the well known and experienced community members and contains instructions/information about various guild tools which simplify various stake-ops (setting up, managing and monitoring pools) for operators. Note that the guides are present to help you simplify your tasks - but as an entity responsible for creating blocks on a financial platform, we expect some basic pre-requisite skill sets - at professional level - before entering the portal:
cardano-cli, and have worked on preview/preprod/guild networks for pool operations without use of wrapper scripts - as an education exercise;Everyone is welcome to contribute to the repository (via documentation, testing, code, videos, etc). Our aim is to work together and reduce confusion rather than hosting 100 versions of documentation - each marketing their pool in a way.
"},{"location":"#support","title":"Support","text":"The Telegram Support channel is used to announce new releases and changes to the code base. This is also the place to ask general questions regarding the documentation and scripts on this site.
To report bugs and issues with scripts and documentation please open a GitHub Issue. Feature requests are best opened as a discussion thread.
"},{"location":"#getting-started","title":"Getting Started","text":"Use the sidebar to navigate through the topics. Note that the instructions assume the folder structure as per here.
Again, Feedback/Contribution and ownership of tasks is always welcome. If you're interested in collaborating regularly, make a start - and you should be part of the guild already .
"},{"location":"basics/","title":"Basics","text":""},{"location":"basics/#architecture","title":"Architecture","text":"The architecture for various components are already described at docs.cardano.org by CF/IOHK. We will not reinvent the wheel
"},{"location":"basics/#manual-software-pre-requirements","title":"Manual Software Pre-Requirements","text":"While we do not intend to hand out step-by-step instructions, the tools are often misused as a shortcut to avoid ensuring base skillsets mentioned on home page. Some of the common gotchas that we often find SPOs to miss out on:
Reminder !!
You're expected to run the commands below from same session, using same working directories as indicated and using a non-root user with sudo access. You are expected to be familiar with this as part of pre-requisite skill sets for stake pool operators.
The pre-requisites for Linux systems are automated to be executed as a single script. This script uses opt-in election of what you'd like the script to do. The defaults without any arguments will only update static part of script contents for you. To download the pre-requisites scripts, execute the below:
mkdir \"$HOME/tmp\";cd \"$HOME/tmp\"\n# Install curl\n# CentOS / RedHat - sudo dnf -y install curl\n# Ubuntu / Debian - sudo apt -y install curl\ncurl -sS -o guild-deploy.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/guild-deploy.sh\nchmod 755 guild-deploy.sh\nImportant !!
Please familiarise with the syntax of guild-deploy.sh before proceeding (using -h as below). The exact parameters you want to run with is dependent on your, below are only sample instructions
The usage syntax can be checked using ./guild-deploy.sh -h , sample output below:
Usage: guild-deploy.sh [-n <mainnet|guild|preprod|preview|sanchonet>] [-p path] [-t <name>] [-b <branch>] [-u] [-s [p][b][l][m][d][c][o][w][x][f][s]]\nSet up dependencies for building/using common tools across cardano ecosystem.\nThe script will always update dynamic content from existing scripts retaining existing user variables\n\n-n Connect to specified network instead of mainnet network (Default: connect to cardano mainnet network) eg: -n guild\n-p Parent folder path underneath which the top-level folder will be created (Default: /opt/cardano)\n-t Alternate name for top level folder - only alpha-numeric chars allowed (Default: cnode)\n-b Use alternate branch of scripts to download - only recommended for testing/development (Default: master)\n-u Skip update check for script itself\n-s Selective Install, only deploy specific components as below:\n p Install common pre-requisite OS-level Dependencies for most tools on this repo (Default: skip)\nb Install OS level dependencies for tools required while building cardano-node/cardano-db-sync components (Default: skip)\nl Build and Install libsodium fork from IO repositories (Default: skip)\nm Download latest (released) binaries for mithril-signer, mithril-client (Default: skip)\nd Download latest (released) binaries for bech32, cardano-address, cardano-node, cardano-cli, cardano-db-sync and cardano-submit-api (Default: skip)\nc Download latest (released) binaries for CNCLI (Default: skip)\no Download latest (released) binaries for Ogmios (Default: skip)\nw Download latest (released) binaries for Cardano Hardware CLI (Default: skip)\nx Download latest (released) binaries for Cardano Signer binary (Default: skip)\nf Force overwrite config files (backups of existing ones will be created) (Default: skip)\ns Force overwrite entire content [including user variables] of scripts (Default: skip)\nglibc, it would likely be due to the build mismatch between pre-compiled binary and your OS, which is not uncommon. You may need to compile cncli manually on your OS as per instructions here - make sure to copy the output binary to \"${HOME}/.local/bin\" folder.A typical example install to install most components but not overwrite static part of existing files for preview network would be:
./guild-deploy.sh -b master -n preview -t cnode -s pdlcowx\n. \"${HOME}/.bashrc\"\nIf instead of download, you'd want to build the components yourself, you could use:
./guild-deploy.sh -b master -n preview -t cnode -s pblcowx\n. \"${HOME}/.bashrc\"\nLastly, if you'd want to update your scripts but not install any additional dependencies, you may simply run:
./guild-deploy.sh -b master -n preview -t cnode\nRunning the script above will create the folder structure as per below, for your reference. You do NOT require CNODE_HOME to be set on shell level as scripts will derive the parent folder and assign it at runtime, the addition in ~/.bashrc is only for your ease to switch to right folder:
/opt/cardano/cnode # Top-Level Folder\n\u251c\u2500\u2500 ...\n\u251c\u2500\u2500 files # Config, genesis and topology files\n\u2502 \u251c\u2500\u2500 ...\n\u2502 \u251c\u2500\u2500 byron-genesis.json # Byron Genesis file referenced in config.json\n\u2502 \u251c\u2500\u2500 shelley-genesis.json # Genesis file referenced in config.json\n\u2502 \u251c\u2500\u2500 alonzo-genesis.json # Alonzo Genesis file referenced in config.json\n\u2502 \u251c\u2500\u2500 config.json # Config file used by cardano-node\n\u2502 \u2514\u2500\u2500 topology.json # Map of chain for cardano-node to boot from\n\u251c\u2500\u2500 db # DB Store for cardano-node\n\u251c\u2500\u2500 guild-db # DB Store for guild-specific tools and additions (eg: cncli, cardano-db-sync's schema)\n\u251c\u2500\u2500 logs # Logs for cardano-node\n\u251c\u2500\u2500 priv # Folder to store your keys (permission: 600)\n\u251c\u2500\u2500 scripts # Scripts to start and interact with cardano-node\n\u2514\u2500\u2500 sockets # Socket files created by cardano-node\nThe documentation here uses instructions from Intersect MBO repositories as foundation, with additional info which we can contribute to where appropriate. Note that not everyone needs to build each component. You can refer to architecture to understand and qualify which of the components built by IO you want to run.
"},{"location":"build/#components","title":"Components","text":"For most Pool Operators, simply building cardano-node should be enough. Use the below to decide whether you need other components:
graph TB A([Interact with HD Walletslocally]) B([Explore blockchainlocally]) C([Easy pool-ops andfund management]) D([Create Custom Assets]) E([Monitor node using Terminal UI]) F([Sign/verify any datausing crypto keys]) N(Node) O(Ogmios) P(gRest/Koios) Q(DBSync) R(Wallet) S(CNTools) T(Tx Submit API) U(GraphQL) V(OfflineMetadataTools) X(gLiveView) Y(cardano-signer) Z[(PostgreSQL)] N --x C --x S N --x D --x S & V N --x E --x X N --x B B --x U --x Q B --x P --x Q P --x O P --x T F ---x Y N --x A --x R Q --x ZImportant
We strongly prefer use of gRest over GraphQL components due to performance, security, simplicity, control and most importantly - consistency benefits. Please refer to official documentations if you're interested in GraphQL or Cardano-Rest components instead.
Note
The instructions are intentionally limited to stack/cabal** to avoid wait times/availability of nix/docker files on a rapidly developing codebase - this also helps us prevent managing multiple versions of instructions.
"},{"location":"build/#description-for-components-built-by-community","title":"Description for components built by community","text":""},{"location":"build/#cntools","title":"CNTools","text":"A swiss army knife for pool operators, primarily built by Ola, to simplify typical operations regarding their wallet keys and pool management. You can read more about it here
"},{"location":"build/#gliveview","title":"gLiveView","text":"A local node monitoring tool, primarily built by Ola, to use in addition to remote monitoring tools like Prometheus/Grafana, Zabbix or IOG's RTView. This is especially useful when moving to a systemd deployment - if you haven't done so already - as it offers an intuitive UI to monitor the node status. You can read more about it here
"},{"location":"build/#topology-updater","title":"Topology Updater","text":"A temporary node-to-node discovery solution, run by Markus, that was started initially to bridge the gap created while awaiting completion of P2P on cardano network, but has since become an important lifeline to the network health - to allow everyone to activate their relay nodes without having to postpone and wait for manual topology completion requests. You can read more about it here
"},{"location":"build/#koiosgrest","title":"Koios/gRest","text":"A full-featured local query layer node to explore blockchain data (via dbsync) using standardised pre-built queries served via API as per standard from Koios - for which user can opt to participate in elastic query layer. You can read more about build steps here and reference API endpoints here
"},{"location":"build/#ogmios","title":"Ogmios","text":"A lightweight bridge interface for cardano-node. It offers a WebSockets API that enables local clients to speak Ouroboros' mini-protocols via JSON/RPC. You can read more about it here
"},{"location":"build/#cncli","title":"CNCLI","text":"A CLI tool written in Rust by Andrew Westberg for low-level communication with cardano-node. It is commonly used by SPOs to check their leader logs (integrates with CNTools as well as gLiveView) or to send their pool's health information to https://pooltool.io. You can read more about it here
"},{"location":"build/#cardano-signer","title":"Cardano Signer","text":"A tool written by Martin to sign/verify data (hex, text or binary) using cryptographic keys to generate data as per CIP-8 or CIP-36 standards. You can read more about it here
"},{"location":"catalystf11/","title":"Catalystf11","text":""},{"location":"catalystf11/#marlowehub-unifying-platform-for-marlowe-smart-contracts-phase-1-smart-contracts","title":"MarloweHub: Unifying Platform for Marlowe Smart Contracts - Phase 1 - Smart Contracts","text":"Category: Concept Applicant: mike (pooltool.io) Requested funds: \u20b3100,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#adastat-cardano-explorer-open-source-improved-reboot-towards-a-first-class-community-blockchain-explorer","title":"AdaStat Cardano Explorer - Open Source Improved Reboot towards a first-class community blockchain explorer","text":"Category: Product Applicant: Dmytro Stashenko (adastat.net) Requested funds: \u20b3180,300.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#adahold-decentralized-price-can-only-go-up-token-solution-for-true-ada-hodlers-smart-contract","title":"AdaHold: Decentralized Price-Can-Only-Go-Up Token, Solution For TRUE Ada Hodlers - Smart Contract","text":"Category: Concept Applicant: Dmytro Stashenko (adastat.net) Requested funds: \u20b399,700.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#sundae-labs-next-gen-uplc-debugger-with-aiken-integration","title":"Sundae Labs Next-Gen UPLC Debugger with Aiken Integration","text":"Category: Developers Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3140,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#margin-pool-enhanced-liquidity-for-margin-trading-on-sundaeswap","title":"Margin Pool: Enhanced Liquidity for Margin Trading on SundaeSwap","text":"Category: Solution Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3300,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#regulated-and-permissioned-defi-with-sundae-and-kora-labs","title":"Regulated and Permissioned DeFi with Sundae and Kora Labs","text":"Category: Concept Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3100,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#sundae-labs-comprehensive-specification-development-for-gummiworm-protocol-on-cardano","title":"Sundae Labs Comprehensive Specification Development for Gummiworm Protocol on Cardano","text":"Category: Concept Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3100,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#nftcdnio-universal-nft-viewer-api-musicvideoweb3d","title":"[nftcdn.io] Universal NFT Viewer API (Music+Video+Web+3D+\u2026)","text":"Category: Product Applicant: Smaug (pool.pm) Requested funds: \u20b3299,999.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#nftcdnio-nsfw-nft-detection-for-marketplaces-wallets-explorers","title":"[nftcdn.io] NSFW NFT Detection for Marketplaces, Wallets & Explorers","text":"Category: Product Applicant: Smaug (pool.pm) Requested funds: \u20b3149,999.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#cardanoscan-analytics-charts","title":"Cardanoscan Analytics Charts","text":"Category: Product Applicant: Strica (cardanoscan.io) Requested funds: \u20b344,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#cardanoscan-api-javascript-sdk","title":"Cardanoscan API Javascript SDK","text":"Category: Developers Applicant: Strica (cardanoscan.io) Requested funds: \u20b364,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#integrate-keystone-hardware-wallet-into-typhon","title":"Integrate Keystone Hardware Wallet into Typhon","text":"Category: Product Applicant: Strica (cardanoscan.io) Requested funds: \u20b384,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#add-support-for-marlowe-on-cardanoscan","title":"Add support for Marlowe on Cardanoscan","text":"Category: Product Applicant: Strica (cardanoscan.io) Requested funds: \u20b3133,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#cardanoscan-data-info-bubbles","title":"Cardanoscan data info bubbles","text":"Category: Ecosystem Applicant: Strica (cardanoscan.io) Requested funds: \u20b335,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#create-a-cnt-marketplace-on-norwegian-block-exchange-nbxcom","title":"Create a CNT marketplace on Norwegian Block Exchange (NBX.COM)","text":"Category: Product Applicant: Eystein Hansen (nbx.com) Requested funds: \u20b3145,000.00 Links: Ideascale, Lidonation
"},{"location":"contributors/","title":"Contributors","text":"Everyone is welcome to contribute to the guide, as well as the repository. Below is just a thank you to people who have been contributing consistently:
Adam Chris Damjan Homer Markus OCG Ola Ahlman Pal Dorogi Papacarp PegasusPool Psychomb RdLrT RedOracle SmaugPool
To start contributing, simply hit the github repository and raise Issue/Pull Request
"},{"location":"grest-meets/","title":"GRest Meeting summaries","text":"Thank you all for joining and contributing to the project
Below you can find a short summary of every GRest meeting held, both for logging purposes and for those who were not able to attend.
"},{"location":"grest-meets/#participants","title":"Participants:","text":"Participant 16Sep2021 02Sep2021 26Aug2021 19Aug2021 12Aug2021 29Jul2021 22Jul2021 15Jul2021 09Jul2021 02Jul2021 25Jun2021 Damjan Homer Markus Ola RdLrT Red Papacarp Paddy GimbaLabs 16Sep2021 02Sep2021 26Aug2021 19Aug2021 12Aug2021 29Jul2021 22Jul2021 15Jul2021 09Jul2021 02Jul2021After the initial stand-up updates from participants, we went through the entire Trello board, updating/deleting existing tickets and creating some new ones.
25Jun2021"},{"location":"grest-meets/#scheduling-running-update-queries","title":"Scheduling running update queries","text":"Solution being tested:
Pool cache table:
we will run the full query on regular intervals, ready for review for first iteration, will see about delta post tx cache query
transaction history:
need to think about how to approach inputs/outputs in the cached table (1 row per transaction with json objects for inputs/outputs or multiple rows for tx hash)
address_txs:
this endpoint should bring back list of txs, and have provision to use after and before block hash - lightweight against public schema
pool cache table:
create a trigger every 2 minutes (or similar) to run stake_distribution query
docker:
EXPLAIN (ANALYZE, BUFFERS)Team
grest schema)Individual
84226d33eed66be8e61d50b7e1dacebdc095cee9 on release/10.1.x<query>.json and sql in <query>.sql), also remove get_ prefixnbthreads in config, tune maxconn, switch to http mode)Ola added automatic deployment of services to the scripts last week. We added new tasks on Trello ticket, including flags for multiple networks (guild, testnet, mainnet), haproxy service dynamically creating hosts and doc updates. Overall, the script works well with some manual interaction still required at the moment.
"},{"location":"grest-meets/#supported-networks","title":"Supported Networks","text":"Just for the record here, a 16GB (or even 8GB) instance is enough to support both testnet and guild networks.
"},{"location":"grest-meets/#db-sync-versioning","title":"db-sync versioning","text":"We agreed to use the release/10.1.x branch which is not yet released but built to include Alonzo migrations to avoid rework later. This version does require Alonzo config and hash to be in the node's config.json. This has to be done manually and the files are available here. Once fully released, all members should rebuild the released version to ensure each instance is running the same code.
For the DNS setup ticket, we started to think about the instance names for the 2 DNS instances (orange in the graph). Submissions for names will be made in the Telegram group, and will probably make a poll once we have the entries finalised.
"},{"location":"grest-meets/#monitoring-system","title":"Monitoring System","text":"Priyank started setting up the monitoring on his instance which can then easily be switched to a separate monitoring instance. We agreed to use Prometheus / Grafana combo for data source / visualisation. We'll probably need to create some custom archiving of data to keep it long term as Prometheus stores only the last 30 days of data.
"},{"location":"grest-meets/#next-meeting","title":"Next meeting","text":"We would like to make Friday @ 07:00 UTC the standard time and keep meetings at weekly frequency. A poll will still be created for next weeks, but if there are no objections / requests for switching the time around (which we have not had so far) we can go ahead with the making Friday the standard with polls no longer required and only reminders / Google invites sent every week.
"},{"location":"grest-meets/#deployment-scripts_1","title":"Deployment scripts","text":"During the last week, work has been done on deployment scripts for all services (db-sync, gRest and haproxy) -> this is now in testing with updated instructions on trello. Everybody can put their name down on the ticket to signify when the setup is complete and note down any comments for bugs/improvements. This is the main priority at the moment as it would allow us to start transferring our setups to mainnet.
"},{"location":"grest-meets/#switch-to-mainnet","title":"Switch to Mainnet","text":"Following on from that, we created a ticket for starting to set up mainnet instances -> we can use 32GB RAM to start and increase later. While making sure everything works against the guild network is priority, people are free to start on this as well as we anticipate we are almost ready for the switch.
"},{"location":"grest-meets/#supported-networks_1","title":"Supported Networks","text":"This brings me to another discussion point which is on which networks are to be supported. After some discussion, it was agreed to keep beefy servers for mainnet, and have small independent instances for testnet maintained by those interested, while guild instance is pretty lightweight and useful to keep.
"},{"location":"grest-meets/#monitoring-system_1","title":"Monitoring System","text":"The ticket for creating a centralised monitoring system was discussed and updated. I would say it would be good to have at least a basic version of the system in place around the time we switch to mainnet. The system could eventually serve for: - analysis of instance - performances and subsequent tuning - endpoints usage - anticipation of system requirements increases - etc.
I would say that this should be an important topic of the next meeting to come up with an approach on how we will structure this system so that we can start building it in time for mainnet switch.
"},{"location":"grest-meets/#handling-ssl","title":"Handling SSL","text":"Enabling SSL was agreed to not be required by each instance, but is optional and documentation should be created for how to automate the process of renewing SSL certificates for those wishing to add it to their instance. The end user facing endpoints \"Instance Checker\" will of course be SSL-enabled.
"},{"location":"grest-meets/#next-meeting_1","title":"Next meeting","text":"We somewhat agreed to another meeting next week again at the same time, but some participants aren't 100% for availability. Friday at 07:00 UTC might be a good standard time we hold on to, but I will make a poll like last time so that we can get more info before confirming the meeting.
"},{"location":"grest-meets/#meeting-structure","title":"Meeting Structure","text":"As this was the first meeting, at the start we discussed about the meeting structure. In general, we agreed to something like listed below, but this can definitely change in the future:
1) 2-liner (60s) round the table stand-ups by everyone to sync up on what they were doing / are planning to do / mention struggles etc. This itself often sparks discussions. 2) going through the Trello board tasks with the intention of discussing and possbily assigning them to individuals / smaller groups (maybe 1-2-3 people choose to work together on a single task)
"},{"location":"grest-meets/#stand-ups","title":"Stand-ups","text":"We then proceeded to give a status of where we are individually in terms of what's been done, a summary below:
prereqs.sh addendum can be done once artifacts are finalised (added a Trello ticket for tracking).All in all, I think we saw that there is need for these meetings as there are a lot of things to discuss and new ideas come up (like the monitoring system). We went for over an hour (~1h15min) and still didn't have enough time to go through the board, we basically only touched the DNS/haproxy part of the board. This tells me that we are in a stage where more frequent meetings are required, weekly instead of biweekly, as we are in the initial stage and it's important to build things right from the start rather than having to refactor later on. With that, the participants in general agreed to another meeting next week, but this will be confirmed in the TG chat and the times can be discussed then.
"},{"location":"sidebar/","title":"Tree","text":"Warning
Make sure you go through upgrade steps for your setup in a non-mainnet environment first!!
guild-deploy.sh (always double check syntax of the script with guild-deploy.sh -h). The scripts modified with user content (env, gLiveView.sh, topologyUpdater.sh, cnode.sh, etc) will be backed up before overwriting. The backed up files will be in the same folder as the original files, and will be named as ${filename}_bkp<timestamp>. More static files (genesis files or some of the scripts themselves) will not be backed up, as they're not expected to be modified.Remember
You are expected to provide appropriate environment-specific parameters (eg: custom top level folder [-p], alternate name for top level folder [-t], network flag [-n], etc) to the examples that pertain to your use case
Depending on node release, you may be able to simply perform an update-in-place of scripts and node, or for some cases, you may need to overwrite configs as well. Some Examples below:
mkdir \"$HOME/tmp\";cd \"$HOME/tmp\"\ncurl -sfS -o guild-deploy.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/guild-deploy.sh && chmod 700 guild-deploy.sh\n./guild-deploy.sh -s dl -b master -n mainnet -t cnode -p /opt/cardano\n\"${CNODE_HOME}\"/files folder if you'd like to compare/reuse previous version.mkdir \"$HOME/tmp\";cd \"$HOME/tmp\"\ncurl -sfS -o guild-deploy.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/guild-deploy.sh && chmod 700 guild-deploy.sh\n./guild-deploy.sh -s dlf -b master -n mainnet -t cnode -p /opt/cardano\nBeware
When upgrading node, depending on node versions (especially for major release) - you'd likely have to wait for node to revalidate/replay ledger. This can take a few hours. Please always plan ahead, do it first on a relay to ensure you've got \"${CNODE_HOME}/db\" folder ready to copy over (while source and target node have been shutdown) - prior to starting on upgrade on new machine. If mithril for target node version is ready, you can also use mithril-client to download snapshot instead of replaying, which may save you some time
\"${HOME}\"/.local/bin is part of your \\(PATH environment variable. If your shell does not auto-run bashrc, you may want to set it a call to \"\\)\"/.bashrc in your .profilesource \"${HOME}\"/.bashrc\necho \"${PATH}\"\nWe've found users often confuse between \\(PATH variable resolution between multiple shell sessions, systemd, etc. While if you only used this guide, the binaries should be in \"\\)/.local/bin\", you may have manually downloaded to another location before. To avoid this, you can edit the following files and uncomment and set the following variables to the appropriate paths as per your deployment (eg: CCLI=\"${HOME}\"/.local/bin/cardano-cli if following above):
The above should take care of tools and services. However, you might still have duplicate binaries in your $PATH (previous artifacts, re-build using old scripts, etc) - it is best that you remove any old binary files from alternate folders. You can do so by executing the below:
whereis bech32 cardano-address cardano-cli cardano-db-sync cardano-hw-cli cardano-node cardano-submit-api cncli ogmios\nFor some cases - you might have no values (eg: you may not use cardano-db-sync, cncli, ogmios and/or cardano-hw-cli. You need not take any actions for the binaries you do not use.
sudo systemctl status cnode. If the node shows as up, you can monitor logs via tail -100f \"${CNODE_HOME}/logs/node.json. If you're unable to start node using systemd itself, you can check the systemd output via sudo journalctl -xeu cnode.service and scroll back until you see a startup attempt with reason (typically this could be a couple of pages back). If nothing obvious (eg: showing files used as /files/config.json instead of $CNODE_HOME/files/config.json )comes up from there, you'd want to view node logs to see if there is something recorded in node.json instead.Let's say you accidentally did an update of the files that you didnt intend to and want to revert all your scripts to previous state. While you have the backups of the scripts created while doing in-place update, you can always make use of branch flag that most scripts that perform update-in-place provide (eg: for gLiveView.sh that you want to restore to 8.1.2 state), this would be -b node-8.1.2. The available tags that can be used can be visited here
Hope the guide above helps you with the migration, but again - we could've missed some edge cases. If so, please report via chat in Koios Discussions channel or open an issue on github. Please DO NOT make edits to the script content based on forum/alternate guide/channels, while done with best intentions - there have been solutions put online that modify files unnecessarily instead of correcting configs and disabling updates, such actions will only cause trouble for future updates.
"},{"location":"Appendix/RecoverByronWallet/","title":"Unofficial Instructions for recovering your Byron Era funds on the new Incentivized Shelley Testnet","text":""},{"location":"Appendix/RecoverByronWallet/#1-grab-and-install-haskell","title":"1. Grab and install Haskell","text":"curl -sSL https://get.haskellstack.org/ | sh\nnote: you must build from source as of today as there are changes that just got into master you need
git clone https://github.com/cardano-foundation/cardano-wallet.git\ncd cardano-wallet\nstack build --test --no-run-tests\nsudo apt install libssl-dev\nsudo apt-get install sqlite3 libsqlite3-dev \nsudo apt-get install libgmp3-dev \nsudo apt install libsystemd-dev\nget coffee... It takes awhile
"},{"location":"Appendix/RecoverByronWallet/#5-when-its-done-install-executables-to-your-path","title":"5. When its done, install executables to your path","text":"stack install\nGenerate your new mnemonics you will need below. Note that this generates 15 words as opposed to your byron era mnemnomics which were only 12 words.
cardano-wallet-jormungandr mnemonic generate\nyou can either open another terminal window or use screen or something. anyway, wherever you run this next command you won't be able to use anymore for a terminal until you stop the wallet
change --node-port 3001 to wherever you have your jormungandr rest interface running. for me it was 5001.. so
change --port 3002 to wherever you want to access the wallet interface at. If you have other things running avoid those ports. for most, 3002 should be free
just to future proof these instructions. genesis should be whatever genesis you are on.
cardano-wallet-jormungandr serve --node-port 3001 --port 3002 --genesis-block-hash e03547a7effaf05021b40dd762d5c4cf944b991144f1ad507ef792ae54603197\n--->in another window
replace foo, foo, foo with all your mnemnomics from the byron wallet you are restoring
Also, if you put your wallet on a different port than 3002, fix that too
curl -X POST -H \"Content-Type: application/json\" -d '{ \"name\": \"legacy_wallet\", \"mnemonic_sentence\": [\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\"], \"passphrase\": \"areallylongpassword\"}' http://localhost:3002/v2/byron-wallets\nRemember all those mnemnomics you made above.. put them here instead of all the foo's.
curl -X POST -H \"Content-Type: application/json\" -d '{ \"name\": \"pool_wallet\", \"mnemonic_sentence\": [\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\"], \"passphrase\": \"areallylongpasswordagain\"}' http://localhost:3002/v2/wallets\nNow you are ready to migrate your wallet. replace the <old wallet id> and <new wallet id> with the values you got above
curl -X POST -H \"Content-Type: application/json\" -d '{\"passphrase\": \"areallylongpassword\"}' http://localhost:3002/v2/byron-wallets/<old wallet id>/migrations/<new wallet id>\nFrom here we recommend you send them to a new address entirely owned and created by jcli or whatever method you have been using for the testnet process.
This technically may not be required. But a lot of us did it and we know it works for setting up pools and stuff.
send a small amount first just to make sure you are in control of the transaction and don't send your funds to la la land.
If you want to send to another address use the command below, but replace the address that you want to send it to, the amount, and your <new wallet id>
curl -X POST -H \"Content-Type: application/json\" -d '{\"payments\": [ { \"address\": \"<address to send to>\"\", \"amount\": { \"quantity\": 83333330000000, \"unit\": \"lovelace\" } } ], \"passphrase\": \"areallylongpasswordagain\"}' http://localhost:3002/v2/wallets/<new wallet id>/transactions\nEnsure the Pre-Requisites are in place before you proceed.
This is an easy-to-use script to automate setting up of monitoring tools. Tasks automates the following tasks: - Installs Prometheus, Node Exporter and Grafana Servers for your respective Linux architecture. - Configure Prometheus to connect to cardano node and node exporter jobs. - Provisions the installed prometheus server to be automatically available as data source in Grafana. - Provisions two of the common grafana dashboards used to monitor cardano-node by SkyLight and IOHK to be readily consumed from Grafana. - Deploy prometheus,node_exporter and grafana-server as systemd service on Linux. - Start and enable those services.
Note that securing prometheus/grafana servers via TLS encryption and other security best practices are out of scope for this document, and its mainly aimed to help you get started with monitoring without much fuss.
!> Ensure that you've opened the firewall port for grafana server (default used in this script is 5000)
"},{"location":"Appendix/monitoring/#download-setup_monsh","title":"Download setup_mon.sh","text":"If you have run guild-deploy.sh, you can skip this step. To download monitoring script, you can execute the commands below:
cd $CNODE_HOME/scripts\nwget https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/setup_mon.sh\nchmod 750 setup_mon.sh\nThe default selection may not always be usable for everyone. You can customise further environment variable settings by opening in editor (eg: vi setup_mon.sh ), and updating variables below to your liking:
#!/usr/bin/env bash\n# shellcheck disable=SC2209,SC2164\n\n######################################################################\n#### Environment Variables\n######################################################################\nCNODE_IP=127.0.0.1\nCNODE_PORT=12798\nGRAFANA_HOST=0.0.0.0\nGRAFANA_PORT=5000\nPROJ_PATH=/opt/cardano/monitoring\nPROM_HOST=127.0.0.1\nPROM_PORT=9090\nNEXP_PORT=$(( PROM_PORT + 1 ))\n````\n\n#### Set up Monitoring\n\nExecute setup_mon.sh with full path to destination folder you want to setup monitoring in. If you're following guild folder structure, you do not need to specify `-d`. Read the usage comments below before you run the actual script.\n\nNote that to deploy services as systemd, the script expect sudo access is available to the user running the script.\n\n``` bash\ncd $CNODE_HOME/scripts\n# To check Usage parameters:\n# ./setup_mon.sh -h\n#Usage: setup_mon.sh [-d directory] [-h hostname] [-p port]\n#Setup monitoring using Prometheus and Grafana for Cardano Node\n#-d directory Directory where you'd like to deploy the packages for prometheus , node exporter and grafana\n#-i IP/hostname IPv4 address or a FQDN/DNS name where your cardano-node (relay) is running (check for hasPrometheus in config.json; eg: 127.0.0.1 if same machine as cardano-node)\n#-p port Port at which your cardano-node is exporting stats (check for hasPrometheus in config.json; eg: 12798)\n./setup_mon.sh\n# \n# Downloading prometheus v2.18.1...\n# Downloading grafana v7.0.0...\n# Downloading exporter v0.18.1...\n# Downloading grafana dashboard(s)...\n# - SKYLight Monitoring Dashboard\n# - IOHK Monitoring Dashboard\n# \n# NOTE: Could not create directory as rdlrt, attempting sudo ..\n# NOTE: No worries, sudo worked !! Moving on ..\n# Configuring components\n# Registering Prometheus as datasource in Grafana..\n# Creating service files as root..\n# \n# =====================================================\n# Installation is completed\n# =====================================================\n# \n# - Prometheus (default): http://127.0.0.1:9090/metrics\n# Node metrics: http://127.0.0.1:12798\n# Node exp metrics: http://127.0.0.1:9091\n# - Grafana (default): http://0.0.0.0:5000\n# \n# \n# You need to do the following to configure grafana:\n# 0. The services should already be started, verify if you can login to grafana, and prometheus. If using 127.0.0.1 as IP, you can check via curl\n# 1. Login to grafana as admin/admin (http://0.0.0.0:5000)\n# 2. Add \"prometheus\" (all lowercase) datasource (http://127.0.0.1:9090)\n# 3. Create a new dashboard by importing dashboards (left plus sign).\n# - Sometimes, the individual panel's \"prometheus\" datasource needs to be refreshed.\n# \n# Enjoy...\n# \n# Cleaning up...\nYou should now be able to Login to grafana dashboard, using the public IP of your server, at port 5000. The initial credentials to login would be admin/admin, and you will be asked to update your password upon first login. Once logged on, you should be able to go to Manage > Dashboards and select the dashboard you'd like to view. Note that if you've just started the server, you might see graphs as empty, as initial interval for dashboards is 12 hours. You can change it to 5 minutes by looking at top right section of the page.
Thanks to Pal Dorogi for the original setup instructions used for modifying.
"},{"location":"Appendix/postgres/","title":"Sample Postgres Setup","text":"These deployment instructions used for reference while building cardano-db-sync tool, with the scope being ease of set up, and some tuning baselines for those who are new to Postgres DB. It is recommended to customise these as per your needs for Production builds.
Important
You'd find it pretty useful to set up ZFS on your system prior to setting up Postgres, to help with your IOPs throughput requirements. You can find sample install instructions here. You can set up your entire root mount to be on ZFS, or you can opt to mount a file as ZFS on \"${CNODE_HOME}\"
"},{"location":"Appendix/postgres/#install-postgresql-server","title":"Install PostgreSQL Server","text":"Execute commands below to set up Postgres Server
# Determine OS platform\nOS_ID=$( (grep -i ^ID_LIKE= /etc/os-release || grep -i ^ID= /etc/os-release) | cut -d= -f 2)\nDISTRO=$(grep -i ^NAME= /etc/os-release | cut -d= -f 2)\n\nif [ -z \"${OS_ID##*debian*}\" ]; then\n#Debian/Ubuntu\nwget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -\n RELEASE=$(lsb_release -cs)\necho \"deb [arch=amd64] http://apt.postgresql.org/pub/repos/apt/ ${RELEASE}\"-pgdg main | sudo tee /etc/apt/sources.list.d/pgdg.list\n sudo apt-get update\n sudo apt-get -y install postgresql-17 postgresql-server-dev-17 postgresql-contrib libghc-hdbc-postgresql-dev\n sudo systemctl enable postgresql\nelse\necho \"We have no automated procedures for this ${DISTRO} system\"\nfi\nBefore you start populating your DB instance using dbsync data, now might be a good time to put some thought on to baseline configuration of your postgres instance by editing /etc/postgresql/17/main/postgresql.conf. Typically, you might find a lot of common standard practices parameters available in tuning guides. For our consideration, it would be nice to start with some baselines - for which we will use inputs from example here, which would need to be customised further to your environment and resources.
In a typical Koios [gRest] setup, we use below for minimum viable specs (i.e. 64GB RAM, > 8 CPUs, >16K IOPs for ioping -q -S512M -L -c 10 -s8k . output when postgres data directory is on ZFS configured with max arc of 4GB), we find the below configuration to be the best common setup:
In addition to above, due to the nature of usage by dbsync (synching from node and restart traversing back to last saved ledger-state snapshot), we leverage data retention on blockchain - as we're not affected by loss of volatile information upon a restart of instance. Thus, we can relax some of the data retention and protection against corruption related settings, as those are IOPs/CPU Load Average impacts that the instance does not need to spend. We'd recommend setting 3 of those below in your /etc/postgresql/17/main/postgresql.conf:
Once your changes are done, ensure to restart postgres service using sudo systemctl restart postgresql.
Login to Postgres instance as superuser:
echo $(whoami)\n# <user>\nsudo su postgres\npsql\nNote the returned as the output of echo $(whoami) command. Replace all instance of in the documentation below. Execute the below in psql prompt. Replace and PasswordYouWant with your OS user (output of echo $(whoami) command executed above) and a password you'd like to authenticate to Postgres with:
CREATE ROLE <user> SUPERUSER LOGIN;\n\\q\nexit at shell to return to your user from postgres"},{"location":"Appendix/postgres/#verify-login-to-postgres-instance","title":"Verify Login to postgres instance","text":"export PGPASSFILE=$CNODE_HOME/priv/.pgpass\necho \"/var/run/postgresql:5432:cexplorer:*:*\" > $PGPASSFILE\nchmod 0600 $PGPASSFILE\npsql postgres\n# psql (17.0)\n# Type \"help\" for help.\n# \n# postgres=#\nImportant
An average pool operator may not require cardano-db-sync at all. Please verify if it is required for your use as mentioned here.
PGPASSFILE environment variable is set as per the instructions in the sample guide, for db-sync to be able to connect.$CNODE_HOME/files/db-sync-config.json to toggle ledger to disable (read dbsync documentation for it's effects).Execute the below to clone the cardano-db-sync repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/intersectmbo/cardano-db-sync\ncd cardano-db-sync\nYou can use the instructions below to build the latest release of cardano-db-sync.
git fetch --tags --all\ngit pull\n# Include the cardano-crypto-praos and libsodium components for db-sync\n# On CentOS 7 (GCC 4.8.5) we should also do\n# echo -e \"package cryptonite\\n flags: -use_target_attributes\" >> cabal.project.local\n# Replace tag against checkout if you do not want to build the latest released version\ngit checkout $(curl -sLf https://api.github.com/repos/intersectmbo/cardano-db-sync/releases/latest | jq -r .tag_name)\n# Use `-l` argument if you'd like to use system libsodium instead of IOG fork of libsodium while compiling\n$CNODE_HOME/scripts/cabal-build-all.sh\ncardano-db-sync binary into ~/.local/bin folder."},{"location":"Build/dbsync/#prepare-db-for-sync","title":"Prepare DB for sync","text":"Now that binaries are available, let's create our database (when going through breaking changes, you may need to use --recreatedb instead of --createdb used for the first time. Again, we expect that PGPASSFILE environment variable is already set (refer to the top of this guide for sample instructions):
cd ~/git/cardano-db-sync\n# scripts/postgresql-setup.sh --dropdb #if exists already, will fail if it doesnt - thats OK\nscripts/postgresql-setup.sh --createdb\n# Password:\n# Password:\n# All good!\nVerify you can see \"All good!\" as above!
"},{"location":"Build/dbsync/#create-symlink-to-schema-folder","title":"Create Symlink to schema folder","text":"DBSync instance requires the schema files from the git repository to be present and available to the dbsync instance. You can either clone the ~/git/cardano-db-sync/schema folder OR create a symlink to the folder and make it available to the startup command we will be using. We will use the latter in sample below:
ln -s ~/git/cardano-db-sync/schema $CNODE_HOME/guild-db/schema\nIf you're running a mainnet/preview/preprod instance of dbsync, you might want to consider use of dbsync snapshots as documented here. The snapshot files from IO for their default configs as of recent epoch are available via links in release notes. Note that the snapshots should only be used pertaining to their specific configs, if using configs from Koios - you'd want to look at snapshots here instead.
At high-level, this would involve steps as below (read and update paths as per your environment):
# Replace the actual link below with the latest one from release notes\ncurl -fL 'https://share.koios.rest/api/public/dl/xFdZDfM4/dbsync/mainnet-dbsyncsnap-latest.tgz' -o /tmp/dbsyncsnap.tgz\nrm -rf ${CNODE_HOME}/guild-db/ledger-state ; mkdir -p ${CNODE_HOME}/guild-db/ledger-state\ncd -; cd ~/git/cardano-db-sync\nscripts/postgresql-setup.sh --restore-snapshot /tmp/dbsyncsnap.tgz ${CNODE_HOME}/guild-db/ledger-state\n# The restore may take a while, please be patient and do not interrupt the restore process. Once restore is successful, you may delete the downloaded snapshot as below:\n# rm -f /tmp/dbsyncsnap.tgz\nIn order to verify that you can run dbsync, before making a start - you'd want to ensure that you can run it interactively once. To do so, try the commands below:
cd $CNODE_HOME/scripts\nexport PGPASSFILE=$CNODE_HOME/priv/.pgpass\n./dbsync.sh\nYou can monitor logs if needed via parallel session using tail -10f $CNODE_HOME/logs/dbsync.json. If there are no error, you would want to press Ctrl-C to stop the dbsync.sh execution and deploy it as a systemd service. To do so, use the commands below (the creation of file is done using sudo permissions, but you can always deploy it manually):
cd $CNODE_HOME/scripts\n./dbsync.sh -d\n# Deploying cnode-dbsync.service as systemd service..\n# cnode-dbsync.service deployed successfully!!\nNow to start dbsync instance, you can run sudo systemctl start cnode-dbsync
Note
Note that dbsync while syncs, it might defer creation of indexes/constraints to speed up initial catch up. Once relatively closer to tip, this will initiate creation of indexes - which can take a while in background. Thus, you might notice the query timings right after reaching to tip might not be as good.
"},{"location":"Build/dbsync/#update-dbsync","title":"Update DBSync","text":"Updating dbsync can have different tasks depending on the versions involved. We attempt to briefly explain the tasks involved:
sudo systemctl stop cnode-dbsync)Go to your git folder, pull and checkout to latest version as in example below (if you were to switch to 13.6.0.2):
cd ~/git/cardano-db-sync\ngit pull\ngit checkout 13.6.0.2\nIf going through major version update (eg: 13.x.x.x to 14.x.x.x), you might need to rebuild and resync db from scratch, you may still follow the section to restore using snapshot to save some time (as long as you use a compatible snapshot).
cardano-node version has changed (specifically if it's ledger-state schema is different), you'd also need to clear the ledger-state directory (eg: rm -rf $CNODE_HOME/guild-db/ledger-state)dbsync.sh starts up fine manually as described above. If it does, stop it and go ahead with startup of systemd service (i.e. sudo systemctl start cnode-dbsync)To validate, connect to your postgres instance and execute commands as per below:
export PGPASSFILE=$CNODE_HOME/priv/.pgpass\npsql cexplorer\nYou should be at the psql prompt, you can check the tables and verify they're populated:
\\dt\nselect * from meta;\nA sample output of the above two commands may look like below (the number of tables and names may vary between versions):
cexplorer=# \\dt\nList of relations\n Schema | Name | Type | Owner\n--------+---------------------------+-------+-------\n public | ada_pots | table | centos\n public | admin_user | table | centos\n public | block | table | centos\n public | delegation | table | centos\n public | delisted_pool | table | centos\n public | epoch | table | centos\n public | epoch_param | table | centos\n public | epoch_stake | table | centos\n public | ma_tx_mint | table | centos\n public | ma_tx_out | table | centos\n public | meta | table | centos\n public | orphaned_reward | table | centos\n public | param_proposal | table | centos\n public | pool_hash | table | centos\n public | pool_meta_data | table | centos\n public | pool_metadata | table | centos\n public | pool_metadata_fetch_error | table | centos\n public | pool_metadata_ref | table | centos\n public | pool_owner | table | centos\n public | pool_relay | table | centos\n public | pool_retire | table | centos\n public | pool_update | table | centos\n public | pot_transfer | table | centos\n public | reserve | table | centos\n public | reserved_ticker | table | centos\n public | reward | table | centos\n public | schema_version | table | centos\n public | slot_leader | table | centos\n public | stake_address | table | centos\n public | stake_deregistration | table | centos\n public | stake_registration | table | centos\n public | treasury | table | centos\n public | tx | table | centos\n public | tx_in | table | centos\n public | tx_metadata | table | centos\n public | tx_out | table | centos\n public | withdrawal | table | centos\n(37 rows)\n\n\n\nselect * from meta;\n id | start_time | network_name\n----+---------------------+--------------\n 1 | 2017-09-23 21:44:51 | mainnet\n(1 row)\nThis release adds support for cardano db sync 13.6.0.2, alongwith underlying components supporting Conway HF. The major chunk of work for this release is behind the scenes, with minor value additions to input/output schema.
"},{"location":"Build/grest-changelog/#new-endpoints-added","title":"New endpoints added:","text":"/block_tx_info and /tx_info - Fix for _bytecode flag set to false not setting all byte fields to null #306/block_tx_info and /tx_info - _scripts flag set to false will no longer suppress reference_inputs, collateral_inputs or collateral_outputs #306/proposal_voting_summary - Add new fields drep_abstain_votes_cast, pool_abstain_votes_cast and committee_abstain_votes_cast #303/drep_info, /drep_metadata - Rename url to meta_url and hash to meta_hash , keeping it consistent with other endpoints #306/tx_cbor - Add new fields block_hash, block_height, epoch_no, absolute_slot, tx_timestamp #306/pool_info - Add new fields reward_addr_delegated_drep and voting_power #306view column with own bech32 utility functions #302This is a finalised release that builds on 1.2.0a to provide support for CIP-129 and add a summary of votes for given proposal. The changes accordingly are primarily only targetting Governance endpoints. This will be the version used for mainnet upgrade as well. Please go through the changelogs below
/proposal_voting_summary - Get a summary of votes cast on specified governance action #300/commitee_votes - Will require _cc_hot_id which will accept committee member hot key formatted in bech32 as per CIP-0005/129 #300/voter_proposal_list - Will require _voter_id which will accept DRep/SPO/Committee member formatted in bech32 as per CIP-0005/129 #300/proposal_votes - Will require _proposal_id which will accept government proposal ID formatted in bech32 as per CIP-129 #300/drep_metadata , /drep_updates, - added column has_script which shows if given credential is a script hash #300/drep_votes , /proposal_list , /committee_info - added column proposal_id to show proposal action ID in accordance with CIP-129 #300/proposal_votes , - voter is renamed to voter_id and shows DRep/Pool/Committee member formatted in bech32 as per CIP-129 #300This release starts providing Conway support providing 14 new endpoints - primarily focusing on new governance data. Also, based on community requests/feedbacks - it introduces a few breaking changes for tx_info and block_tx_info endpoints. Please go through the changelogs below
/tx_cbor - Raw transaction CBOR against a transaction #298/drep_epoch_summary - Summary of voting power and DRep count for each epoch #298/drep_list - List of all active delegated representatives (DReps) #298/drep_info - Get detailed information about requested delegated representatives (DReps) #298/drep_metadata - List metadata for requested delegated representatives (DReps) #298/drep_updates - List of updates for requested (or all) delegated representatives (DReps) #298/drep_votes - List of all votes casted by requested delegated representative (DRep) #298/drep_delegators - List of all delegators to requested delegated representative (DRep) #298/committee_info - Information about active committee and its members #298/committee_votes - List of all votes casted by given committee member or collective #298/proposal_list - List of all governance proposals #298/voter_proposal_list - List of all governance proposals for specified DRep, SPO or Committee credential #298/proposal_votes - List of all votes cast on specified governance action #298/pool_votes - List of all votes casted by a pool #298/block_tx_info, /tx_info - collateral_tx_out -> asset_list - Outputs for collateral tx out are never created on-chain and thus, cannot be queried from ma_tx_out. Instead a rough description of assets involved are saved , which is now returned as info from ledger. This is returned as-is from dbsync and does not adhere to asset_list schema we use in other endpoints. #298/tx_info , /block_tx_info - These endpoints now require you to specify what all information you'd like to retrieve from a transaction, providing flags _inputs , _metadata, _assets , _withdrawals, _certs, _scripts, _bytecode, _governance #298/policy_asset_mints , /policy_asset_info, /asset_info - Will return latest mint transaction that has metadata (instead of latest mint transaction) details (excluding burn transactions) #298/account_info , /pool_info , /pool_list - Add deposit field to output for deposit associated with registration #298/account_info - Add delegated_drep field to the output #298/block_tx_info , /tx_info - Add treasury_deposit, voting_procedures and proposal_procedures to the output #298/epoch_params - Add various fields to epoch_params as per Conway protocol parameters #298/pool_metadata, /pool_relays - Remove pool_status field from output (it's already listed in pool_info and list) #298/pool_updates - owners is now a JSONB field instead of JSONB array #298asset_info_cache - first_mint_tx_id , first_mint_keys , last_mint_keys are not used/required #286last_mint_meta_tx_id field to asset_info_cache - to return latest asset that does have metadata #286pool_history_cache #289pool_stat instead of epoch_stake for pool_history_cache #294instant_reward table in dbsync moved to reward_restada_pots : deposit now split into three different types of depositsThis release is minor bugfix for data consistency changes behind the scenes. It has no impact to any of the API endpoints.
"},{"location":"Build/grest-changelog/#chores_3","title":"Chores:","text":"pool_info : Add a pool_delegators_list RPC that brings across only minimal information required by pool_info, thereby improving it's performance #281pool_history stats for latest epochs by restricting cache to current - 3 epoch while calculating the subsequent information using live status from the database #282account_info , account_info_cached and stake_distribution_cache.This release primarily focuses on backend performance fixes and work with dbsync 13.2.0.2 - while also, we have started preparing compatibility with upcoming koios lite release, to make it a seamless swap for specific endpoints without any impact to consumers. There are no breaking (impact to existing columns or inputs) changes with this release, but we have retired 2 deprecated endpoints that were almost unused on mainnet. Due to the amount of backend changes in queries, there is a chance that we may have missed some data accuracy checks, and - hence - would like to test in non-mainnet networks first before marking final release. Accordingly, any testing/reports of data inconsistency would be welcome.
"},{"location":"Build/grest-changelog/#new-endpoints-added_3","title":"New endpoints added:","text":"/asset_policy_mints - List of mint/burn count for all assets minted under a policy #269/block_tx_info - Equivalent of tx_info but uses blocks as inputs to fetch tx_info against all tx in the block[s] requested, also contains additional flags to control performance and output #255/cli_protocol_params - Return protocl-parameters as returned by cardano-cli from Koios servers #269/reserve_withdrawals , /treasury_withdrawals - Add earned_epoch and spendable_epoch fields #269/block - Add parent_hash field #263/account_list - Add stake_address_hex and script_hash fields #263/asset_list - Add script_hash field #263/asset_summary - Add addresses field #263/asset_addresses , /asset_nft_address and /policy_asset_addresses - Add stake_address field #262/script_utxos as it was incorrectly returning object instead of array for asset_list #269/tx_info - Add plutus_contract -> spends_input to plutus_contracts to point the input transaction being consumed by the script #269asset_address_list and asset_policy_info endpoints are now retired, as they were marked as deprecated in Koios 1.0.10 , and we have seen it's usage to be negligible (only a single hit in 48 hours on mainnet while marking this release). #269stake_distribution_new_accounts and stake_snapshot_cache cache, as we directly perform lookup on live tables for newly registered accounts #269epoch_sync_progress table #269consumed_by_tx_in_id references in SQL by consumed_by_tx_id #269pool_history_cache now breaks into populating 500 epochs at a time (on guildnet, this query used to run for hours against ~20K epochs) #269reward table into instant_reward #269stake_distribution_cache to ensure that epoch info cache was run for current - 1 epoch. #269grest.cip67_strip_label from text to bytea #269tx_in as it is no longer required #269pool_offline_data with off_chain_pool_data #269asset-txo-cache-update as the endpoints leveraging asset-txo will be moved to koios-liteblock, account_list, asset_list asset_token_registry from view to function #263asset_info_cache - ensure mint tx is only against a positive mint quantity #262tx_info - Fix spend_redeemers CTE Join condition #269This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (/api/v1). The major work with this release was to start making use of newer flags in dbsync which help performance of queries under new endpoints. Please ensure to check out the release notes for 1.1.0rc below. The list for this section is only a small addendum to 1.1.0rc:
asset_addresses and policy_asset_addresses #250pool_registrations and pool_retirements) #249This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (/api/v1). The major work with this release was to start making use of newer flags in dbsync which help performance of queries under new endpoints. Also, you'd see quite a few new endpoint additions below, that'd be helping out with slightly lighter version of queries. To keep migration paths easier, we will ensure both v0 and v1 versions of the release is up for a month post release, before retiring v0.
/pool_registrations - List of all pool registrations initiated in the requested epoch #239/pool_retirements - List of all pool retirements initiated in the requested epoch #239/treasury_withdrawals - List of withdrawals made from treasury #239/reserve_withdrawals - List of withdrawals made from reserves (MIRs) #239/account_txs - Transactions associated with a given stake address #239/address_utxos - Get UTxO details for requested addresses #239/asset_utxos - Get UTxO details for requested assets #239/script_utxos - Get UTxO details for requested script hashes #239/utxo_info - Details for requested UTxO arrays #239/script_info - Information about a given script FROM script hashes #239/ogmios/ - Expose stateless ogmios endpoints #1690/account_utxos , /credential_utxos - Accept extended as an additional flag - which enables asset_list, reference_script and inline_datum to the output #239/block_txs - Flatten output with transaction details (tx_hash, epoch_no, block_height, block_time) instead of tx_hashes array #239/epoch_params - Update cost_models to JSON (upstream change in node) #239/account_assets , /address_assets - Flatten the output result (instead of asset_list array) making it easier to apply horizontal filtering based on any of the fields/account_utxos , /address_utxos, /asset_utxos , /script_utxos and /utxo_info to return same schema giving complete details about UTxOs involved, with few fields toggled based on extended input flag #239/pool_list - Add various details to the endpoint for each pool (pool_id_hex,active_epoch_no,margin,fixed_cost,pledge,reward_addr,owners,relays,ticker,meta_url,meta_hash,pool_status,retiring_epoch) - this should mean some of the requests to pool_info should no longer be required #239/pool_updates - In v0, pool_updates only provided pool registration updates, while pool_status corresponded to current status of pool. With v1, we will have registration as well as deregistration transactions, and each transaction will have update_type (enum of either registration or deregistration) instead of pool_status. As a side-effect, since a registration transaction only has retiring_epoch as metadata, all the other fields will show up as null for such a transaction #239/pool_metadata , /pool_relays - Add pool_status field to denote whether pool is retired #239/datum_info - Rename hash to datum_hash and add creation_tx_hash #239/native_script_list - Remove script column (as it has pretty large output better queried against script_info), add size and change type to text #239/plutus_script_list - Add type and size to output #239/asset_info - Add cip68_metadata JSONB field #239/pool_history - Add member_rewards #225/tx_utxos - No longer required as replaced by /utxo_info #239v1 from v0 #1690epoch_info_cache Remove protocol parameters, as they can be queried from live table. Accordingly update dependent queries #239consumed_by_tx_in_id column in tx_out from dbsync 13.1.1.3 across endpoints #239_last_active_stake_validated_epoch in active_stake_cache #222The release is effectively same as 1.0.10rc except with one minor modification below.
cs.[{\"key\":\"value\"}] in PostgREST #172This release primarily focuses on ability to support better DeFi projects alongwith some value addition for existing clients by bringing in 10 new endpoints (paired with 2 deprecations), and few additional optional input parameters , and some additional output columns to existing endpoints. The only breaking change/fix is for output returned for tx_info.
Also, dbsync 13.1.x.x has been released and is recommended to be used for this release
"},{"location":"Build/grest-changelog/#new-endpoints-added_5","title":"New endpoints added","text":"/asset_addresses - Equivalent of deprecated /asset_address_list #149/asset_nft_address - Returns address where the specified NFT sits on #149/account_utxos - Returns brief details on non-empty UTxOs associated with a given stake address #149/asset_info_bulk - Bulk version of /asset_info #142/asset_token_registry - Returns assets registered via token registry on github #145/credential_utxos - Returns UTxOs associated with a payment credential #149/param_updates - Returns list of parameter update proposals applied to the network #149/policy_asset_addresses - Returns addresses with quantity for each asset on a given policy #149/policy_asset_info - Equivalent of deprecated /asset_policy_info but with more details in output #149/policy_asset_list - Returns list of asset under the given policy (including supply) #142, #149/account_addresses - Add optional _first_only and _empty flags to show only first address with tx or to include empty addresses to output #149/epoch_info - Add optional _include_next_epoch field to show next epoch stats if available (eg: nonce, active stake) #143/account_assets , /address_assets , /address_info, /tx_info, /tx_utxos - Add decimals to output #142/policy_asset_info - Add minting_tx_hash, total_supply, mint_cnt, burn_cnt and creation_time fields to the output #149/tx_info - Change _invalid_before and _invalid_after to text field #141tx_info - Remove the field plutus_contracts > [array] > outputs as there is no logic to connect it to inputs spending #163/asset_address_list - Renamed to asset_addresses keeping naming line with other endpoints (old one still present, but will be deprecated in future release) #149/asset_policy_info - Renamed to policy_asset_info keeping naming line with other endpoints (old one still present, but will be deprecated in future release) #149/epoch_info, /epoch_params - Restrict output to current epoch #149/block_info - Use /previous_id field to show previous/next blocks (previously was using block_id/height) #145/asset_info/asset_policy_info - Fix mint tx data to be latest #141grest.asset_info_cache to hold mint/burn counts alongwith first/last mint tx/keys #142/pool_delegators output column latest_delegation_tx_hash #149authenticator user, whose default statement_timeout is set to 65s and update configs accordingly #1606This release is effectively same as 1.0.9rc below (please check out the notes accordingly), just with minor bug fix on setup-grest.sh itself.
This release candidate is non-breaking for existing methods and inputs, but breaking for output objects for endpoints. The aim with release candidate version is to allow folks couple of weeks to test, adapt their libraries before applying to mainnet.
"},{"location":"Build/grest-changelog/#new-endpoints-added_6","title":"New endpoints added","text":"datum_info - List of datum information for given datum hashesaccount_info_cached - Same as account_info, but serves cached information instead of live oneaddress_info, address_assets, account_assets, tx_info, asset_list asset_summary to align output asset_list object to return array of policy_id, asset_name, fingerprint (and quantity, minting_txs where applicable) #120asset_history - Fix metadata to wrap in array to refer to right object #122asset_txs - Add optional boolean parameter _history (default: false) to toggle between querying current UTxO set vs entire history for asset #122pool_history - fixed_cost, pool_fees, deleg_rewards, epoch_ros will be returned as 0 when null #122tx_info - plutus_contracts->outputs can be null #122guild-operators repository to koios-artifacts repository. This is to ensure that the updates made to scripts and other tooling do not have a dependency on Koios query versioning #122block_info - Use block_no instead of id to check for previous/next block hash #122This release is contains minor bug-fixes that were discovered in koios-1.0.7. No major changes to output for this one.
"},{"location":"Build/grest-changelog/#changes-for-api","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#new-endpoints-added_7","title":"New endpoints added","text":"tx_info and tx_metadata - Align metadata for JSON output format #1542blocks - Query Output aligned to specs (epoch => epoch_no)epoch_block_protocols - [ ** Specs only ** ] Fix Documentation schema , which was accidentally showing wrong outputpool_delegators_history - List all epochs instead of current, if no _epoch_no is specified #1545asset_info - Fix metadata aggregaton for minting transactions with multiple metadata keys #1543stake_distribution_new_accounts - Leftover reference for account_info which now accepts array, resulted in error to populate stake distribution cache for new accounts #1541grest-poll.sh - Remove query view section from polling script, and remove grestrpcs re-creation per hour (it's already updated when setup-grest.sh is run) , in preparation for #1545This release continues updates from koios-1.0.6 to further utilise stake-snapshot cache tables which would be useful for SPOs as well as reduce downtime post epoch transition. One largely requested feature to accept bulk inputs for many block/address/account endpoints is now complete. Additionally, koios instance providers are now recommended to use cardano-node 1.35.3 with dbsync 13.0.5.
"},{"location":"Build/grest-changelog/#changes-for-api_1","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#new-endpoints-added_8","title":"New endpoints added","text":"pool_delegators_history - Provides historical record for pool's delegators #1486pool_stake_snapshot - Provides mark, set and go snapshot values for pool being queried. #1489pool_delegators - No longer accepts _epoch_no as parameter, as it only returns live delegators. Additionally provides latest_delegation_hash in output. #1486tx_info - epoch => epoch_no #1494tx_info - Change collateral_outputs (array) to collateral_output (object) as collateral output is only singular in current implementation #1496address_info - Add inline_datum and reference_script to output #1500pool_info - Add sigma field to output #1511pool_updates - Add historical metadata information to output #1503_stake_address text becomes _stake_addresses text[]). The additional changes in output as below:block_txs - Now returns block_hash and array of tx_hashesaddress_info - Additional field address returned in outputaddress_assets - Now returns address and an array of assets JSONaccount_addresses - Accepts stake_addresses array and outputs stake_address and array of addressesaccount_assets - Accepts stake_addresses array and outputs stake_address and array of assets JSONaccount_history - Accepts stake_addresses array alongwith epoch_no integer and outputs stake_address and array of history JSONaccount_info - Accepts stake_addresses array and returns additional field stake_address to outputaccount_rewards - Now returns stake_address and an array of rewards JSONaccount_updates - Now returns stake_address and an array of updates JSONasset_info - Change minting_tx_metadata from array to object #1533account_addresses - Sort results by oldest address first #1538epoch_info_cache - Only update last_tx_id of previous epoch on epoch transition #1490 and #1502epoch_info_cache / stake_snapshot_cache - Store total snapshot stake to epoch stake cache, and active pool stake to stake snapshot cache #1485The backlog of items not being added to mainnet has been increasing due to delays with Vasil HFC event to Mainnet. As such we had to come up with a split update approach. The mainnet nodes are still not qualified to be Vasil-ready (in our opinion) for 1.35.x , but dbsync 13 can be used against node 1.34.1 fine. In order to cater for this split, we have added an intermediate koios-1.0.6m tag that brings dbsync updates while maintaining node-1.34.1.
"},{"location":"Build/grest-changelog/#changes-for-api_2","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#data-output-changes","title":"Data Output Changes","text":"pool_delegators - epoch_no => active_epoch_no #1454asset_history - Add block_time and metadata fields for all previous mint transactions #1468asset_info - Retain latest mint transaction instead of first (in line with most CIPs as well as pool metadata - latest valid meta being live) #1468/tip , /blocks, /block_info => block_time/genesis => systemStart/epoch_info => start_time, first_block_time, last_block_time, end_time/tx_info => tx_timestamp/asset_info => creation_timetx_info - Add Vasil data #1464collaterals => collateral_inputscollateral_outputs, reference_inputs to tx_infodatum_hash, inline_datum, reference_script to collateral input/outputs, reference inputs & inputs/outputs JSON.cost_model instead of cost_model_id referenceepoch_params - Update leftover lovelace references to text for consistency: #1484key_depositpool_depositmin_utxo_valuemin_pool_costcoins_per_utxo_sizeget-metrics.sh - Add active/idle connections to database #1459grest-poll.sh: Bump haproxy to 2.6.1 and set default value of API_STRUCT_DEFINITION to be dependent on network used. #1450grest.account_active_stake_cache - optimise code and delete historical view (beyond 4 epochs). [#1451(https://github.com/cardano-community/guild-operators/pull/1451)tx_metalabels - Move metalabels from view to RPC using lose indexscan (much better performance) #1474grest.stake_snapshot_cache - Fix rewards for new accounts #1476Since there have been a few deviations wrt Vasil for testnet and mainnet, this version only targets networks except Mainnet!
"},{"location":"Build/grest-changelog/#changes-for-api_3","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#data-output-changes_1","title":"Data Output Changes","text":"/epoch_info - Add total_rewards and avg_block_reward for a given epoch #43/tip , /blocks, /block_info => block_time/genesis => systemStart/epoch_info => start_time, first_block_time, last_block_time, end_time/tx_info => tx_timestamp/asset_info => creation_time/blocks, /block_info => Add proto_major and proto_minor for a given block to output #55asset_registry_update.sh script to rely on commit hash instead of POSIX timestamps, and performance bump. #1428epoch_no, block_no to /address_txs, /credential_txs and /asset_txs endpoints. #1409/asset_txs, returning transactions as an array - allows for leveraging native PostgREST filtering. #1409/pool_info. #1414setup-grest.sh with -r (reset flag), as the delta registry records to insert depends on file (POSIX) timestamps. #1410grest-poll.sh. Important
gRest is an open source implementation of a query layer built over dbsync using PostgREST and HAProxy. The package is built as part of Koios team's efforts to unite community individual stream of work together and give back a more aligned structure to query dbsync and adopt standardisation to queries utilising open-source tooling as well as collaboration. In addition to these, there are also accessibility features to deploy rules for failover, do healthchecks, set up priorities, have ability to prevent DDoS attacks, provide timeouts, report tips for analysis over a longer period, etc - which can prove to be really useful when performing any analysis for instances.
Note
Note that the scripts below do allow for provisioning ogmios integration too, but Ogmios - currently - is not designed to provide advanced session management for a server-client architecture in absence of a middleware. Thus, the availability for ogmios from monitoring instance is restricted to avoid ability to DDoS an instance.
"},{"location":"Build/grest/#components","title":"Components","text":"PostgREST: An RPC JSON interface for any PostgreSQL database (in our case, database served via cardano-db-sync) to provide a RESTful Web Service. The endpoints of PostgREST in itself are essentially the table/functions defined in elected schema via grest config file. You can read more about advanced query syntax using PostgREST API here, but we will provide a simpler view using examples towards the end of the page. It is an easy alternative - with almost no overhead as it directly serves the underlying database as an API, as compared to Cardano GraphQL component (which may often have lags). Some of the other advantages of PostgREST over graphql based projects are also performance, being stateless, 0 overhead, support for JWT / native Postgres DB authentication against the Rest Interface as well.
HAProxy: An easy gateway proxy that automatically provides failover/basic DDoS protection, specify rules management for load balancing, setup multiple frontend/backends, provide easy means to have TLS enabled for public facing instances, etc. You may alter the settings for proxy layer as per your SecOps preferences. This component is optional (eg: if you prefer to expose your PostgREST server itself, you can do so using similar steps below).
To start with you'd want to ensure your current shell session has access to Postgres credentials, continuing from examples from the above mentioned Sample Postgres deployment guide.
cd $CNODE_HOME/priv\nPGPASSFILE=$CNODE_HOME/priv/.pgpass\npsql cexplorer\nEnsure that you can connect to your Postgres DB fine using above (quit from psql once validated using \\q). As part of guild-deploy.sh execution, you'd find setup-grest.sh file made available in ${CNODE_HOME}/scripts folder, which will help you automate installation of PostgREST, HAProxy as well as brings in latest queries/functions provided via Koios to your instances.
Warning
As of now, gRest services are in alpha stage - while can be utilised, please remember there may be breaking changes and every collaborator is expected to work with the team to keep their instances up-to-date using alpha branch.
Familiarise with the usage options for the setup script , the syntax can be viewed as below:
cd \"${CNODE_HOME}\"/scripts\n./setup-grest.sh -h\n#\n# Usage: setup-grest.sh [-f] [-i [p][r][m][c][d]] [-u] [-b <branch>]\n# \n# Install and setup haproxy, PostgREST, polling services and create systemd services for haproxy, postgREST and dbsync\n# \n# -f Force overwrite of all files including normally saved user config sections\n# -i Set-up Components individually. If this option is not specified, components will only be installed if found missing (eg: -i prcd)\n# p Install/Update PostgREST binaries by downloading latest release from github.\n# r (Re-)Install Reverse Proxy Monitoring Layer (haproxy) binaries and config\n# m Install/Update Monitoring agent scripts\n# c Overwrite haproxy, postgREST configs\n# d Overwrite systemd definitions\n# -u Skip update check for setup script itself\n# -q Run all DB Queries to update on postgres (includes creating grest schema, and re-creating views/genesis table/functions/triggers and setting up cron jobs)\n# -b Use alternate branch of scripts to download - only recommended for testing/development (Default: master)\n#\nTo run the setup overwriting all standard deployment tasks from a branch (eg: koios-1.0.9 branch), you may want to use:
./setup-grest.sh -f -i prmcd -r -q -b koios-1.0.9\nSimilarly - if you'd like to re-install all components and force overwrite all configs but not reset cache tables, you may run:
./setup-grest.sh -f -i prmcd -q\nAnother example could be to preserve your config, but only update queries using an alternate branch (eg: let's say you want to try the branch alpha prior to a tagged release). To do so, you may run:
./setup-grest.sh -q -b alpha\nPlease ensure to follow the on-screen instructions, if any (for example restarting deployed services, or updating configs to specify correct target postgres URLs/enable TLS/add peers etc in ${CNODE_HOME}/priv/grest.conf and ${CNODE_HOME}/files/haproxy.cfg).
The default ports used will make haproxy instance available at port 8053 or 8453 if TLS is enabled (you might want to enable firewall rule to open this port to services you would like to access). If you want to prevent unauthenticated access to grest schema, uncomment the jwt-secret and specify a custom secret-token.
Reminder
Once you've successfully deployed the grest instance, it will deploy certain cron jobs that will ensure the relevant cache tables are updated periodically. Until these have finished (especially on first run, it could take an hour or so on mainnet, your instance will likely not pass any tests from grest-poll.sh but that's expected.
In order to enable SSL on your haproxy, all you need to do is edit the file ${CNODE_HOME}/files/haproxy.cfg and update the frontend app section to uncomment ssl bind (and comment normal bind).
Info
If you're not familiar with how to configure TLS OR would not like to buy one, you can find tips on how to create a TLS certificate for free via LetsEncrypt using tutorials here. Once you do have a TLS Certificate generated, you need to chain the private key and full chain cert together in a file - /etc/ssl/server.pem - which can be then referenced as below:
frontend app\n #bind 0.0.0.0:8053\n ## If using SSL, comment line above and uncomment line below\n bind :8453 ssl crt /etc/ssl/server.pem no-sslv3\n http-request set-log-level silent\n acl srv_down nbsrv(grest_postgrest) eq 0\n acl is_wss hdr(Upgrade) -i websocket\n ...\nWith the setup, you also have a checkstatus.sh script, which will query the Postgres DB instance via haproxy (coming through postgREST), and only show an instance up if the latest block in your DB instance is within 180 seconds.
Important
If you'd like to participate in joining to the elastic cluster via Koios, please raise a PR request by editing topology files in this folder to do so!!
If you were using guild network, you could do a couple of very basic sanity checks as per below:
To query active stake for pool pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr in epoch 122, we can execute the below:
curl -d _pool_bech32=pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr -d _epoch_no=122 -s http://localhost:8053/rpc/pool_active_stake\n## {\"active_stake_sum\" : 19409732875}\nTo check latest owner key(s) for a given pool pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr, you can execute the below:
curl -d _pool_bech32=pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr -s http://localhost:8050/rpc/pool_owners\n## [{\"owner\" : \"stake_test1upx5p04dn3t6dvhfh27744su35vvasgaaq565jdxwlxfq5sdjwksw\"}, {\"owner\" : \"stake_test1uqak99cgtrtpean8wqwp7d9taaqkt9gkkxga05m5azcg27chnzfry\"}]\nYou may want to explore what all endpoints come out of the box, and test them out, to do so - refer to API documentation for OpenAPI3 documentation. Each endpoint has a pre-filled example for mainnet and connects by default to primary Koios endpoint, allowing you to test endpoints and if needed - grab the curl commands to start testing yourself against your local or remote instances.
If you're interested to participate in decentralised infrastructure by providing an instance, there are a few additional steps you'd need:
Enable ports for your HAProxy instance (default: 8053), gRest Exporter service (default: 8059) and (optionally) submit API instance (default: 8090) against the monitoring instance (do not need to open these ports to internet) of corresponding network.
Ensure that each of the service above is listening on your public IP address (for instance, submitapi.sh might need to be edited to change HOSTADDR to 0.0.0.0 and restarted).
Create a PR specifying connectivity information to your HAProxy port here.
Make sure to join the telegram discussions group to participate in any discussions, actions, polls for new-features, etc. Feel free to give a shout in the group in case you have trouble following any of the above
Reminder !!
Unless there is a very particular reason you want to compile (eg: running on non-popular OS flavor), you DO NOT \"need\" to build node binaries - guild-deploy.sh already provides an option to download pre-compiled binaries. Ensure the Pre-Requisites are in place before you proceed.
Execute the below to clone the cardano-node repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/intersectmbo/cardano-node\ncd cardano-node\nYou can use the instructions below to build the latest release of cardano-node.
git fetch --tags --recurse-submodules --all\ngit pull\n# Replace tag against checkout if you do not want to build the latest released version, we recommend using battle tested node versions - which may not always be latest\ngit checkout $(curl -sLf https://api.github.com/repos/intersectmbo/cardano-node/releases/latest | jq -r .tag_name)\n\n# Use `-l` argument if you'd like to use system libsodium instead of IOG fork of libsodium while compiling\n$CNODE_HOME/scripts/cabal-build-all.sh\nThe above would copy the binaries built into ~/.local/bin folder.
While certain folks might want to build the node themselves (could be due to OS/arch compatibility, trust factor or customisations), for most it might not make sense to build the node locally. Instead, you can download the binaries using cardano-node release notes, where-in you can find the download links for every version. This is already taken care of by guild-deploy.sh if you used the option to download binaries (you can always re-run with specific arguments if unsure).
Execute cardano-cli and cardano-node to verify output as below (the exact version and git rev should depend on your checkout tag on github repository):
cardano-cli version\n# cardano-cli 9.x.x - linux-x86_64 - ghc-8.10\n# git rev <...>\ncardano-node version\n# cardano-node 9.x.x - linux-x86_64 - ghc-8.10\n# git rev <...>\nBefore you go ahead with starting your node, you may want to update values for CNODE_PORT in $CNODE_HOME/scripts/env. Note that it is imperative for operational relays and pools to ensure that the port mentioned is opened via firewall to the destination your node is supposed to connect from. Update your network/firewall configuration accordingly. Future executions of guild-deploy.sh will preserve and not overwrite these values (or atleast back up if forced to overwrite).
CNODEBIN=\"${HOME}/.local/bin/cardano-node\"\nCCLI=\"${HOME}/.local/bin/cardano-cli\"\nCNODE_PORT=6000\nPOOL_NAME=\"GUILD\"\nImportant
POOL_NAME is the name of folder that you will use when registering pools and starting node in core mode. This folder would typically contain your hot.skey,vrf.skey and op.cert files required. If the mentioned files are absent (expected if this is a fresh install), the node will automatically start in a relay mode.
To test starting the node in interactive mode, we will make use of pre-built script cnode.sh. This script automatically determines whether to start the node as a relay or block producer (if the required pool keys are present in the $CNODE_HOME/priv/pool/<POOL_NAME> as mentioned above). If the <MITHRIL_DOWNLOAD> variable is set to 'Y' it will download the latest snapshot from a Mithril aggregator to speed up the blockchain synchronization. The script contains a user-defined variable CPU_CORES which determines the number of CPU cores the node will use upon start-up:
######################################\n# User Variables - Change as desired #\n# Common variables set in env file #\n######################################\n\n#CPU_CORES=4 # Number of CPU cores cardano-node process has access to (please don't set higher than physical core count, 4 recommended)\nNow let's test starting the node in interactive mode.
Note
At this stage, upon executing cnode.sh, you are expected to see the live config and a line ending with Listening on http://127.0.0.1:12798 - this is expected, as your logs are being written to $CNODE_HOME/logs/node.json . If so, you should be alright to return to your console by pressing Ctrl-C. The node will be started later using instructions below using systemd (Linux's service management). In case you receive any errors, please troubleshoot and fix those before proceeding.
cd \"${CNODE_HOME}\"/scripts\n./cnode.sh\nPress Ctrl-C to exit node and return to console.
"},{"location":"Build/node-cli/#modify-the-nodes-config-files","title":"Modify the node's config files","text":"Now that you've tested the basic node operation, you might want to customise your config files (assuming you are in top-level folder , i.e. cd \"${CNODE_HOME}\") :
files/config.json : This file contains the logging configurations (tracers of to tune logging, paths for other genesis config files, address/ports on which the prometheus/EKG monitoring will listen, etc). Unless running more than one node on same machine (not recommended), you should be alright to use most of this file as-is. You might - however - want to double-check PeerSharing in this file, if using a relay node where you'd like connecting peers (marked as \"advertise\": \"true\" in topology.json) to be shared , you may turn this setting to true.
files/topology.json : This file tells your node how to connect to other nodes (especially initially to start synching). You would want to update this file as below:
localRoots > accessPoints section to include your local nodes that you want persistent connection against (eg: this could be your BP and own relay nodes) against definition where trustable is set to true.advertise to true for that peer group. You do NOT want to do that on BPlocalRoots > valency (valency is the same as hotValency, not yet replaced since the example in cardano-node-wiki repo still suggests valency) to number of connections from your localRoots that you always want to keep active connection to for that node.publicRoots section as well as localRoots > accessPoints as desired, tho defaults populated should work fine. On mainnet, we did add a few additional nodes to help add more redundancy for initial sync.useLedgerAfterSlot tells the node to establish networking with nodes from defined peers to sync the node initially until reaching an absolute slot number, after which - it can start attempting to connect to peers registered as pool relays on the network. You may want this number to be relatively recent (eg: not have it 50 epochs old).Important
On BP, You'd want to set useLedgerAfterSlot to -1 for your Block Producing (Core) node - thereby, telling your Core node to remain in non-P2P mode, and ensure PeerSharing is to false.
The resultant topology file on a relay could look something like below:
{\n\"bootstrapPeers\": [\n{\n\"address\": \"backbone.cardano.iog.io\",\n\"port\": 3001\n},\n{\n\"address\": \"backbone.mainnet.emurgornd.com\",\n\"port\": 3001\n},\n{\n\"address\": \"backbone.mainnet.cardanofoundation.org\",\n\"port\": 3001\n}\n],\n\"localRoots\": [\n{\n\"accessPoints\": [\n{\"address\": \"xx.xx.xx.xx\", \"port\": 6000 , \"description\": \"Core\"},\n{\"address\": \"zz.zz.zz.zz\", \"port\": 6000 , \"description\": \"Relay2\"}\n],\n\"advertise\": false,\n\"trustable\": true,\n\"hotValency\": 2\n},\n{\n\"accessPoints\": [\n{\"address\": \"node-dus.poolunder.com\", \"port\": 6900, \"pool\": \"UNDR\", \"location\": \"EU/DE/Dusseldorf\" },\n{\"address\": \"node-syd.poolunder.com\", \"port\": 6900, \"pool\": \"UNDR\", \"location\": \"OC/AU/Sydney\" },\n{\"address\": \"194.36.145.157\", \"port\": 6000, \"pool\": \"RDLRT\", \"location\": \"EU/DE/Baden\" },\n{\"address\": \"95.216.38.251\", \"port\": 6000, \"pool\": \"RDLRT\", \"location\": \"EU/FI/Helsinki\" },\n{\"address\": \"148.72.153.168\", \"port\": 16000, \"pool\": \"AAA\", \"location\": \"NA/US/StLouis\" },\n{\"address\": \"78.47.99.41\", \"port\": 6000, \"pool\": \"AAA\", \"location\": \"EU/DE/Nuremberg\" },\n{\"address\": \"relay1-pub.ahlnet.nu\", \"port\": 2111, \"pool\": \"AHL\", \"location\": \"EU/SE/Malmo\" },\n{\"address\": \"relay2-pub.ahlnet.nu\", \"port\": 2111, \"pool\": \"AHL\", \"location\": \"EU/SE/Malmo\" },\n{\"address\": \"relay1.clio.one\", \"port\": 6010, \"pool\": \"CLIO\", \"location\": \"EU/IT/Milan\" },\n{\"address\": \"relay2.clio.one\", \"port\": 6010, \"pool\": \"CLIO\", \"location\": \"EU/IT/Bozlano\" },\n{\"address\": \"relay3.clio.one\", \"port\": 6010, \"pool\": \"CLIO\", \"location\": \"EU/IT/Bozlano\" }\n],\n\"advertise\": false,\n\"trustable\": false,\n\"hotValency\": 5,\n\"warmValency\": 10\n}\n],\n\"publicRoots\": [\n{\n\"accessPoints\": [],\n\"advertise\": false\n}\n],\n\"useLedgerAfterSlot\": 128908821\n}\nSimilarly, a typical topology file on a Core could look something like below:
{\n\"bootstrapPeers\": [],\n\"localRoots\": [\n{\n\"accessPoints\": [\n{\"address\": \"yy.yy.yy.yy\", \"port\": 6000, \"description\": \"Relay1\"},\n{\"address\": \"zz.zz.zz.zz\", \"port\": 6000, \"description\": \"Relay2\"}\n],\n\"advertise\": false,\n\"trustable\": true,\n\"hotValency\": 2\n}\n],\n\"publicRoots\": [\n{\n\"accessPoints\": [],\n\"advertise\": false\n}\n],\n\"useLedgerAfterSlot\": -1\n}\nOnce above two files are updated, since you modified the file manually - there is always a chance of human errors (eg: missing comma/quotes). Thus, we would recommend you to start the node interactively once again before proceeding.
cd \"${CNODE_HOME}\"/scripts\n./cnode.sh\nAs before, ensure you do not have any errors in the console. To stop the node, hit Ctrl-C - we will start the node as systemd later in the document.
"},{"location":"Build/node-cli/#start-the-submit-api","title":"Start the submit-api","text":"Note
An average pool operator may not require cardano-submit-api at all. Please verify if it is required for your use as mentioned here. If - however - you do run submit-api for accepting sizeable transaction load, you would want to override the default MEMPOOL_BYTES by uncommenting it in cnode.sh.
cardano-submit-api is one of the binaries built as part of cardano-node repository and allows you to submit transactions over a Web API. To run this service interactively, you can use the pre-built script below (submitapi.sh). Make sure to update submitapi.sh script to change listen IP or Port that you'd want to make this service available on.
cd $CNODE_HOME/scripts\n./submitapi.sh\nTo stop the process, hit Ctrl-C
"},{"location":"Build/node-cli/#systemd","title":"Run as systemd service","text":"The preferred way to run the node (and submit-api) is through a service manager like systemd. This section explains how to setup a systemd service file.
1. Deploy as a systemd service Execute the below command to deploy your node as a systemd service (from the respective scripts folder):
cd $CNODE_HOME/scripts\n./cnode.sh -d\n# Deploying cnode.service as systemd service..\n# cnode.service deployed successfully!!\n\n./submitapi.sh -d\n# Deploying cnode-submit-api.service as systemd service..\n# cnode-submit-api deployed successfully!!\n2. Start the service Run below commands to enable automatic start of service on startup and start it.
sudo systemctl start cnode.service\nsudo systemctl start cnode-submit-api.service\n3. Check status and stop/start commands Replace status with stop/start/restart depending on what action to take.
sudo systemctl status cnode.service\nsudo systemctl status cnode-submit-api.service\nImportant
In case you see the node exit unsuccessfully upon checking status, please verify you've followed the transition process correctly as documented below, and that you do not have another instance of node already running. It would help to check your system logs, you can also check journalctl -f -u cnode to examine startup attempt for services, and scroll up until you see output for node startup attempt) for any errors while starting node.
You can use gLiveView to monitor your node that was started as a systemd service.
cd $CNODE_HOME/scripts\n./gLiveView.sh\nImportant
In the Cardano multi-asset era, this project helps you create and submit metadata describing your assets, storing them off-chain.
"},{"location":"Build/offchain-metadata-tools/#download-pre-built-binaries","title":"Download pre-built binaries","text":"Go to input-output-hk/offchain-metadata-tools to download the binaries and place in a directory specified by PATH, e.g. $HOME/.local/bin/.
An alternative to pre-built binaries - instructions describe how to build the token-metadata-creator tool but the offchain-metadata-tools repository contains other tools as well. Build the ones needed for your installation.
Execute the below to clone the offchain-metadata-tools repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/input-output-hk/offchain-metadata-tools.git\ncd offchain-metadata-tools/token-metadata-creator\nYou can use the instructions below to build token-metadata-creator, same steps can be executed in future to update the binaries (replacing appropriate tag) as well.
git fetch --tags --all\ngit pull\n# Replace master with appropriate tag if you'd like to avoid compiling against master\ngit checkout master\n$CNODE_HOME/scripts/cabal-build-all.sh\n~/.local/bin folder."},{"location":"Build/offchain-metadata-tools/#verify","title":"Verify","text":"Verify that the tool is executable from anywhere by running:
token-metadata-creator -h\n!> - An average pool operator may not require cardano-wallet at all. Please verify if it is required for your use as mentioned here.
Ensure the Pre-Requisites are in place before you proceed.
"},{"location":"Build/wallet/#build-instructions","title":"Build Instructions","text":"Follow instructions below for building the cardano-wallet binary:
"},{"location":"Build/wallet/#clone-the-repository","title":"Clone the repository","text":"Execute the below to clone the cardano-wallet repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/cardano-foundation/cardano-wallet\ncd cardano-wallet\nYou can use the instructions below to build the latest release of cardano-wallet.
!> - Note that the latest release of cardano-wallet may not work with the latest release of cardano-node. Please check the compatibility of each cardano-wallet release yourself in the official docs, e.g. https://github.com/cardano-foundation/cardano-wallet/releases/latest.
git fetch --tags --all\ngit pull\n# Replace tag against checkout if you do not want to build the latest released version\ngit checkout $(curl -s https://api.github.com/repos/cardano-foundation/cardano-wallet/releases/latest | jq -r .tag_name)\n$CNODE_HOME/scripts/cabal-build-all.sh\nThe above would copy the binaries into ~/.local/bin folder.
You can run the below to connect to a cardano-node instance that is expected to be already running and the wallet will start syncing.
cardano-wallet serve /\n --node-socket $CNODE_HOME/sockets/node.socket /\n --mainnet / # if using the testnet flag you also need to specify the testnet shelley-genesis.json file\n--database $CNODE_HOME/priv/wallet\ncardano-wallet network information\nOk.\n{\n\"network_tip\": {\n\"time\": \"2021-06-01T17:31:05Z\",\n\"epoch_number\": 269,\n\"absolute_slot_number\": 31002374,\n\"slot_number\": 157574\n},\n\"node_era\": \"mary\",\n\"node_tip\": {\n\"height\": {\n\"quantity\": 5795127,\n\"unit\": \"block\"\n},\n\"time\": \"2021-06-01T17:31:00Z\",\n\"epoch_number\": 269,\n\"absolute_slot_number\": 31002369,\n\"slot_number\": 157569\n},\n\"sync_progress\": {\n\"status\": \"ready\"\n},\n\"next_epoch\": {\n\"epoch_start_time\": \"2021-06-04T21:44:51Z\",\n\"epoch_number\": 270\n}\n}\nIf you're creating a new wallet, you'd first want to generate a mnemonic for use (see below):
cardano-wallet recovery-phrase generate\n# false brother typical saddle settle phrase foster sauce ask sunset firm gate service render burger\ncardano-wallet wallet create from-recovery-phrase MyWalletName\nPlease enter a 15\u201324 word recovery phrase: false brother typical saddle settle phrase foster sauce ask sunset firm gate service render burger\n(Enter a blank line if you do not wish to use a second factor.)\nPlease enter a 9\u201312 word second factor:\nPlease enter a passphrase: **********\nEnter the passphrase a second time: **********\nOk.\n{\n ...\n}\nMithril Networks provide the ability to download and bootstrap cardano nodes via snapshots of the the Cardano blockchain. This is a great way to speed up the process of syncing a new node, especially for stake pool operators. The tools provided by Guild Operators are designed to facilitate the ease of use in setting up and managing the:
The env file contains a new environment variable MITHRIL_DOWNLOAD that when enabled allows the cnode.sh script to automatically download the latest Mithril snapshot if the local db directory is empty. This is useful for new nodes that need to be bootstrapped with the latest snapshot to avoid synchronizing the entire blockchain from scratch. While also providing a high level of trust that the snapshot is valid since it is signed by multiple pool operators.
The architecture for Mithril Networks is described in detail at Mithril network architecture by CF/IOHK. However the architecture suggested and supported by the Guild Operators tools is not identical to the upstream documentation in that we provide a more simplified approach to the setup and management of the Mithril Network components and tools that allow setting up a Squid Mithril relays and an Nginx loadbalancer (aka sidecar) local to the Mithril signer. The Nginx sidecar provides the ability to loadbalance requests to multiple Squid based Mithril Relays running on each of the SPO's Cardano Relay nodes.
"},{"location":"Mithril/mithril-overview/#single-relay-architecture","title":"Single Relay Architecture","text":"For SPO's who only have a single Cardano relay node, an Squid based Mithril relay can be run on the same node as the Cardano relay. This can be used by the Mithril signer to submit the snapshot signatures to the Mithril Aggregator.
"},{"location":"Mithril/mithril-overview/#multi-relay-architecture","title":"Multi Relay Architecture","text":"For SPO's who have multiple Cardano relay nodes, a Nginx relay sidecar can be run on the Block Producer and load balance requests over mutliple Cardano relay nodes, each running its own Nginx Mithril relay to pass the signature along to the Mithril aggregator. This can be used to avoid a single point of failure in case a Relay server is offline for any reason. This provides high availability for the Mithril signer through multiple relays as long as the local Nginx Mithril relay is running on the same server as the Cardano Block Producer node.
"},{"location":"Mithril/mithril-overview/#installation","title":"Installation","text":"The installation of the Mithril tools is automated via guild-deploy.sh. To participate in a Mithril network include the -s m flag which will install the Mithril Client and Mithril Signer release binaries to \"${HOME}\"/.local/bin.
guild-deploy.sh -s m\nThe Mithril client is used to download a snapshot of the Cardano blockchain from a Mithril Aggregator. The snapshot is then used to bootstrap a new Cardano node. The Mithril client can be used to download the latest snapshot, list all available snapshots, or show details of a specific snapshot.
To bootstrap a Cardano node using the Mithril client, follow these steps:
Setup the Cardano Node: Use the guild tools to setup the Cardano node, either by building the binaries or using pre-compiled binaries. Follow the instructions in the guild-operators documentation.
Create the Mithril environment file: Run the script with the environment setup command. This will create a new mithril.env file with all the necessary environment variables for the Mithril client.
./mithril-client.sh environment setup\nsnapshot download command. This snapshot contains the latest state of the Cardano blockchain db from a Mithril Aggregator../mithril-client.sh snapshot download\nThe Mithril signer is used to participate in the creation of stake based signatures of snapshots. The Mithril signer can be used to sign a snapshots. The signed snapshot is then submitted to a Mithril Aggregator, via a Squid based Mithril Relay.
The first step to participate in the Mithril network is to deploy your Squid based Mithril Relays. The Mithril relay is used to provide a private and highly available network for submitting the snapshots to a Mithril Aggregator.
"},{"location":"Mithril/mithril-overview/#deploying-the-squid-mithril-relay","title":"Deploying the Squid Mithril Relay","text":"To deploy your Squid based Mithril Relays with your Cardano relay node, follow these steps:
Deploy the Squid Mithril Relay: Run the mithril-relay.sh script:
Use the -d flag to deploy the Squid Mithril Relay.
./mithril-relay.sh -d\n\nInstalling squid proxy\nEnter the IP address of your Block Producer: 1.2.3.4\nEnter the relays listening port (press Enter to use default 3132):\nUsing port 3132 for relays listening port.\nCreate the appropriate firewall rule: sudo ufw allow from 1.2.3.4 to any port 3132 proto tcp\n sudo systemctl enable --now squid\nDeploy the Mithril Signer: Run the mithril-signer.sh script:
Use the -e flag to update the mithril.env file with the Mithril Signer environment variables.
Optionally provide the relays listening port when prompted to use a port.
./mithril-signer.sh -e\n Enter the IP address of the relay endpoint: 4.5.6.7\n Enter the port of the relay endpoint (press Enter to use default 3132):\n Using RELAY_ENDPOINT=4.5.6.7:3132 for the Mithril signer relay endpoint.\nUse the -d flag to deploy the Mithril Signer.
./mithril-signer.sh -d\n Creating cnode-mithril-signer systemd service environment file..\n Mithril signer service successfully deployed\nEnable the Systemd service to start the Mithril Signer on boot.
sudo systemctl enable cnode-mithril-signer\nDeploy the Nginx sidecar loadbalancer: Run the mithril-relay.sh script:
Use the -l flag to deploy the Nginx Mithril Relay.
Create the appropriate firewall rule to allow traffic from your Block Producer to the Mithril Relay(s).
./mithril-relay.sh -l\n\nInstalling nginx load balancer\nEnter the IP address of a relay: 4.5.6.7\nAre there more relays? (y/n) y\nEnter the IP address of a relay: 8.9.10.11\nAre there more relays? (y/n) n\nEnter the IP address of the load balancer (press Enter to use default 127.0.0.1):\nUsing IP address 127.0.0.1 for the load balancer configuration.\nEnter the relays listening port (press Enter to use default 3132):\nUsing port 3132 for relays listening port.\nStarting Mithril relay sidecar (nginx load balancer)\nEnable the Systemd Nginx Mithril Relay service to start on boot.
sudo systemctl enable --now nginx\nDeploy the Mithril Signer: Run the mithril-signer.sh script:
Use the -e flag to update the mithril.env file with the Mithril Signer environment variables.
Optionally provide the loadbalancer listening port when prompted to use a port.
./mithril-signer.sh -e\n Enter the IP address of the relay endpoint: 127.0.0.1\n Enter the port of the relay endpoint (press Enter to use default 3132):\n Using RELAY_ENDPOINT=127.0.0.1:3132 for the Mithril signer relay endpoint.\nUse the -d flag to deploy the Mithril Signer.
./mithril-signer.sh -d\n Creating cnode-mithril-signer systemd service environment file..\n Mithril signer service successfully deployed\nEnable the Systemd service to start the Mithril Signer on boot.
sudo systemctl enable cnode-mithril-signer\nReminder !!
Ensure the Pre-Requisites are in place before you proceed.
blockPerf.sh is a script to monitor the network propagation of new blocks as seen by the local cardano-node.
Although blockPerf can also run on the block producer, it makes the most sense to run it on the upstream relays. There it waits for each new block announced to the relay over the network by its remote peers.
It looks for the delay times that result
You can view this data locally as a console stream, or run it as a systemd service in background.
BlockPerf also sends this data to the TopologyUpdater server, so that there is a possibility to compare this data (similar to sendtip to pooltool). As a contributing operator you get the possibility to see how your own relays compare to other nodes regarding receive quality, delay times and thus performance.
There is no connection or constraint between the TopologyUpdater Relay subscription and the BlockPerf analysis. BlockPerf is even designed to work outside the cnTools suite.
The results of these data are a good basis to make optimizations and to evaluate which changes were useful or might by required to improve the performance compared to other relay nodes.
"},{"location":"Scripts/blockperf/#installation","title":"Installation","text":"The script is best run as a background process. This can be accomplished in many ways but the preferred method is to run it as a systemd service. A terminal multiplexer like tmux or screen could also be used but not covered here.
"},{"location":"Scripts/blockperf/#run-as-service","title":"Run as service","text":"Use the deploy-as-systemd.sh script to create a systemd unit file. In this setup the script is started in \"service\" mode. Error/Warn level log output is handled by journald. journalctl -f -u cnode-tu-blockperf.service can be used to check service output (follow mode).
Outside the cnTools environment call blockPerf.sh -d to install it as a systemd service.
If you run blockPerf local in the console (scripts/blockPerf.sh) , immediately after the appearance of a new block it shows where it came from, how many slots away from the previous block it was, and how many milliseconds the individual steps took.
Block:.... 6860534\n Slot..... 52833850 (+59s)\n ......... 2022-02-09 09:49:01\n Header... 2022-02-09 09:49:02,780 (+1780 ms)\n Request.. 2022-02-09 09:49:02,780 (+0 ms)\n Block.... 2022-02-09 09:49:02,830 (+50 ms)\n Adopted.. 2022-02-09 09:49:02,900 (+70 ms)\n Size..... 79976 bytes\n delay.... 1.819971868 sec\n From..... 104.xxx.xxx.61:3001\n\nBlock:.... 6860535\n Slot..... 52833857 (+7s)\n ......... 2022-02-09 09:49:08\n Header... 2022-02-09 09:49:08,960 (+960 ms)\n Request.. 2022-02-09 09:49:08,970 (+10 ms)\n Block.... 2022-02-09 09:49:09,020 (+50 ms)\n Adopted.. 2022-02-09 09:49:09,090 (+70 ms)\n Size..... 64950 bytes\n delay.... 1.028341023 sec\n From..... 34.xxx.xxx.15:4001\nA further aim of the blockPerf project is to use the data that individual nodes send to the central TopologyUpdater database to produce graphical visualisations and evaluations that provide the participating node operators with useful insights into their performance compared to all others.
"},{"location":"Scripts/cncli/","title":"CNCLI","text":"Reminder !!
Ensure the Pre-Requisites are in place before you proceed.
cncli.sh is a script to download and deploy CNCLI created and maintained by Andrew Westberg. It's a community-based CLI tool written in RUST for low-level cardano-node communication. Usage is optional and no script is dependent on it. The main features include:
gLiveView for peer analysis if available. sqlite database. firstSlotOfNextEpoch - (3 * k / f)).cncli.sh script's main functions, sync, leaderlog, validate and PoolTool sendslots/sendtip are not meant to be run manually, but instead deployed as systemd services that run in the background to do the block scraping and validation automatically. Additional commands exist for manual execution to initiate the sqlite db, filling the blocklog DB with all blocks created by the pool known to the blockchain, migration of old cntoolsBlockCollector JSON blocklog, and re-validation of blocks and leaderlogs. See usage output below for a complete list of available commands.
The script works in tandem with Log Monitor to provide faster adopted status but mainly to catch slots the node is leader for but are unable to create a block for. These are marked as invalid. Blocklog will however work fine without the logMonitor service and CNCLI is able to handle everything except catching invalid blocks.
guild-deploy.sh with guild-deploy.sh -s c to download and install RUST and CNCLI. IOG fork of libsodium required by CNCLI is automatically compiled by CNCLI build process. If a previous installation is found, RUST and CNCLI will be updated to the latest version.deploy-as-systemd.sh to deploy the systemd services that handle all the work in the background. Six systemd services in total are deployed whereof four are related to CNCLI. See above for the different purposes they serve.If you want to disable some of the deployed services, run sudo systemctl disable <service>
cnode.service (main cardano-node launcher)
cnode-cncli-sync.servicecnode-cncli-leaderlog.servicecnode-cncli-validate.servicecnode-cncli-ptsendtip.servicecnode-cncli-ptsendslots.servicecnode-logmonitor.service (see Log Monitor)You can override the values in the script at the User Variables section shown below. POOL_ID, POOL_VRF_SKEY and POOL_VRF_VKEY should automatically be detected if POOL_NAME is set in the common env file and can be left commented. PT_API_KEY and POOL_TICKER need to be set in the script if PoolTool sendtip/sendslots are to be used before starting the services. For the rest of the commented values, if the defaults do not provide the right values, uncomment and make adjustments.
#POOL_ID=\"\" # Automatically detected if POOL_NAME is set in env. Required for leaderlog calculation & pooltool sendtip, lower-case hex pool id\n#POOL_VRF_SKEY=\"\" # Automatically detected if POOL_NAME is set in env. Required for leaderlog calculation, path to pool's vrf.skey file\n#POOL_VRF_VKEY=\"\" # Automatically detected if POOL_NAME is set in env. Required for block validation, path to pool's vrf.vkey file\n#PT_API_KEY=\"\" # POOLTOOL sendtip: set API key, e.g \"a47811d3-0008-4ecd-9f3e-9c22bdb7c82d\"\n#POOL_TICKER=\"\" # POOLTOOL sendtip: set the pools ticker, e.g. \"TCKR\"\n#PT_HOST=\"127.0.0.1\" # POOLTOOL sendtip: connect to a remote node, preferably block producer (default localhost)\n#PT_PORT=\"${CNODE_PORT}\" # POOLTOOL sendtip: port of node to connect to (default is CNODE_PORT from the env file)\n#CNCLI_DIR=\"${CNODE_HOME}/guild-db/cncli\" # path to the directory for cncli sqlite db\n#SLEEP_RATE=60 # CNCLI leaderlog/validate: time to wait until next check (in seconds)\n#CONFIRM_SLOT_CNT=600 # CNCLI validate: require at least these many slots to have passed before validating\n#CONFIRM_BLOCK_CNT=15 # CNCLI validate: require at least these many blocks on top of minted before validating\n#TIMEOUT_LEDGER_STATE=300 # CNCLI leaderlog: timeout in seconds for ledger-state query\n#BATCH_AUTO_UPDATE=N # Set to Y to automatically update the script if a new version is available without user interaction\nServices are controlled by sudo systemctl <status|start|stop|restart> <service name> All services are configured as child services to cnode.service and as such, when an action is taken against this service it's replicated to all child services. E.g running sudo systemctl start cnode.service will also start all child services.
Log output is handled by journald. journalctl -f -u <service> can be used to check service output (follow mode). Other logging configurations are not covered here.
Recommended workflow to get started with CNCLI blocklog.
$CNODE_HOME/scripts/cncli.sh migrate <path> where is the location to the directory containing all blocks_.json files. sudo systemctl start cnode-cncli-sync.service (starts leaderlog, validate & ptsendslots automatically)sudo systemctl start cnode-logmonitor.servicesudo systemctl start cnode-cncli-ptsendtip.service (optional but recommended)sudo systemctl restart cnode.service$CNODE_HOME/scripts/cncli.sh initUsage: cncli.sh [operation <sub arg>]\nScript to run CNCLI, best launched through systemd deployed by 'deploy-as-systemd.sh'\n\nsync Start CNCLI chainsync process that connects to cardano-node to sync blocks stored in SQLite DB (deployed as service)\nleaderlog One-time leader schedule calculation for current epoch, then continuously monitors and calculates schedule for coming epochs, 1.5 days before epoch boundary on the mainnet (deployed as service)\n force Manually force leaderlog calculation and overwrite even if already done, exits after leaderlog is calculated\nvalidate Continuously monitor and confirm that the blocks made actually was accepted and adopted by chain (deployed as service)\n all One-time re-validation of all blocks in blocklog db\n epoch One-time re-validation of blocks in blocklog db for the specified epoch \nptsendtip Send node tip to PoolTool for network analysis and to show that your node is alive and well with a green badge (deployed as service)\nptsendslots Securely sends PoolTool the number of slots you have assigned for an epoch and validates the correctness of your past epochs (deployed as service)\ninit One-time initialization adding all minted and confirmed blocks to blocklog\nmigrate One-time migration from old blocklog (cntoolsBlockCollector) to new format (post cncli)\n path Path to the old cntoolsBlockCollector blocklog folder holding json files with blocks created\nBest and easiest viewed in CNTools and gLiveView but the blocklog database is a SQLite DB so if you are comfortable with SQL, the sqlite3 command can be used to query the DB.
Block status
- Leader : Scheduled to make block at this slot\n- Ideal : Expected/Ideal number of blocks assigned based on active stake (sigma)\n- Luck : Leader slots assigned vs ideal slots for this epoch\n- Adopted : Block created successfully\n- Confirmed : Block created validated to be on-chain with the certainty set in `cncli.sh` for `CONFIRM_BLOCK_CNT`\n- Missed : Scheduled at slot but no record of it in CNCLI DB and no other pool has made a block for this slot\n- Ghosted : Block created but marked as orphaned and no other pool has made a valid block for this slot -> height battle or block propagation issue\n- Stolen : Another pool has a valid block registered on-chain for the same slot\n- Invalid : Pool failed to create block, base64 encoded error message can be decoded with `echo <base64 hash> | base64 -d | jq -r`\nOpen CNTools and select [b] Blocks to open the block viewer. Either select Epoch and enter the epoch you want to see a detailed view for or choose Summary to display blocks for last x epochs.
If the node was elected to create blocks in the selected epoch it could look something like this:
Summary >> BLOCKS\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nCurrent epoch: 96\n\n+--------+---------------------------+----------------------+--------------------------------------+\n| Epoch | Leader | Ideal | Luck | Adopted | Confirmed | Missed | Ghosted | Stolen | Invalid |\n+--------+---------------------------+----------------------+--------------------------------------+\n| 96 | 34 | 31.66 | 107.39% | 18 | 18 | 0 | 0 | 0 | 0 |\n| 95 | 32 | 30.57 | 104.68% | 32 | 32 | 0 | 0 | 0 | 0 |\n+--------+---------------------------+----------------------+--------------------------------------+\n\n[h] Home | [b] Block View | [i] Info | [*] Refresh\n >> BLOCKS\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nCurrent epoch: 96\n\n+---------------------------+----------------------+--------------------------------------+\n| Leader | Ideal | Luck | Adopted | Confirmed | Missed | Ghosted | Stolen | Invalid |\n+---------------------------+----------------------+--------------------------------------+\n| 34 | 31.66 | 107.39% | 18 | 18 | 0 | 0 | 0 | 0 |\n+---------------------------+----------------------+--------------------------------------+\n\n+-----+------------+----------+---------------------+--------------------------+-------+-------------------------------------------------------------------+\n| # | Status | Block | Slot | SlotInEpoch | Scheduled At | Size | Hash |\n+-----+------------+----------+---------------------+--------------------------+-------+-------------------------------------------------------------------+\n| 1 | confirmed | 2043444 | 11142827 | 40427 | 2020-11-16 08:34:03 CET | 3 | ec216d3fb01e4a3cc3e85305145a31875d9561fa3bbcc6d0ee8297236dbb4115 |\n| 2 | confirmed | 2044321 | 11165082 | 62682 | 2020-11-16 14:44:58 CET | 3 | b75c33a5bbe49a74e4b4cc5df4474398bfb10ed39531fc65ec2acc51f89ddce5 |\n| 3 | confirmed | 2044397 | 11166970 | 64570 | 2020-11-16 15:16:26 CET | 3 | c1ea37fd72543779b6dab46e3e29e0e422784b5fd6188f828ace9eabcc87088f |\n| 4 | confirmed | 2044879 | 11178909 | 76509 | 2020-11-16 18:35:25 CET | 3 | 35a116cec80c5dc295415e4fc8e6435c562b14a5d6833027006c988706c60307 |\n| 5 | confirmed | 2046965 | 11232557 | 130157 | 2020-11-17 09:29:33 CET | 3 | d566e5a1f6a3d78811acab4ae3bdcee6aa42717364f9afecd6cac5093559f466 |\n| 6 | confirmed | 2047101 | 11235675 | 133275 | 2020-11-17 10:21:31 CET | 3 | 3a638e01f70ea1c4a660fe4e6333272e6c61b11cf84dc8a5a107b414d1e057eb |\n| 7 | confirmed | 2047221 | 11238453 | 136053 | 2020-11-17 11:07:49 CET | 3 | 843336f132961b94276603707751cdb9a1c2528b97100819ce47bc317af0a2d6 |\n| 8 | confirmed | 2048692 | 11273507 | 171107 | 2020-11-17 20:52:03 CET | 3 | 9b3eb79fe07e8ebae163870c21ba30460e689b23768d2e5f8e7118c572c4df36 |\n| 9 | confirmed | 2049058 | 11282619 | 180219 | 2020-11-17 23:23:55 CET | 3 | 643396ea9a1a2b6c66bb83bdc589fa19c8ae728d1f1181aab82e8dfe508d430a |\n| 10 | confirmed | 2049321 | 11289237 | 186837 | 2020-11-18 01:14:13 CET | 3 | d93d305a955f40b2298247d44e4bc27fe9e3d1486ef3ef3e73b235b25247ccd7 |\n| 11 | confirmed | 2049747 | 11299205 | 196805 | 2020-11-18 04:00:21 CET | 3 | 19a43deb5014b14760c3e564b41027c5ee50e0a252abddbfcac90c8f56dc0245 |\n| 12 | confirmed | 2050415 | 11316075 | 213675 | 2020-11-18 08:41:31 CET | 3 | dd2cb47653f3bfb3ccc8ffe76906e07d96f1384bafd57a872ddbab3b352403e3 |\n| 13 | confirmed | 2050505 | 11318274 | 215874 | 2020-11-18 09:18:10 CET | 3 | deb834bc42360f8d39eefc5856bb6d7cabb6b04170c842dcbe7e9efdf9dbd2e1 |\n| 14 | confirmed | 2050613 | 11320754 | 218354 | 2020-11-18 09:59:30 CET | 3 | bf094f6fde8e8c29f568a253201e4b92b078e9a1cad60706285e236a91ec95ff |\n| 15 | confirmed | 2050807 | 11325239 | 222839 | 2020-11-18 11:14:15 CET | 3 | 21f904346ba0fd2bb41afaae7d35977cb929d1d9727887f541782576fc6a62c9 |\n| 16 | confirmed | 2050997 | 11330062 | 227662 | 2020-11-18 12:34:38 CET | 3 | 109799d686fe3cad13b156a2d446a544fde2bf5d0e8f157f688f1dc30f35e912 |\n| 17 | confirmed | 2051286 | 11336791 | 234391 | 2020-11-18 14:26:47 CET | 3 | bb1beca7a1d849059110e3d7dc49ecf07b47970af2294fe73555ddfefb9561a8 |\n| 18 | confirmed | 2051734 | 11348498 | 246098 | 2020-11-18 17:41:54 CET | 3 | 87940b53c2342999c1ba4e185038cda3d8382891a16878a865f5114f540683de |\n| 19 | leader | - | 11382001 | 279601 | 2020-11-19 03:00:17 CET | - | - |\n| 20 | leader | - | 11419959 | 317559 | 2020-11-19 13:32:55 CET | - | - |\n| 21 | leader | - | 11433174 | 330774 | 2020-11-19 17:13:10 CET | - | - |\n| 22 | leader | - | 11434241 | 331841 | 2020-11-19 17:30:57 CET | - | - |\n| 23 | leader | - | 11435289 | 332889 | 2020-11-19 17:48:25 CET | - | - |\n| 24 | leader | - | 11440314 | 337914 | 2020-11-19 19:12:10 CET | - | - |\n| 25 | leader | - | 11442361 | 339961 | 2020-11-19 19:46:17 CET | - | - |\n| 26 | leader | - | 11443861 | 341461 | 2020-11-19 20:11:17 CET | - | - |\n| 27 | leader | - | 11446997 | 344597 | 2020-11-19 21:03:33 CET | - | - |\n| 28 | leader | - | 11453110 | 350710 | 2020-11-19 22:45:26 CET | - | - |\n| 29 | leader | - | 11455323 | 352923 | 2020-11-19 23:22:19 CET | - | - |\n| 30 | leader | - | 11505987 | 403587 | 2020-11-20 13:26:43 CET | - | - |\n| 31 | leader | - | 11514983 | 412583 | 2020-11-20 15:56:39 CET | - | - |\n| 32 | leader | - | 11516010 | 413610 | 2020-11-20 16:13:46 CET | - | - |\n| 33 | leader | - | 11518958 | 416558 | 2020-11-20 17:02:54 CET | - | - |\n| 34 | leader | - | 11533254 | 430854 | 2020-11-20 21:01:10 CET | - | - |\n+-----+------------+----------+---------------------+--------------------------+-------+-------------------------------------------------------------------+\nCurrently shows a block summary for current epoch. For full block details use CNTools for now. Invalid, missing, ghosted and stolen blocks only shown in case of a non-zero value.
\u2502--------------------------------------------------------------\u2502\n\u2502 BLOCKS Leader | Ideal | Luck | Adopted | Confirmed \u2502\n\u2502 24 27.42 87.53% 1 1 \u2502\n\u2502 08:07:57 until leader XXXXXXXXX.....................\u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nAll notable changes to this tool will be documented in this file.
Whenever you're updating between versions where format/hash of keys have changed , or you're changing networks - it is recommended to Backup your Wallet and Pool folders before you proceed with launching cntools on a fresh network.
The format is based on Keep a Changelog, and this adheres to Semantic Versioning.
"},{"location":"Scripts/cntools-changelog/#1330-2024-11-21","title":"[13.3.0] - 2024-11-21","text":""},{"location":"Scripts/cntools-changelog/#added","title":"Added","text":"light mode support for all governance functions.test_koios call from cntools.library to cntools.shdialog by default, it is an optional component - and no longer installed by default.--whole-utxo flag, as it returns all address and will not accept --address--whole-utxo flag when query UTxO, as required by cardano-cli 1.28, to keep behaviour same as before.AdvancedThough mostly unchanged in the user interface, this is a major update with most of the code re-written/touched in the back-end. Only the most noticeable changes added to changelog.
"},{"location":"Scripts/cntools-changelog/#added_15","title":"Added","text":"--cold-verification-key-file instead of --verification-key-fileThis is a major release with a lot of changes. It is highly recommended that you familiarise yourself with the usage for Hybrid or Online v/s Offline mode on a testnet environment before doing it on production. Please visit https://cardano-community.github.io/guild-operators/upgrade for details.
"},{"location":"Scripts/cntools-changelog/#added_18","title":"Added","text":"cardano-address and bech32 in yout $PATH to use this feature (available if you rebuild cardano-node using updated cabal-build-all.sh), reusing guide from @ilap.srm) when available when deleting files.,) in user input for sending ADA and pledge/cost at pool registration to make it easier to count the zeroscardano-node 1.19.0, please upgrade if you're not using this version.Pool >> Show now moved to its own menu option This is to de-clutter and because it takes time to parse this data from ledger-statePool >> Delegators removed.pool >> show stake distribution showing up as always 0.prereqs.sh -t) fix for internal update--output-format hex when extracting pool ID in hex formatWallet >> Encrypt as these are re-generated from keys and need to be writableFunds >> Withdraw for base address as this is used to pay the withdraw transaction feePool >> Show delegator rewards parsing from ledger-statemainnet_candidate, and add second argument (g) to run prereqs against guild network[c] to [Esc]Wallet >> Show2.1.1 included a change to env file and thus require a major version bump.Pool >> ShowPool >> Show(stake + reward) is below pledge (single-owner only for now)Pool >> ShowPool >> New to Pool >> Register.Wallet >> List Not a registered wallet on chain information from Wallet listingPool >> ShowImportant
Familiarize yourself with the Online workflow of creating wallets and pools on the Preview/Preprod/Guild network first. You can then move on to test the Offline Workflow. The Offline workflow means that the private keys never touch the Online node. When comfortable with both the online and offline CNTools workflow, it's time to deploy what you learnt on the mainnet.
This chapter describes some common use-cases for wallet and pool creation when running CNTools in Online mode. CNTools contains much more functionality not described here.
Create WalletA wallet is needed for pledge and to pay for pool registration fee.
[w] Wallet and you will be presented with the following menu: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Wallet Management\n\n ) New - create a new wallet\n ) Import - import a Daedalus/Yoroi 24/25 mnemonic or Ledger/Trezor HW wallet\n ) Register - register a wallet on chain\n ) De-Register - De-Register (retire) a registered wallet\n ) List - list all available wallets in a compact view\n ) Show - show detailed view of a specific wallet\n ) Remove - remove a wallet\n ) Decrypt - remove write protection and decrypt wallet\n ) Encrypt - encrypt wallet keys and make all files immutable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Select Wallet Operation\n\n [n] New\n [i] Import\n [r] Register\n [z] De-Register\n [l] List\n [s] Show\n [x] Remove\n [d] Decrypt\n [e] Encrypt\n [h] Home\n[n] New to create a new wallet. [i] Import can also be used to import a Daedalus/Yoroi based 15 or 24 word wallet seed ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET >> NEW\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nName of new wallet: Test\n\nNew Wallet : Test\nAddress : addr_test1qpq5qjr774cyc6kxcwp060k4t4hwp42q43v35lmcg3gcycu5uwdwld5yr8m8fgn7su955zf5qahtrgljqfjfa4nr8jfsj4alxk\nEnterprise Address : addr_test1vpq5qjr774cyc6kxcwp060k4t4hwp42q43v35lmcg3gcyccuxhdka\n\nYou can now send and receive Ada using the above addresses.\nNote that Enterprise Address will not take part in staking.\nWallet will be automatically registered on chain if you\nchoose to delegate or pledge wallet when registering a stake pool.\nThe Import feature of CNTools is originally based on this guide from Ilap.
If you would like to use Import function to import a Daedalus/Yoroi based 15 or 24 word wallet seed, please ensure that cardano-address and bech32 bineries are available in your $PATH environment variable:
bech32 --version\n1.1.0\n\ncardano-address --version\n3.5.0\nIf the version is not as per above, please run the latest guild-deploy.sh from here and rebuild cardano-node as instructed here.
To import a Daedalus/Yoroi wallet to CNTools, open CNTools and select the [w] Wallet option, and then select the [i] Import, the following menu will appear:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET >> IMPORT\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Wallet Import\n\n ) Mnemonic - Daedalus/Yoroi 24 or 25 word mnemonic\n ) HW Wallet - Ledger/Trezor hardware wallet\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Select Wallet operation\n\n [m] Mnemonic\n [w] HW Wallet\n [h] Home\nNote
You can import Hardware wallet using [w] HW Wallet above, but please note that before you are able to use hardware wallet in CNTools, you need to ensure you can detect your hardware device at OS level using cardano-hw-cli
Select the wallet you want to import, for Daedalus / Yoroi wallets select [m] Mnemonic:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET >> IMPORT >> MNEMONIC\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nName of imported wallet: TEST\n\n24 or 15 word mnemonic(space separated):\nCreate the necessary pool keys.
[p] Pool ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Pool Management\n\n ) New - create a new pool\n ) Register - register created pool on chain using a stake wallet (pledge wallet)\n ) Modify - change pool parameters and register updated pool values on chain\n ) Retire - de-register stake pool from chain in specified epoch\n ) List - a compact list view of available local pools\n ) Show - detailed view of specified pool\n ) Rotate - rotate pool KES keys\n ) Decrypt - remove write protection and decrypt pool\n ) Encrypt - encrypt pool cold keys and make all files immutable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Select Pool Operation\n\n [n] New\n [r] Register\n [m] Modify\n [x] Retire\n [l] List\n [s] Show\n [o] Rotate\n [d] Decrypt\n [e] Encrypt\n [h] Home\n[n] New to create a new pool ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL >> NEW\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPool Name: TEST\n\nPool: TEST\nID (hex) : 8d5a3510f18ce241115da38a1b2419ed82d308599c16e98caea1b4c0\nID (bech32) : pool134dr2y833n3yzy2a5w9pkfqeakpdxzzenstwnr9w5x6vqtnclue\nRegister the pool on-chain.
[p] Pool [r] Register Make sure you set your pledge low enough to insure your funds in your wallet will cover pledge plus pool registration fees.
Test in our case. As this is a newly created wallet, you will be prompted to continue with wallet registration. When complete and if successful, both wallet and pool will be registered on-chain.It will look something like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL >> REGISTER\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOnline mode - The default mode to use if all keys are available\n\nHybrid mode - 1) Go through the steps to build a transaction file\n 2) Copy the built tx file to an offline node\n 3) Sign it using 'Sign Tx' with keys on offline node\n (CNTools started in offline mode '-o' without node connection)\n 4) Copy the signed tx file back to the online node and submit using 'Submit Tx'\n\nSelected value: [o] Online\n\n# Select pool\nSelected pool: TEST\n\n# Pool Parameters\npress enter to use default value\n\nPledge (in Ada, default: 50,000):\nMargin (in %, default: 7.5):\nCost (in Ada, minimum: 340, default: 340):\n\n# Pool Metadata\n\nEnter Pool's JSON URL to host metadata file - URL length should be less than 64 chars (default: https://foo.bat/poolmeta.json):\n\nEnter Pool's Name (default: TEST):\nEnter Pool's Ticker , should be between 3-5 characters (default: TEST):\nEnter Pool's Description (default: No Description):\nEnter Pool's Homepage (default: https://foo.com):\n\nOptionally set an extended metadata URL?\nSelected value: [n] No\n{\n \"name\": \"TEST\",\n \"ticker\": \"TEST\",\n \"description\": \"No Description\",\n \"homepage\": \"https://foo.com\",\n \"nonce\": \"1613146429\"\n}\n\nPlease host file /opt/cardano/guild/priv/pool/TEST/poolmeta.json as-is at https://foo.bat/poolmeta.json\n\n# Pool Relay Registration\nSelected value: [d] A or AAAA DNS record (single)\nEnter relays's DNS record, only A or AAAA DNS records: relay.foo.com\nEnter relays's port: 6000\nAdd more relay entries?\nSelected value: [n] No\n\n# Select main owner/pledge wallet (normal CLI wallet)\nSelected wallet: Test (100,000.000000 Ada)\nWallet Test3 not registered on chain\n\nWaiting for new block to be created (timeout = 600 slots, 600s)\nINFO: press any key to cancel and return (won't stop transaction)\n\nOwner #1 : Test added!\n\nRegister a multi-owner pool (you need to have stake.vkey of any additional owner in a seperate wallet folder under $CNODE_HOME/priv/wallet)?\nSelected value: [n] No\n\nUse a separate rewards wallet from main owner?\nSelected value: [n] No\n\nWaiting for new block to be created (timeout = 600 slots, 600s)\nINFO: press any key to cancel and return (won't stop transaction)\n\nPool TEST successfully registered!\nOwner #1 : Test\nReward Wallet : Test\nPledge : 50,000 Ada\nMargin : 7.5 %\nCost : 340 Ada\n\nUncomment and set value for POOL_NAME in ./env with 'TEST'\n\nINFO: Total balance in 1 owner/pledge wallet(s) are: 99,497.996518 Ada\nPOOL_NAME in ./env with 'TEST' (in our case, the POOL_NAME is TEST). The cnode.sh script will automatically detect whether the files required to run as a block producing node are present in the $CNODE_HOME/priv/pool/<POOL_NAME> directory.The node runs with an operational certificate, generated using the KES hot key. For security reasons, the protocol asks to re-generate (or rotate) your KES key once reaching expiry. On mainnet, this expiry is in 62 cycles of 18 hours (thus, to ask for rotation quarterly), after which your node will not be able to forge valid blocks unless rotated. To be able to rotate KES keys, your cold keys files (cold.skey, cold.vkey and cold.counter) need to be present on the machine where you run CNTools to rotate your KES key.
To Rotate KES keys and generate the operational certificate - op.cert.
From the main menu select [p] Pool
[o] Rotate The output should look like:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL >> ROTATE KES\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSelect pool to rotate KES keys on\nSelected pool: TEST\n\nPool KES keys successfully updated\nNew KES start period : 240\nKES keys will expire : 302 - 2021-09-04 11:24:31 UTC\n\nRestart your pool node for changes to take effect\n\npress any key to return to home menu\ncardano-node. If deployed as a systemd service as shown here, you can run sudo systemctl restart cnode.You can use gLiveView - the output at the top should say > Cardano Node - (Core - Guild).
Alternatively, you can check the node logs in $CNODE_HOME/logs/ to see whether the node is performing leadership checks (TraceStartLeadershipCheck, TraceNodeIsNotLeader, etc.)
Important
Koios CNTools is like a swiss army knife for pool operators to simplify typical operations regarding their wallet keys and pool management. Please note that this tool only aims to simplify usual tasks for its users, but it should NOT act as an excuse to skip understanding how to manually work through things or basics of Linux operations. The skills highlighted on the home page are paramount for a stake pool operator, and so is the understanding of configuration files and network. Please ensure you've read and understood the disclaimers before proceeding.
Visit the Changelog section to see progress and current release.
"},{"location":"Scripts/cntools/#overview","title":"Overview","text":"The tool consist of three files.
cntools.sh - the main script to launch cntools.cntools.library - internal script with helper functions.In addition to the above files, there is also a dependency on the common env file. CNTools connects to your node through the configuration in the env file located in the same directory as the script. Customize env and cntools.sh files to your needs.
Additionally, CNTools can integrate and enable optional functionalities based on external components:
cncli.sh is a companion script with optional functionalities to run on the core node (block producer) such as monitoring created blocks, calculating leader schedules and block validation.logMonitor.sh is another companion script meant to be run together with the cncli.sh script to give a more complete picture.See CNCLI and Log Monitor sections for more details.
Koios CNTools can operate in following modes:
-n - The local node is used to query the blockchain for needed data. This is the default mode when you start CNTools without parameters.-l - Koios query layer is used and removes the need for a local node deployment. This mode is both quicker and lighter on resources but comes with a third party dependency.-o - Launches CNTools with a limited set of features. This mode does not require access to cardano-node or access to an internet connection. It is mainly used to create Wallet/Pool and access Transaction >> Sign to sign an offline transaction file created in Hybrid mode.-a - Exposes a new Advanced menu, which allows users to manage (create/mint/burn) new assets.In addition to above mentioned runtime arguments to launch CNTools in different modes, it can also be persisted by editing User Variables section within cntools.sh script.
The update functionality is provided from within CNTools. In case of breaking changes, please follow the prompts post-upgrade. If stuck, it's always best to re-run the latest guild-deploy.sh before proceeding.
If you have not updated in a while, it is possible that you might come from a release with breaking changes. If so, please be sure to check out the upgrade instructions.
"},{"location":"Scripts/cntools/#navigation","title":"Navigation","text":"The scripts menu supports both arrow key navigation and shortcut key selection. The character within the square brackets is the shortcut to press for quick navigation. For other selections like wallet and pool menu that don't contain shortcuts, there is a third way to navigate. Key pressed is compared to the first character of the menu option and if there is a match the selection jumps to this location. A handy way to quickly navigate a large menu.
"},{"location":"Scripts/cntools/#hardware-wallet","title":"Hardware Wallet","text":"CNTools includes hardware wallet support since version 7.0.0 through Vacuumlabs cardano-hw-cli application. Initialize and update firmware/app on the device to the latest version before usage following the manufacturer instructions.
To enable hardware support run guild-deploy.sh -s w. This downloads and installs Vacuumlabs cardano-hw-cli including udev configuration. When a new version of Vacuumlabs cardano-hw-cli is released, run guild-deploy.sh -s w again to update. For additional runtime options, run guild-deploy.sh -h.
Trezor Bridge for your system before trying to use your Trezor device in CNTools. You can find the latest version of the bridge at https://wallet.trezor.io/#/bridgeCNTools can be run in online and offline mode. At a very high level, for working with offline devices, remember that you need to use CNTools in an online node to generate a staging transaction for the desired type of transaction, and then move the staging transaction to an offline node to sign (authorize) using the signing keys on your offline node - and then bring back the signed transaction to the online node for submission to the chain.
For the offline workflow, all the wallet and pool keys should be kept on the offline node. The backup function in CNTools has an option to create a backup without private keys (sensitive signing keys) to be transferred to online node. All other files are included in the backup to be transferred to the online node.
Keys excluded from backup when created without private keys: Wallet - payment.skey, stake.skey Pool - cold.skey
Note that setting up an offline server requires good SysOps background (you need to be aware of how to set up your server with offline mirror repository, how to transfer files across and be fairly familiar with the disk layout presented in the documentation). The guild-deploy.sh in its current state is not expected to run on an offline machine. Essentially, you simply need the cardano-cli, bech32, cardano-address binaries in your $PATH, OS level dependency packages [jq, coreutils, pkgconfig, gcc-c++ and bc ], and perhaps a copy from your online cnode directory (to ensure you have the right genesis/config files on your offline server). We strongly recommend you to familiarise yourself with the workflow on the preview / preprod / guild networks first, before attempting on mainnet.
Example workflow for creating a wallet and pool:
sequenceDiagram Note over Offline: Create/Import a wallet Note over Offline: Create a new pool Note over Offline: Rotate KES keys to generate op.cert Note over Offline: Create a backup w/o private keys Offline->>Online: Transfer backup to online node Note over Online: Fund the wallet base address with enough Ada Note over Online: Register wallet using ' Wallet \u00bb Register ' in hybrid mode Online->>Offline: Transfer built tx file back to offline node Note over Offline: Use ' Transaction >> Sign ' with payment.skey from wallet to sign transaction Offline->>Online: Transfer signed tx back to online node Note over Online: Use ' Transaction >> Submit ' to send signed transaction to blockchain Note over Online: Register pool in hybrid mode loop Offline-->Online: Repeat steps to sign and submit built pool registration transaction end Note over Online: Verify that pool was successfully registered with ' Pool \u00bb Show ' Online modeTo start CNTools in Online (advanced) Mode, execute the script from the $CNODE_HOME/scripts/ directory:
cd $CNODE_HOME/scripts\n./cntools.sh -a\nYou should get a screen that looks something like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> Koios CNTools vX.X.X - Guild - CONNECTED <<\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Main Menu Telegram Announcement / Support channel: t.me/CardanoKoios/9759\n\n ) Wallet - create, show, remove and protect wallets\n ) Funds - send, withdraw and delegate\n ) Pool - pool creation and management\n ) Transaction - Sign and Submit a cold transaction (hybrid/offline mode)\n ) Blocks - show core node leader schedule & block production statistics\n ) Backup - backup & restore of wallet/pool/config\n ) Advanced - Developer and advanced features: metadata, multi-assets, ...\n ) Refresh - reload home screen content\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Epoch 276 - 3d 19:08:27 until next\n What would you like to do? Node Sync: 12 :)\n\n [w] Wallet\n [f] Funds\n [p] Pool\n [t] Transaction\n [b] Blocks\n [u] Update\n [z] Backup & Restore\n [a] Advanced\n [r] Refresh\n [q] Quit\nTo start CNTools in Offline Mode, execute the script from the $CNODE_HOME/scripts/ directory using the -o flag:
cd $CNODE_HOME/scripts\n./cntools.sh -o\nThe main menu header should let you know that node is started in offline mode:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> Koios CNTools vX.X.X - Guild - OFFLINE <<\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Main Menu Telegram Announcement / Support channel: t.me/CardanoKoios/9759\n\n ) Wallet - create, show, remove and protect wallets\n ) Funds - send, withdraw and delegate\n ) Pool - pool creation and management\n ) Transaction - Sign and Submit a cold transaction (hybrid/offline mode)\n\n ) Backup - backup & restore of wallet/pool/config\n\n ) Refresh - reload home screen content\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Epoch 276 - 3d 19:03:46 until next\n What would you like to do?\n\n [w] Wallet\n [f] Funds\n [p] Pool\n [t] Transaction\n [z] Backup & Restore\n [r] Refresh\n [q] Quit\nA common environment file called env is sourced by most scripts in the Guild Operators repository. This file holds common variables and functions needed by other scripts. There are several benefits to this, not having to specify duplicate settings and being able to reuse functions decreasing the risk of misconfiguration and inconsistency.
env file is downloaded together with the rest of the scripts when Pre-Requisites if followed and located in the $CNODE_HOME/scripts/ directory. The file is also automatically downloaded/updated by some of the individual scripts if missing, like cntools.sh, gLiveView.sh and topologyUpdater.sh. All custom changes in User Variables section are untouched on updates unless a forced overwrite is selected when running guild-deploy.sh.
Most variables can be left commented to use the automatically detected or default value. But there are some that need to be set as explained below.
CNODE_PORT - This is the most important variable and needs to be set. Used when launching the node through cnode.sh and to identify the correct process of the node.CNODE_HOME - The root directory of the Cardano node holding all the files needed. Can be left commented if guild-deploy.sh has been run as this variable is then exported and added as a system environment variable.POOL_NAME - If the node is to be started as a block producer by cnode.sh this variable needs to be uncommented and set. This is the name given to the pool in CNTools (not ticker), i.e. the pool directory name under $CNODE_HOME/priv/pool/<POOL_NAME>Take your time and look through the different variables and their explanations and decide if you need/want to change the default setting. For a default deployment using guild-deploy.sh, the CNODE_PORT (all installs) and POOL_NAME (only block producer) should be the only variables needed to be set.
######################################\n# User Variables - Change as desired #\n# Leave as is if unsure #\n######################################\n\n#CCLI=\"${HOME}/.local/bin/cardano-cli\" # Override automatic detection of path to cardano-cli executable\n#CNCLI=\"${HOME}/.local/bin/cncli\" # Override automatic detection of path to cncli executable (https://github.com/AndrewWestberg/cncli)\n#CNODE_HOME=\"/opt/cardano/cnode\" # Override default CNODE_HOME path (defaults to /opt/cardano/cnode)\nCNODE_PORT=6000 # Set node port\n#CONFIG=\"${CNODE_HOME}/files/config.json\" # Override automatic detection of node config path\n#SOCKET=\"${CNODE_HOME}/sockets/node.socket\" # Override automatic detection of path to socket\n#TOPOLOGY=\"${CNODE_HOME}/files/topology.json\" # Override default topology.json path\n#LOG_DIR=\"${CNODE_HOME}/logs\" # Folder where your logs will be sent to (must pre-exist)\n#DB_DIR=\"${CNODE_HOME}/db\" # Folder to store the cardano-node blockchain db\n#UPDATE_CHECK=\"Y\" # Check for updates to scripts, it will still be prompted before proceeding (Y|N).\n#TMP_DIR=\"/tmp/cnode\" # Folder to hold temporary files in the various scripts, each script might create additional subfolders\n#EKG_HOST=127.0.0.1 # Set node EKG host IP\n#EKG_PORT=12788 # Override automatic detection of node EKG port\n#PROM_HOST=127.0.0.1 # Set node Prometheus host IP\n#PROM_PORT=12798 # Override automatic detection of node Prometheus port\n#EKG_TIMEOUT=3 # Maximum time in seconds that you allow EKG request to take before aborting (node metrics)\n#CURL_TIMEOUT=10 # Maximum time in seconds that you allow curl file download to take before aborting (GitHub update process)\n#BLOCKLOG_DIR=\"${CNODE_HOME}/guild-db/blocklog\" # Override default directory used to store block data for core node\n#BLOCKLOG_TZ=\"UTC\" # TimeZone to use when displaying blocklog - https://en.wikipedia.org/wiki/List_of_tz_database_time_zones\n#SHELLEY_TRANS_EPOCH=208 # Override automatic detection of shelley epoch start, e.g 208 for mainnet\n#TG_BOT_TOKEN=\"\" # Uncomment and set to enable telegramSend function. To create your own BOT-token and Chat-Id follow guide at:\n#TG_CHAT_ID=\"\" # https://cardano-community.github.io/guild-operators/Scripts/sendalerts\n#USE_EKG=\"N\" # Use EKG metrics from the node instead of Promethus. Promethus metrics(default) should yield slightly better performance\n#TIMEOUT_LEDGER_STATE=300 # Timeout in seconds for querying and dumping ledger-state\n#IP_VERSION=4 # The IP version to use for push and fetch, valid options: 4 | 6 | mix (Default: 4)\n\n#WALLET_FOLDER=\"${CNODE_HOME}/priv/wallet\" # Root folder for Wallets\n#POOL_FOLDER=\"${CNODE_HOME}/priv/pool\" # Root folder for Pools\n# Each wallet and pool has a friendly name and subfolder containing all related keys, certificates, ...\n#POOL_NAME=\"\" # Set the pool's name to run node as a core node (the name, NOT the ticker, ie folder name)\n\n#WALLET_PAY_VK_FILENAME=\"payment.vkey\" # Standardized names for all wallet related files\n#WALLET_PAY_SK_FILENAME=\"payment.skey\"\n#WALLET_HW_PAY_SK_FILENAME=\"payment.hwsfile\"\n#WALLET_PAY_ADDR_FILENAME=\"payment.addr\"\n#WALLET_BASE_ADDR_FILENAME=\"base.addr\"\n#WALLET_STAKE_VK_FILENAME=\"stake.vkey\"\n#WALLET_STAKE_SK_FILENAME=\"stake.skey\"\n#WALLET_HW_STAKE_SK_FILENAME=\"stake.hwsfile\"\n#WALLET_STAKE_ADDR_FILENAME=\"reward.addr\"\n#WALLET_STAKE_CERT_FILENAME=\"stake.cert\"\n#WALLET_STAKE_DEREG_FILENAME=\"stake.dereg\"\n#WALLET_DELEGCERT_FILENAME=\"delegation.cert\"\n\n#POOL_ID_FILENAME=\"pool.id\" # Standardized names for all pool related files\n#POOL_HOTKEY_VK_FILENAME=\"hot.vkey\"\n#POOL_HOTKEY_SK_FILENAME=\"hot.skey\"\n#POOL_COLDKEY_VK_FILENAME=\"cold.vkey\"\n#POOL_COLDKEY_SK_FILENAME=\"cold.skey\"\n#POOL_OPCERT_COUNTER_FILENAME=\"cold.counter\"\n#POOL_OPCERT_FILENAME=\"op.cert\"\n#POOL_VRF_VK_FILENAME=\"vrf.vkey\"\n#POOL_VRF_SK_FILENAME=\"vrf.skey\"\n#POOL_CONFIG_FILENAME=\"pool.config\"\n#POOL_REGCERT_FILENAME=\"pool.cert\"\n#POOL_CURRENT_KES_START=\"kes.start\"\n#POOL_DEREGCERT_FILENAME=\"pool.dereg\"\n\n#ASSET_FOLDER=\"${CNODE_HOME}/priv/asset\" # Root folder for Multi-Assets containing minted assets and subfolders for Policy IDs\n#ASSET_POLICY_VK_FILENAME=\"policy.vkey\" # Standardized names for all multi-asset related files\n#ASSET_POLICY_SK_FILENAME=\"policy.skey\"\n#ASSET_POLICY_SCRIPT_FILENAME=\"policy.script\" # File extension '.script' mandatory\n#ASSET_POLICY_ID_FILENAME=\"policy.id\"\nReminder !!
Ensure the Pre-Requisites are in place before you proceed.
Koios gLiveView is a local monitoring tool to use in addition to remote monitoring tools like Prometheus/Grafana, Zabbix or IOG's RTView. This is especially useful when moving to a systemd deployment - if you haven't done so already - as it offers an intuitive UI to monitor the node status.
"},{"location":"Scripts/gliveview/#configuration-startup","title":"Configuration & Startup","text":"For most setups, it's enough to set CNODE_PORT in the env file. The rest of the variables should automatically be detected. If required, modify User Variables in env and gLiveView.sh to suit your environment (if the environment is customised). This should lead you to a stage where you can now start running ./gLiveView.sh in the folder you downloaded the script (the default location would be $CNODE_HOME/scripts). Note that the script is smart enough to automatically detect when you're running as a Core or Relay and will show fields accordingly.
The tool can be run in legacy mode with only standard ASCII characters for terminals with trouble displaying the box-drawing characters. Run ./gLiveView.sh -h to show available command-line parameters or permanently set it directly in script.
Note !!
Keeping gLiveView to it's intent of being a dashboard and not a full-fledged monitoring tool, we intend to keep most relevant information for a node operator in a minimalistic dashboard, accordingly - gLiveView runs by default in compact mode. One can enable verbose mode by pressing 'v' to unhide additional fields.
A sample output from both core and relay together with peer analysis:
Core Relay Peer Analysis "},{"location":"Scripts/gliveview/#upper-main-section","title":"Upper main section","text":"Displays live metrics from cardano-node gathered through the nodes EKG/Prometheus(env setting) endpoint.
activeSlotsCoeff). A slot on MainNet happens every 1 second(slotLength), thus the max chain density can be calculated as slotLength/activeSlotsCoeff = 5%. Normally, the value should fluctuate around this value. starting|sync xx.x% or if close to reference tip, the tip difference Tip (ref) - Tip (node) to see how far of the tip (diff value) the node is. With current parameters a slot diff up to 40 from reference tip is considered good but it should usually stay below 30. It's perfectly normal to see big differences in slots between blocks. It's the built in randomness at play. To see if a node is really healthy and staying on tip you would need to compare the tip between multiple nodes. Cold peers indicate the number of inactive but known peers to the node.Warm peers tell how many established connections the node has.Hot peers how many established connections are actually active.Bi-Dir(bidirectional) and Uni-Dir(unidirectional) indicate how the handshake protocol negotiated the connection. The connection between p2p nodes will always be bidirectional, but it will be unidirectional between p2p nodes and non-p2p nodes. Duplex shows the connections that are actually used in both directions, only bidirectional connections have this potential.If the node is run as a core, identified by the 'forge-about-to-lead' parameter, a second core section is displayed.
Missed slot checks - A value that show if the node have missed slots for attempting leadership checks (as absolute value and percentage since node startup). !!! info \"Missed Slot Leadership Check\"
Note that while this counter should ideally be close to zero, you would often see a higher value if the node is busy (e.g. paused for garbage collection or busy with reward calculations). A consistently high percentage of missed slots would need further investigation (assistance for troubleshooting can be seeked here ), as in extremely remote cases - it can overlap with a slot that your node could be a leader for.
Blocks - If CNCLI is activated to store blocks created in a blocklog DB, data from this blocklog is displayed. See linked CNCLI documentation for details regarding the different block metrics. If CNCLI is not deployed, block metrics displayed are taken from node metrics and show blocks created by the node since node start.
A manual peer analysis can be triggered by key press p. A latency test will be done on incoming and outgoing connections to the node.
Note
Note that with P2P enabled, an incoming/outgoing connection can be reused for bi-directional traffic. There isnt a way to distinctly identify the P2P peer's direction yet for a given IP.
Outgoing connections(peers in topology file), ping type used is done in this order: 1. cncli - If available, this gives the most accurate measure as it checks the entire handshake process against the remote peer. 2. ss - Sends a TCP SYN package to ping the remote peer on the cardano-node port. Should give ~100% success rate. 2. tcptraceroute - Same as ss. 3. ping - fallback method using ICMP ping against IP. Will only work if firewall of remote peer accept ICMP traffic.
For incoming connections, only ICMP ping is used as remote peer port is unknown. It's not uncommon to see many undetermined peers for incoming connections as it's a good security practice to disable ICMP in firewall.
Once the analysis is finished, it will display the RTTs (return-trip times) for the peers and group them in ranges 0-50, 50-100, 100-200, 200<. The analysis is NOT live. Press [h] Home to go back to default view or [i] Info to show in-script help text. Up and Down arrow keys is used to select incoming or outgoing detailed list of IPs and their RTT value. Left (<) and Right (>) arrow keys can be used to navigate the pages in the selected list.
In case you run into trouble while running the script, you might want to edit env & gLiveView.sh and look at User Variables section. You can override the values if the automatic detection do not provide the right information, but we would appreciate if you could also notify us by raising an issue against the GitHub repository:
gLiveView.sh
######################################\n# User Variables - Change as desired #\n######################################\n\nNODE_NAME=\"Cardano Node\" # Change your node's name prefix here, keep at or below 19 characters!\nREFRESH_RATE=2 # How often (in seconds) to refresh the view (additional time for processing and output may slow it down)\nLEGACY_MODE=false # (true|false) If enabled unicode box-drawing characters will be replaced by standard ASCII characters\nRETRIES=3 # How many attempts to connect to running Cardano node before erroring out and quitting\nPEER_LIST_CNT=6 # Number of peers to show on each in/out page in peer analysis view\nTHEME=\"dark\" # dark = suited for terminals with a dark background\n# light = suited for terminals with a bright background\nENABLE_IP_GEOLOCATION=\"Y\" # Enable IP geolocation on outgoing and incoming connections using ip-api.com\nTo claim rewards earned during the Incentivized TestNet the private and public keys from ITN must be converted to Shelley stake keys. A script called itnRewards.sh has been created to guide you through the process of converting the keys and to create a CNTools compatible wallet from were the rewards can be withdrawn.
jcli account in ITN was ed25519_sk (not extended), you can run the itnRewards.sh script providing the name for the CNTools wallet and ITN owner public/secret keys that were used to register your pool as below. cd $CNODE_HOME/scripts\n./itnRewards.sh MyITNWallet ~/jormu/account/priv/owner.sk ~/jormu/account/priv/owner.pk\nFUNDS >> WITHDRAW to move rewards to the base address of walletDisclaimer
Currently this is to protect the existing pools from the ITN who already have a delegator base against spoofing - to avoid scammers building on results of ITN from known pools. There would be a solution in the future for Mainnet nodes too - but it doesn't apply to those in its current form.
"},{"location":"Scripts/itnwitness/#concept","title":"Concept","text":"Due to the expected ticker spoofing attack for pools that were famous during ITN, some of the community members have proposed an interim solution to verify the legitimacy of a pool for delegators. You can check the high-level workflow below:
graph TB A(\"ITN Owner skey (ed25519/ed25519e) ..\") --x C([\"jcli key sign ..\"]) B(\"Haskell Pool ID (pool.id) ..\") --x C C --x D(\"Signature key, (pool.sig) ..\") E(\"ITN Owner vkey (ed25519_pk) ..\") --x F(\"Extended Metadata JSON (poolmeta_extended.json) ..\") D --x F F --x G(\"Pool Meta JSON (poolmeta.json) ..\") ;"},{"location":"Scripts/itnwitness/#steps","title":"Steps","text":"The actual implementation is pretty straightforward, we will keep it brisk - as we assume ones participating are fairly familiar with jcli usage.
mainnet_pool.id)owner_skey) as per below: jcli key sign --secret-key ~/jormu/account/priv/owner.sk $CNODE_HOME/priv/pool/TEST/pool.id --output mainnet_pool.sig\ncat mainnet_pool.sig\n# ed25519_sig1sn32v3z...d72rg7rc6gs\n{\n\"itn\": {\n\"owner\": \"ed25519_pk1...\",\n\"witness\": \"ed25519_sig1...\"\n}\n}\nIf the process is approved to appear for wallets, we may consider providing easier alternatives. If any queries about the process, or any additions please create a git issue/PR against guild repository - to capture common queries and update instructions/help text where appropriate.
"},{"location":"Scripts/itnwitness/#sample-output-of-json-files-generated","title":"Sample output of JSON files generated","text":"Metadata JSON used for registering pool (one that will be hosted URL used to define pool, eg: https://hosting.site/poolmeta.json)
{\n\"name\":\"Test\",\n\"ticker\":\"TEST\",\n\"description\":\"For demo purposes only\",\n\"homepage\":\"https://hosting.site\",\n\"nonce\":\"1595816423\",\n\"extended\":\"https://hosting.site/poolmeta_extended.json\"\n}\nExtended Metadata JSON used for hosting additional metadata (hosted at URL referred in extended field above, thus - eg : https://hosting.site/poolmeta_extended.json)
{\n\"itn\": {\n\"owner\": \"ed25519_pk1...\",\n\"witness\": \"ed25519_sig1...\"\n}\n}\nReminder !!
Ensure the Pre-Requisites are in place before you proceed.
logMonitor.sh is a general purpose JSON log monitoring script for traces created by cardano-node. Currently, it looks for traces related to leader slots and block creation but other uses could be added in the future.
For the core node (block producer) the logMonitor.sh script can be run to monitor the JSON log file created by cardano-node for traces related to leader slots and block creation.
For optimal coverage, it's best run together with CNCLI scripts as they provide different functionalities. Together, they create a complete picture of blocks assigned, created, validated or invalidated due to node issues.
"},{"location":"Scripts/logmonitor/#installation","title":"Installation","text":"The script is best run as a background process. This can be accomplished in many ways but the preferred method is to run it as a systemd service. A terminal multiplexer like tmux or screen could also be used but not covered here.
Use the deploy-as-systemd.sh script to create a systemd unit file (deployed together with CNCLI). Log output is handled by journald. journalctl -f -u cnode-logmonitor.service can be used to check service output (follow mode). Other logging configurations are not covered here.
Best viewed in CNTools or gLiveView. See CNCLI for example output.
"},{"location":"Scripts/mithril-client/","title":"Client","text":"mithril-client.sh is a script to manage the Mithril client, a tool used to set up the Mithril client environment and manage downloading Mithril snapshots and stake distributions. The main features include:
mithril.env file with all the necessary environment variables for the Mithril client.Usage: bash [-u] <command> <subcommand> [<sub arg>]\nA script to run Cardano Mithril Client\n\n-u Skip script update check overriding UPDATE_CHECK value in env (must be first argument to script)\n\nCommands:\nenvironment Manage mithril environment file\n setup Setup mithril environment file\n override Override default variable in the mithril environment file\n update Update mithril environment file\ncardano-db Interact with Cardano DB\n download Download Cardano DB from Mithril snapshot\n snapshot Interact with Mithril snapshots\n list List available Mithril snapshots\n json List availble Mithril snapshots in JSON format\n show Show details of a Mithril snapshot\n json Show details of a Mithril snapshot in JSON format\nstake-distribution Interact with Mithril stake distributions\n download Download latest stake distribution\n list List available stake distributions\n json Output latest Mithril snapshot in JSON format\nTo prepare a relay or block producer node, you should follow these steps:
environment setup command. This will create a new mithril.env file with all the necessary environment variables for the Mithril client../mithril-client.sh environment setup\nsnapshot download command. This snapshot contains the latest state of the Cardano blockchain db from a Mithril Aggregator../mithril-client.sh cardano-db download\nYou can investigate the available snapshots by using the snapshot list and snapshot show commands:
snapshot list command. Add json at the end to get the output in JSON format../mithril-client.sh cardano-dbsnapshot list\n./mithril-client.sh cardano-dbsnapshot list json\nsnapshot show <DIGEST> command, where <DIGEST> is the digest of the snapshot. Add json at the end to get the output in JSON format../mithril-client.sh cardano-dbsnapshot show <DIGEST>\n./mithril-client.sh cardano-dbsnapshot show <DIGEST> json\n./mithril-client.sh cardano-dbsnapshot show json <DIGEST>\nYou can manage stake distributions by using the stake-distribution download and stake-distribution list commands:
stake-distribution download command../mithril-client.sh stake-distribution download\nstake-distribution list command. Add json at the end to get the output in JSON format../mithril-client.sh stake-distribution list\n./mithril-client.sh stake-distribution list json\nmithril-relay.sh is a bash script for deployment of Squid Mithril Relays and a Nginx loadbalancer. It provides functionalities such as:
bash [-d] [-l] [-u] [-h]\nA script to setup Cardano Mithril relays\n\n-d Install squid and configure as a relay\n-l Install nginx and configure as a load balancer\n-u Skip update check\n-h Show this help text\nThe mithril-relay.sh script is a bash script for managing the Mithril Relay Server. It provides functionalities such as installing and configuring Squid as a relay, installing and configuring Nginx as a load balancer.
The script uses the following environment variable:
RELAY_LISTENING_PORT: The port on which the relay server listens.The script parses command line options and performs the corresponding actions based on the options provided. If the -d option is provided, it installs Squid and configures it as a relay. If the -l option is provided, it installs Nginx and configures it as a load balancer. If no options are provided, it displays the usage message.
mithril-signer.sh is a bash script for managing the Mithril Signer Server. It provides functionalities such as deploying the server as a systemd service and updating the environment file to contain variables specific to the Mithril Signer.
Usage: bash [-d] [-D] [-e] [-k] [-r] [-s] [-u] [-h]\nA script to setup, run and verify Cardano Mithril Signer\n\n-d Deploy mithril-signer as a systemd service\n-D Run mithril-signer as a daemon\n-e Update mithril environment file\n-k Stop signer using SIGINT\n-r Verify signer registration\n-s Verify signer signature\n-u Skip update check\n-h Show this help text\nThis script is a bash script for managing the Mithril Signer Server. It provides functionalities such as deploying the server as a systemd service, updating the environment file, and running the server.
"},{"location":"Scripts/mithril-signer/#environment-variables","title":"Environment Variables","text":"The script uses several environment variables, some of which are:
MITHRILBIN: Path for mithril-signer binary, if not in $PATH.HOSTADDR: Default Listen IP/Hostname for Mithril Signer Server.POOL_NAME: The name of the pool.NETWORK_NAME: The name of the network.The script parses command line options, sources the environment file, sets default values, and performs basic sanity checks. It then checks if the -d or -u options were specified and performs the corresponding actions. If no options were specified, it runs the Mithril Signer Server.
!> Ensure the Pre-Requisites are in place before you proceed.
This section describes the ways in which CNTools can send important messages to the operator.
"},{"location":"Scripts/sendalerts/#telegram-alerts","title":"Telegram alerts","text":"If known but unwanted errors occur on your node, or if characteristic values indicate an unusual status , CNTools can send you Telegram alert messages.
To do this, you first have to activate your own bot and link it to your own Telegram user. Here is an explanation of how this works:
Open Telegram and search for \"botfather\".
Write him your wish: /newbot.
Define a name for your bot, such as cntools_[POOLNAME]_alerts.
Botfather will confirm the creation of your bot by giving you the unique bot access token. Keep it safe and private.
Now send at least one direct message to your new bot.
Open this URL in your browser by using your own, just created bot access token:
https://api.telegram.org/bot<your-access-token>/getUpdates\nresult.message.chat.id. This chat id should be a large integer number.This is all you need to enable your Telegram alerts in the scripts/env file - uncomment and add the chat ID to the TG_CHAT_ID user variable in the env file:
...\nTG_CHAT_ID=\"<YOUR_TG_CHAT_ID>\"\n... \nReminder !!
The topologyUpdater shell script must be executed on the relay node as a cronjob exactly every 60 minutes. After 4 consecutive requests (3 hours) the node is considered a new relay node in listed in the topology file. If the node is turned off, it's automatically delisted after 3 hours.
"},{"location":"Scripts/topologyupdater/#download","title":"Download and Configure","text":"If you have run guild-deploy.sh, this should already be available in your scripts folder and make this step unnecessary.
Before the updater can make a valid request to the central topology service, it must query the current tip/blockNo from the well-synced local node. It connects to your node through the configuration in the script as well as the common env configuration file. Customize these files for your needs.
To download topologyUpdater.sh manually, you can execute the commands below and test executing Topology Updater once (it's OK if first execution gives back an error):
cd $CNODE_HOME/scripts\ncurl -s -o topologyUpdater.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/topologyUpdater.sh\ncurl -s -o env https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/env\nchmod 750 topologyUpdater.sh\n./topologyUpdater.sh\nOut of the box, the scripts might come with some assumptions, that may or may not be valid for your environment. One of the common changes as an SPO would be to the complete CUSTOM_PEERS section as below to include your local relays/BP nodes (described in the How do I add my own nodes section), and any additional peers you'd like to be always available at minimum. Please do take time to update the variables in User Variables section in env & topologyUpdater.sh:
### topologyUpdater.sh\n\n######################################\n# User Variables - Change as desired #\n######################################\n\nCNODE_HOSTNAME=\"CHANGE ME\" # (Optional) Must resolve to the IP you are requesting from\nCNODE_VALENCY=1 # (Optional) for multi-IP hostnames\nMAX_PEERS=15 # Maximum number of peers to return on successful fetch\n#CUSTOM_PEERS=\"None\" # Additional custom peers to (IP,port[,valency]) to add to your target topology.json\n# eg: \"10.0.0.1,3001|10.0.0.2,3002|relays.mydomain.com,3003,3\"\n#BATCH_AUTO_UPDATE=N # Set to Y to automatically update the script if a new version is available without user interaction\nAny customisations you add above, will be saved across future guild-deploy.sh executions, unless you specify the -f flag to overwrite completely.
systemd service The script can be deployed as a background service in different ways but the recommended and easiest way if guild-deploy.sh was used, is to utilize the deploy-as-systemd.sh script to setup and schedule the execution. This will deploy both push & fetch service files as well as timers for a scheduled 60 min node alive message and cnode restart at the user set interval (default: 24 hours) when running the deploy script.
cnode-tu-push.service : pushes a node alive message to Topology Updater APIcnode-tu-push.timer : schedules the push service to execute once every hourcnode-tu-fetch.service : fetches a fresh topology file before the cnode.service file is started/restartedcnode-tu-restart.service : handles the restart of cardano-node (cnode.sh)cnode-tu-restart.timer : schedules the cardano-node restart service, default every 24hsystemctl list-timers can be used to to check the push and restart service schedule.
crontab job Another way to deploy the topologyUpdater.sh script is as a crontab job. Add the script to be executed once per hour at a minute of your choice (eg xx:25 o'clock in the example below). The example below will handle both the fetch and push in a single call to the script once an hour. In addition to the below crontab job for topologyUpdater, it's expected that you also add a scheduled restart of the relay node to pick up a fresh topology file fetched by topologyUpdater script with relays that are alive and well.
25 * * * * /opt/cardano/cnode/scripts/topologyUpdater.sh\nYou can check the last result of push message in logs/topologyUpdater_lastresult.json. If deployed as systemd service, use sudo journalctl -u <service> to check output from service.
If one of the parameters is outside the allowed ranges, invalid or missing the returned JSON will tell you what needs to be fixed.
Don't try to execute the script more often than once per hour. It's completely useless and may lead to a temporary blacklisting.
"},{"location":"Scripts/topologyupdater/#why-does-my-topology-file-only-contain-iog-peers","title":"Why does my topology file only contain IOG peers?","text":"Each subscribed node (4 consecutive requests) is allowed to fetch a subset of other nodes to prove loyalty/stability of the relay. Until reaching this point, your fetch calls will only return IOG peers combined with any custom peers added in USER VARIABLES section of topologyUpdater.sh script
The engineers of cardano-node network stack suggested to use around 20 peers. More peers create unnecessary and unwanted system load and delays.
In its default setting, topologyUpdater returns a list of 15 remote peers.
Note that the change in topology is only effective upon restart of your node. Make sure you account for some scheduled restarts on your relays, to help onboard newer relays onto the network (as described in the systemd section).
"},{"location":"Scripts/topologyupdater/#how-do-i-add-my-own-relaysstatic-nodes-in-addition-to-dynamic-list-generated-by-topologyupdater","title":"How do I add my own relays/static nodes in addition to dynamic list generated by topologyUpdater?","text":"Most of the Stake Pool Operators may have few preferences (own relays, close friends, etc) that they would like to add to their topology by default. This is where the CUSTOM_PEERS variable in topologyUpdater.sh comes in. You can add a list of peers in the format of: hostname/IP:port[:valency] here and the output topology.json formed will already include the custom peers that you supplied. Every custom peer is defined in the form [address]:[port] and optional :[valency] (if not specified, the valency defaults to 1). Multiple custom peers are separated by |. An example of a valid CUSTOM_PEERS variable would be:
CUSTOM_PEERS=\"foo.bar.io,3001,2|198.175.21.197,6001|36.233.3.89,6000\n2)."},{"location":"Scripts/topologyupdater/#how-are-the-peers-for-my-topology-file-selected","title":"How are the peers for my topology file selected?","text":"We calculate the distance on the Earth's surface from your node's IP to all subscribed peers. We then order the peers by distance (closest first) and start by selecting one peer. We then skip some, pick the next, skip, pick, skip, pick ... until we reach the end of the list (furthest away). The number of skipped records is calculated in a way to have the desired number of peers at the end.
Every requesting node has its personal distance to all other nodes.
We assume this should result in a well-distributed and interconnected peering network.
"},{"location":"docker/build/","title":"Build","text":""},{"location":"docker/build/#intro","title":"Intro","text":"\ud83d\udca1 Docker containers are the fastest way to run a Cardano node in both \"Relay\" and \"Block-Producing\" (Pool) mode.
"},{"location":"docker/build/#how-to-build","title":"How to build","text":"docker build -t cardanocommunity/cardano-node:latest - < dockerfile_bin\nWith Powershell on Windows, you can run docker by typing the following command:
Get-Content dockerfile_bin | docker build -t guild-operators/cardano-node:latest -\nDocker Tips
Docker Official Docs
"},{"location":"docker/docker/","title":"Overview","text":"Running your own Cardano node has never been so fast and easy.
But first, a kind reminder to the security aspects of running docker containers.
"},{"location":"docker/docker/#external-resources","title":"External resources","text":"Modular docker images based on Debian.
Based on the Guild's work the Cardano Node image is built in a single stage: -> dockerfile_bin
guild-deploy.sh to:If you prefer to build the images your own than you can check:
The dockerfiles are located in ./files/docker/
Node Ports Wallet Ports Flavor Node (6000) Wallet (8090) Debian Prometheus (12798) Prometheus (12798) EKG (12781)"},{"location":"docker/run/","title":"Run","text":""},{"location":"docker/run/#os-requirements","title":"OS Requirements","text":"docker-ce installed - Get Docker.Note
1) --entrypoint=bash # This option won't start the node's container but only the OS running (the node software wont actually start, you'll need to manually execute entrypoint.sh ), ready to get in (trough the command docker exec -it < container name or hash > /bin/bash) and play/explore around with it in command line mode. 2) all guild tools env variable can be used to start a new container using custom values by using the \"-e\" option. 3) CPU and RAM and Shared Memory allocation option for the container can be used when you start the container (i.e. --shm-size or --memory or --cpus official docker resource docs) 4) --env MITHRIL_DOWNLOAD=Y # This option will allow Mithril client to download the latest Mithril snapshot of the blockchain when the container starts and does not have a copy of the blockchain yet. This is useful when you want to start a new node from scratch and don't want to wait for the node to sync from the network. This option is not currently available for the guild network. 5) --env ENTRYPOINT_PROCESS=mithril-signer.sh # This option will allow the container to start the Mithril signer process instead of the node process. This is useful when you want to run a Mithril signer node and have the container setup the configuration files based on the NETWORK environment varaible.
docker run --init -dit\n--name <YourCName>\n--security-opt=no-new-privileges\n-e NETWORK=mainnet\n-v <your_custom_path>:/opt/cardano/cnode/priv\n-v <your_custom_db_path>:/opt/cardano/cnode/db\ncardanocommunity/cardano-node\ndocker run --init -dit\n--name <YourCName>\n--security-opt=no-new-privileges\n-e NETWORK=mainnet\n-p 6000:6000\n-v <your_custom_path>:/opt/cardano/cnode/priv\n-v <your_custom_db_path>:/opt/cardano/cnode/db\ncardanocommunity/cardano-node\ndocker run --init -dit\n--name <YourCName>\n--security-opt=no-new-privileges\n-e NETWORK=mainnet\n-e CONFIG=/opt/cardano/cnode/priv/<your own configuration files>.yml\n-p 6000:6000\n-v <your_custom_path>:/opt/cardano/cnode/priv\n-v <your_custom_db_path>:/opt/cardano/cnode/db\ncardanocommunity/cardano-node\nOn the security front, Docker developers are faced with different types of security attacks such as:
Docker containers are now being exploited to covertly mine for cryptocurrency, marking a shift from ransomware to cryptocurrency malware. As with all things in security, also Docker security is a moving target \u2014 so it\u2019s helpful to have access to up-to-date information, including experience-based best practices, for securing your containerized environments.
"},{"location":"docker/security/#here-below-some-key-concepts","title":"Here below some key concepts:","text":"Use a Third-Party Security Tool Docker allows you to use containers from untrusted public repositories, which increases the need to scrutinize whether the container was created securely and whether it is free of any corrupt or malicious files. For this, use a multi-purpose security tool that gives extensive dev-to-production security controls.(keep reading below)
Manage Vulnerability It is best to have a sound vulnerability management program that has multiple checks throughout the container lifecycle. Vulnerability management should incorporate quality gates to detect access issues and weaknesses for a potential exploit from dev-to-production environments.
Monitor and Audit Container Activity It is vital to monitor the container ecosystem and detect suspicious activity. Container monitoring activities provide real-time reports that can help you react promptly to a security breach.
Enable Docker Content Trust Docker Content Trustis a new feature incorporated into Docker 1.8. It is disabled by default, but once enabled, allows you to verify the integrity, authenticity, and publication date of all Docker images from the Docker Hub Registry.
Use Docker Bench for Security You should consider Docker Bench for Security as your must-use script. Once the script is run, you will notice a lot of information regarding configuration best practices for deploying Docker containers that can be used to further secure your Docker server and containers.
Resource Utilization To reduce performance impacts and denial-of-service attacks, it is a good practice to implement limits on the system resources that the containers can consume. If, for example, a web server is compromised, it helps to limit the impact to the other processes that are running on a host.
RBAC RBAC is role-based access control. If you have multiple users accessing you enviroment, this is a must-have. It can be quite expensive to implement but portainer makes it super easy.
Guild tips:
NEVER NEVER NEVER expose Docker API publicly!!! (disabled by default)
Keep Docker Host Up-to-date
Reverse ProxyDocker Socket OwnershipRun Docker Containers as RootUse Trusted Docker ImagesUse Privileged Mode Carefully (This is usually done by adding --privileged you can use --security-opt=no-new-privileges instead)Some more general tips:
\"--cap-drop ALL\"DOCKER_OPTS= \"--iptables=false\"With this quick guide you will be able to run a cardano node in seconds and also have the powerfull Koios SPO scripts built-in.
"},{"location":"docker/tips/#how-to-operate-interactively-within-the-container","title":"How to operate interactively within the container","text":"Once executed the container as a deamon with attached tty you are then able to enter the container by using the flag -dit .
While if you have a hook within the container console, use the following command (change CN with your container name):
docker exec -it CN bash This command will bring you within the container bash env ready to use the Koios tools.
"},{"location":"docker/tips/#docker-flags-explained","title":"Docker flags explained","text":"\"docker build\" options explained:\n -t : option is to \"tag\" the image you can name the image as you prefer as long as you maintain the references between dockerfiles.\n\n\"docker run\" options explained:\n -d : for detach the container\n -i : interactive enabled -t : terminal session enabled\n -e : set an Env Variable\n -p : set exposed ports (by default if not specified the ports will be reachable only internally)\n--hostname : Container's hostname\n --name : Container's name\ndocker run --init -itd \n-name Relay # Optional (recommended for quick access): set a name for your newly created container.\n-p 9000:6000 # Optional: to expose the internal container's port (6000) to the host <IP> port 9000\n-e NETWORK=mainnet # Mandatory: mainnet / preprod / guild-mainnet / guild\n--security-opt=no-new-privileges # Option to prevent privilege escalations\n-v <YourNetPath>:/opt/cardano/cnode/sockets # Optional: useful to share the node socket with other containers\n-v <YourCfgPath>:/opt/cardano/cnode/priv # Optional: if used has to contain all the sensitive keys needed to run a node as core\n-v <YourDBbk>:/opt/cardano/cnode/db # Optional: if not set a fresh DB will be downloaded from scratch\ncardanocommunity/cardano-node:latest # Mandatory: image to run\nNote
To be able to use the CNTools encryption key feature you need to manually change in \"cntools.config\" ENABLE_CHATTR to \"true\" and not use the --security-opt=no-new-privileges docker run option.
The docker container has an optional backup and restore functionality that can be used to backup the /opt/cardano/cnode/db directory. To have the backup persist longer than the countainer, the backup directory should be mounted as a volume.
[!NOTE] The backup and restore functionality is disabled by default.
[!WARNING] Make sure adequate space exists on the host as the backup will double the space consumed by the database.
"},{"location":"docker/tips/#creating-a-backup","title":"Creating a Backup","text":"When the container is started with the ENABLE_BACKUP environment variable set to Y the container will automatically create a backup in the /opt/cardano/cnode/backup/$NETWORK-db directory. The backup will be created when the container is started and if the backup directory is smaller than the db directory.
When the container is started with the ENABLE_RESTORE environment variable set to Y the container will automatically restore the latest backup from the /opt/cardano/cnode/backup/$NETWORK-db directory. The database will be restored when the container is started and if the backup directory is larger than the db directory.
The container now includes a static copy of each network's configuration files (Mainnet, Preprod, Preview, Sanchonet, and Guild networks). The NETWORK environment variable passed into the container determines which configuration files are copied into $CNODE_HOME/files.
The UPDATE_CHECK environment variable controls whether the container updates these configuration files from GitHub before starting. By default, the container has the environment variable set to UPDATE_CHECK=N, meaning the container uses the configuration files it was built with. This can be overriden either persistently or dynamically.
To always update the configuration files from GitHub, set the UPDATE_CHECK environment variable when creating the container by using the --env option, for example --env UPDATE_CHECK=Y.
To always update the configuration files from a specific GitHub account, set the G_ACCOUNT environment variable when creating the container by using the --env option, for example --env G_ACCOUNT=gh-fork-user.
[!NOTE] There is no way to change the environment variable of an already running container. To rollback the configuration files and scripts stop and remove the container and start it without setting the environment variable.
"},{"location":"docker/tips/#dynamically-updating-configuration-files","title":"Dynamically updating configuration files","text":"Set an environment file during create/run using --env-file=file, for example --env-file=/opt/cardano/cnode/.env.
UPDATE_CHECK is not defined in the environment file, the container will use the built-in configs.UPDATE_CHECK=Y is defined in the environment file the container will update configs and scripts from the cardano-community GitHub repository.G_ACCOUNT is defined in the environment file, the container will update configs and scripts from the GitHub repository of the specified account.To rollback the configuration files to the built-in versions, remove the UPDATE_CHECK=Y or set it to UPDATE_CHECK=N in the environment file. The static configuration files in the container will be used, however the scripts will remain updated. If you want both the configuration files and scripts to be rolled back, you will need to stop and remove the container and create a new one.
Run the Docker Image GitHub Action to build and push images to the ghcr.io registry.
G_ACCOUNT will be inherited from the GITHUB_REPOSITORY_OWNER.ghcr.io.ghcr.io registry as long as the Testing workflow remains checked.This documentation site (rather the repository itself) is created by some of the well known and experienced community members and contains instructions/information about various guild tools which simplify various stake-ops (setting up, managing and monitoring pools) for operators. Note that the guides are present to help you simplify your tasks - but as an entity responsible for creating blocks on a financial platform, we expect some basic pre-requisite skill sets - at professional level - before entering the portal:
cardano-cli, and have worked on preview/preprod/guild networks for pool operations without use of wrapper scripts - as an education exercise;Everyone is welcome to contribute to the repository (via documentation, testing, code, videos, etc). Our aim is to work together and reduce confusion rather than hosting 100 versions of documentation - each marketing their pool in a way.
"},{"location":"#support","title":"Support","text":"The Telegram Support channel is used to announce new releases and changes to the code base. This is also the place to ask general questions regarding the documentation and scripts on this site.
To report bugs and issues with scripts and documentation please open a GitHub Issue. Feature requests are best opened as a discussion thread.
"},{"location":"#getting-started","title":"Getting Started","text":"Use the sidebar to navigate through the topics. Note that the instructions assume the folder structure as per here.
Again, Feedback/Contribution and ownership of tasks is always welcome. If you're interested in collaborating regularly, make a start - and you should be part of the guild already .
"},{"location":"basics/","title":"Basics","text":""},{"location":"basics/#architecture","title":"Architecture","text":"The architecture for various components are already described at docs.cardano.org by CF/IOHK. We will not reinvent the wheel
"},{"location":"basics/#manual-software-pre-requirements","title":"Manual Software Pre-Requirements","text":"While we do not intend to hand out step-by-step instructions, the tools are often misused as a shortcut to avoid ensuring base skillsets mentioned on home page. Some of the common gotchas that we often find SPOs to miss out on:
Reminder !!
You're expected to run the commands below from same session, using same working directories as indicated and using a non-root user with sudo access. You are expected to be familiar with this as part of pre-requisite skill sets for stake pool operators.
The pre-requisites for Linux systems are automated to be executed as a single script. This script uses opt-in election of what you'd like the script to do. The defaults without any arguments will only update static part of script contents for you. To download the pre-requisites scripts, execute the below:
mkdir \"$HOME/tmp\";cd \"$HOME/tmp\"\n# Install curl\n# CentOS / RedHat - sudo dnf -y install curl\n# Ubuntu / Debian - sudo apt -y install curl\ncurl -sS -o guild-deploy.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/guild-deploy.sh\nchmod 755 guild-deploy.sh\nImportant !!
Please familiarise with the syntax of guild-deploy.sh before proceeding (using -h as below). The exact parameters you want to run with is dependent on your, below are only sample instructions
The usage syntax can be checked using ./guild-deploy.sh -h , sample output below:
Usage: guild-deploy.sh [-n <mainnet|guild|preprod|preview|sanchonet>] [-p path] [-t <name>] [-b <branch>] [-u] [-s [p][b][l][m][d][c][o][w][x][f][s]]\nSet up dependencies for building/using common tools across cardano ecosystem.\nThe script will always update dynamic content from existing scripts retaining existing user variables\n\n-n Connect to specified network instead of mainnet network (Default: connect to cardano mainnet network) eg: -n guild\n-p Parent folder path underneath which the top-level folder will be created (Default: /opt/cardano)\n-t Alternate name for top level folder - only alpha-numeric chars allowed (Default: cnode)\n-b Use alternate branch of scripts to download - only recommended for testing/development (Default: master)\n-u Skip update check for script itself\n-s Selective Install, only deploy specific components as below:\n p Install common pre-requisite OS-level Dependencies for most tools on this repo (Default: skip)\nb Install OS level dependencies for tools required while building cardano-node/cardano-db-sync components (Default: skip)\nl Build and Install libsodium fork from IO repositories (Default: skip)\nm Download latest (released) binaries for mithril-signer, mithril-client (Default: skip)\nd Download latest (released) binaries for bech32, cardano-address, cardano-node, cardano-cli, cardano-db-sync and cardano-submit-api (Default: skip)\nc Download latest (released) binaries for CNCLI (Default: skip)\no Download latest (released) binaries for Ogmios (Default: skip)\nw Download latest (released) binaries for Cardano Hardware CLI (Default: skip)\nx Download latest (released) binaries for Cardano Signer binary (Default: skip)\nf Force overwrite config files (backups of existing ones will be created) (Default: skip)\ns Force overwrite entire content [including user variables] of scripts (Default: skip)\nglibc, it would likely be due to the build mismatch between pre-compiled binary and your OS, which is not uncommon. You may need to compile cncli manually on your OS as per instructions here - make sure to copy the output binary to \"${HOME}/.local/bin\" folder.A typical example install to install most components but not overwrite static part of existing files for preview network would be:
./guild-deploy.sh -b master -n preview -t cnode -s pdlcowx\n. \"${HOME}/.bashrc\"\nIf instead of download, you'd want to build the components yourself, you could use:
./guild-deploy.sh -b master -n preview -t cnode -s pblcowx\n. \"${HOME}/.bashrc\"\nLastly, if you'd want to update your scripts but not install any additional dependencies, you may simply run:
./guild-deploy.sh -b master -n preview -t cnode\nRunning the script above will create the folder structure as per below, for your reference. You do NOT require CNODE_HOME to be set on shell level as scripts will derive the parent folder and assign it at runtime, the addition in ~/.bashrc is only for your ease to switch to right folder:
/opt/cardano/cnode # Top-Level Folder\n\u251c\u2500\u2500 ...\n\u251c\u2500\u2500 files # Config, genesis and topology files\n\u2502 \u251c\u2500\u2500 ...\n\u2502 \u251c\u2500\u2500 byron-genesis.json # Byron Genesis file referenced in config.json\n\u2502 \u251c\u2500\u2500 shelley-genesis.json # Genesis file referenced in config.json\n\u2502 \u251c\u2500\u2500 alonzo-genesis.json # Alonzo Genesis file referenced in config.json\n\u2502 \u251c\u2500\u2500 config.json # Config file used by cardano-node\n\u2502 \u2514\u2500\u2500 topology.json # Map of chain for cardano-node to boot from\n\u251c\u2500\u2500 db # DB Store for cardano-node\n\u251c\u2500\u2500 guild-db # DB Store for guild-specific tools and additions (eg: cncli, cardano-db-sync's schema)\n\u251c\u2500\u2500 logs # Logs for cardano-node\n\u251c\u2500\u2500 priv # Folder to store your keys (permission: 600)\n\u251c\u2500\u2500 scripts # Scripts to start and interact with cardano-node\n\u2514\u2500\u2500 sockets # Socket files created by cardano-node\nThe documentation here uses instructions from Intersect MBO repositories as foundation, with additional info which we can contribute to where appropriate. Note that not everyone needs to build each component. You can refer to architecture to understand and qualify which of the components built by IO you want to run.
"},{"location":"build/#components","title":"Components","text":"For most Pool Operators, simply building cardano-node should be enough. Use the below to decide whether you need other components:
graph TB A([Interact with HD Walletslocally]) B([Explore blockchainlocally]) C([Easy pool-ops andfund management]) D([Create Custom Assets]) E([Monitor node using Terminal UI]) F([Sign/verify any datausing crypto keys]) N(Node) O(Ogmios) P(gRest/Koios) Q(DBSync) R(Wallet) S(CNTools) T(Tx Submit API) U(GraphQL) V(OfflineMetadataTools) X(gLiveView) Y(cardano-signer) Z[(PostgreSQL)] N --x C --x S N --x D --x S & V N --x E --x X N --x B B --x U --x Q B --x P --x Q P --x O P --x T F ---x Y N --x A --x R Q --x ZImportant
We strongly prefer use of gRest over GraphQL components due to performance, security, simplicity, control and most importantly - consistency benefits. Please refer to official documentations if you're interested in GraphQL or Cardano-Rest components instead.
Note
The instructions are intentionally limited to stack/cabal** to avoid wait times/availability of nix/docker files on a rapidly developing codebase - this also helps us prevent managing multiple versions of instructions.
"},{"location":"build/#description-for-components-built-by-community","title":"Description for components built by community","text":""},{"location":"build/#cntools","title":"CNTools","text":"A swiss army knife for pool operators, primarily built by Ola, to simplify typical operations regarding their wallet keys and pool management. You can read more about it here
"},{"location":"build/#gliveview","title":"gLiveView","text":"A local node monitoring tool, primarily built by Ola, to use in addition to remote monitoring tools like Prometheus/Grafana, Zabbix or IOG's RTView. This is especially useful when moving to a systemd deployment - if you haven't done so already - as it offers an intuitive UI to monitor the node status. You can read more about it here
"},{"location":"build/#topology-updater","title":"Topology Updater","text":"A temporary node-to-node discovery solution, run by Markus, that was started initially to bridge the gap created while awaiting completion of P2P on cardano network, but has since become an important lifeline to the network health - to allow everyone to activate their relay nodes without having to postpone and wait for manual topology completion requests. You can read more about it here
"},{"location":"build/#koiosgrest","title":"Koios/gRest","text":"A full-featured local query layer node to explore blockchain data (via dbsync) using standardised pre-built queries served via API as per standard from Koios - for which user can opt to participate in elastic query layer. You can read more about build steps here and reference API endpoints here
"},{"location":"build/#ogmios","title":"Ogmios","text":"A lightweight bridge interface for cardano-node. It offers a WebSockets API that enables local clients to speak Ouroboros' mini-protocols via JSON/RPC. You can read more about it here
"},{"location":"build/#cncli","title":"CNCLI","text":"A CLI tool written in Rust by Andrew Westberg for low-level communication with cardano-node. It is commonly used by SPOs to check their leader logs (integrates with CNTools as well as gLiveView) or to send their pool's health information to https://pooltool.io. You can read more about it here
"},{"location":"build/#cardano-signer","title":"Cardano Signer","text":"A tool written by Martin to sign/verify data (hex, text or binary) using cryptographic keys to generate data as per CIP-8 or CIP-36 standards. You can read more about it here
"},{"location":"catalystf11/","title":"Catalystf11","text":""},{"location":"catalystf11/#marlowehub-unifying-platform-for-marlowe-smart-contracts-phase-1-smart-contracts","title":"MarloweHub: Unifying Platform for Marlowe Smart Contracts - Phase 1 - Smart Contracts","text":"Category: Concept Applicant: mike (pooltool.io) Requested funds: \u20b3100,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#adastat-cardano-explorer-open-source-improved-reboot-towards-a-first-class-community-blockchain-explorer","title":"AdaStat Cardano Explorer - Open Source Improved Reboot towards a first-class community blockchain explorer","text":"Category: Product Applicant: Dmytro Stashenko (adastat.net) Requested funds: \u20b3180,300.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#adahold-decentralized-price-can-only-go-up-token-solution-for-true-ada-hodlers-smart-contract","title":"AdaHold: Decentralized Price-Can-Only-Go-Up Token, Solution For TRUE Ada Hodlers - Smart Contract","text":"Category: Concept Applicant: Dmytro Stashenko (adastat.net) Requested funds: \u20b399,700.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#sundae-labs-next-gen-uplc-debugger-with-aiken-integration","title":"Sundae Labs Next-Gen UPLC Debugger with Aiken Integration","text":"Category: Developers Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3140,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#margin-pool-enhanced-liquidity-for-margin-trading-on-sundaeswap","title":"Margin Pool: Enhanced Liquidity for Margin Trading on SundaeSwap","text":"Category: Solution Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3300,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#regulated-and-permissioned-defi-with-sundae-and-kora-labs","title":"Regulated and Permissioned DeFi with Sundae and Kora Labs","text":"Category: Concept Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3100,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#sundae-labs-comprehensive-specification-development-for-gummiworm-protocol-on-cardano","title":"Sundae Labs Comprehensive Specification Development for Gummiworm Protocol on Cardano","text":"Category: Concept Applicant: Dan Gonzalez (sundae.fi) Requested funds: \u20b3100,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#nftcdnio-universal-nft-viewer-api-musicvideoweb3d","title":"[nftcdn.io] Universal NFT Viewer API (Music+Video+Web+3D+\u2026)","text":"Category: Product Applicant: Smaug (pool.pm) Requested funds: \u20b3299,999.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#nftcdnio-nsfw-nft-detection-for-marketplaces-wallets-explorers","title":"[nftcdn.io] NSFW NFT Detection for Marketplaces, Wallets & Explorers","text":"Category: Product Applicant: Smaug (pool.pm) Requested funds: \u20b3149,999.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#cardanoscan-analytics-charts","title":"Cardanoscan Analytics Charts","text":"Category: Product Applicant: Strica (cardanoscan.io) Requested funds: \u20b344,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#cardanoscan-api-javascript-sdk","title":"Cardanoscan API Javascript SDK","text":"Category: Developers Applicant: Strica (cardanoscan.io) Requested funds: \u20b364,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#integrate-keystone-hardware-wallet-into-typhon","title":"Integrate Keystone Hardware Wallet into Typhon","text":"Category: Product Applicant: Strica (cardanoscan.io) Requested funds: \u20b384,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#add-support-for-marlowe-on-cardanoscan","title":"Add support for Marlowe on Cardanoscan","text":"Category: Product Applicant: Strica (cardanoscan.io) Requested funds: \u20b3133,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#cardanoscan-data-info-bubbles","title":"Cardanoscan data info bubbles","text":"Category: Ecosystem Applicant: Strica (cardanoscan.io) Requested funds: \u20b335,000.00 Links: Ideascale, Lidonation
"},{"location":"catalystf11/#create-a-cnt-marketplace-on-norwegian-block-exchange-nbxcom","title":"Create a CNT marketplace on Norwegian Block Exchange (NBX.COM)","text":"Category: Product Applicant: Eystein Hansen (nbx.com) Requested funds: \u20b3145,000.00 Links: Ideascale, Lidonation
"},{"location":"contributors/","title":"Contributors","text":"Everyone is welcome to contribute to the guide, as well as the repository. Below is just a thank you to people who have been contributing consistently:
Adam Chris Damjan Homer Markus OCG Ola Ahlman Pal Dorogi Papacarp PegasusPool Psychomb RdLrT RedOracle SmaugPool
To start contributing, simply hit the github repository and raise Issue/Pull Request
"},{"location":"grest-meets/","title":"GRest Meeting summaries","text":"Thank you all for joining and contributing to the project
Below you can find a short summary of every GRest meeting held, both for logging purposes and for those who were not able to attend.
"},{"location":"grest-meets/#participants","title":"Participants:","text":"Participant 16Sep2021 02Sep2021 26Aug2021 19Aug2021 12Aug2021 29Jul2021 22Jul2021 15Jul2021 09Jul2021 02Jul2021 25Jun2021 Damjan Homer Markus Ola RdLrT Red Papacarp Paddy GimbaLabs 16Sep2021 02Sep2021 26Aug2021 19Aug2021 12Aug2021 29Jul2021 22Jul2021 15Jul2021 09Jul2021 02Jul2021After the initial stand-up updates from participants, we went through the entire Trello board, updating/deleting existing tickets and creating some new ones.
25Jun2021"},{"location":"grest-meets/#scheduling-running-update-queries","title":"Scheduling running update queries","text":"Solution being tested:
Pool cache table:
we will run the full query on regular intervals, ready for review for first iteration, will see about delta post tx cache query
transaction history:
need to think about how to approach inputs/outputs in the cached table (1 row per transaction with json objects for inputs/outputs or multiple rows for tx hash)
address_txs:
this endpoint should bring back list of txs, and have provision to use after and before block hash - lightweight against public schema
pool cache table:
create a trigger every 2 minutes (or similar) to run stake_distribution query
docker:
EXPLAIN (ANALYZE, BUFFERS)Team
grest schema)Individual
84226d33eed66be8e61d50b7e1dacebdc095cee9 on release/10.1.x<query>.json and sql in <query>.sql), also remove get_ prefixnbthreads in config, tune maxconn, switch to http mode)Ola added automatic deployment of services to the scripts last week. We added new tasks on Trello ticket, including flags for multiple networks (guild, testnet, mainnet), haproxy service dynamically creating hosts and doc updates. Overall, the script works well with some manual interaction still required at the moment.
"},{"location":"grest-meets/#supported-networks","title":"Supported Networks","text":"Just for the record here, a 16GB (or even 8GB) instance is enough to support both testnet and guild networks.
"},{"location":"grest-meets/#db-sync-versioning","title":"db-sync versioning","text":"We agreed to use the release/10.1.x branch which is not yet released but built to include Alonzo migrations to avoid rework later. This version does require Alonzo config and hash to be in the node's config.json. This has to be done manually and the files are available here. Once fully released, all members should rebuild the released version to ensure each instance is running the same code.
For the DNS setup ticket, we started to think about the instance names for the 2 DNS instances (orange in the graph). Submissions for names will be made in the Telegram group, and will probably make a poll once we have the entries finalised.
"},{"location":"grest-meets/#monitoring-system","title":"Monitoring System","text":"Priyank started setting up the monitoring on his instance which can then easily be switched to a separate monitoring instance. We agreed to use Prometheus / Grafana combo for data source / visualisation. We'll probably need to create some custom archiving of data to keep it long term as Prometheus stores only the last 30 days of data.
"},{"location":"grest-meets/#next-meeting","title":"Next meeting","text":"We would like to make Friday @ 07:00 UTC the standard time and keep meetings at weekly frequency. A poll will still be created for next weeks, but if there are no objections / requests for switching the time around (which we have not had so far) we can go ahead with the making Friday the standard with polls no longer required and only reminders / Google invites sent every week.
"},{"location":"grest-meets/#deployment-scripts_1","title":"Deployment scripts","text":"During the last week, work has been done on deployment scripts for all services (db-sync, gRest and haproxy) -> this is now in testing with updated instructions on trello. Everybody can put their name down on the ticket to signify when the setup is complete and note down any comments for bugs/improvements. This is the main priority at the moment as it would allow us to start transferring our setups to mainnet.
"},{"location":"grest-meets/#switch-to-mainnet","title":"Switch to Mainnet","text":"Following on from that, we created a ticket for starting to set up mainnet instances -> we can use 32GB RAM to start and increase later. While making sure everything works against the guild network is priority, people are free to start on this as well as we anticipate we are almost ready for the switch.
"},{"location":"grest-meets/#supported-networks_1","title":"Supported Networks","text":"This brings me to another discussion point which is on which networks are to be supported. After some discussion, it was agreed to keep beefy servers for mainnet, and have small independent instances for testnet maintained by those interested, while guild instance is pretty lightweight and useful to keep.
"},{"location":"grest-meets/#monitoring-system_1","title":"Monitoring System","text":"The ticket for creating a centralised monitoring system was discussed and updated. I would say it would be good to have at least a basic version of the system in place around the time we switch to mainnet. The system could eventually serve for: - analysis of instance - performances and subsequent tuning - endpoints usage - anticipation of system requirements increases - etc.
I would say that this should be an important topic of the next meeting to come up with an approach on how we will structure this system so that we can start building it in time for mainnet switch.
"},{"location":"grest-meets/#handling-ssl","title":"Handling SSL","text":"Enabling SSL was agreed to not be required by each instance, but is optional and documentation should be created for how to automate the process of renewing SSL certificates for those wishing to add it to their instance. The end user facing endpoints \"Instance Checker\" will of course be SSL-enabled.
"},{"location":"grest-meets/#next-meeting_1","title":"Next meeting","text":"We somewhat agreed to another meeting next week again at the same time, but some participants aren't 100% for availability. Friday at 07:00 UTC might be a good standard time we hold on to, but I will make a poll like last time so that we can get more info before confirming the meeting.
"},{"location":"grest-meets/#meeting-structure","title":"Meeting Structure","text":"As this was the first meeting, at the start we discussed about the meeting structure. In general, we agreed to something like listed below, but this can definitely change in the future:
1) 2-liner (60s) round the table stand-ups by everyone to sync up on what they were doing / are planning to do / mention struggles etc. This itself often sparks discussions. 2) going through the Trello board tasks with the intention of discussing and possbily assigning them to individuals / smaller groups (maybe 1-2-3 people choose to work together on a single task)
"},{"location":"grest-meets/#stand-ups","title":"Stand-ups","text":"We then proceeded to give a status of where we are individually in terms of what's been done, a summary below:
prereqs.sh addendum can be done once artifacts are finalised (added a Trello ticket for tracking).All in all, I think we saw that there is need for these meetings as there are a lot of things to discuss and new ideas come up (like the monitoring system). We went for over an hour (~1h15min) and still didn't have enough time to go through the board, we basically only touched the DNS/haproxy part of the board. This tells me that we are in a stage where more frequent meetings are required, weekly instead of biweekly, as we are in the initial stage and it's important to build things right from the start rather than having to refactor later on. With that, the participants in general agreed to another meeting next week, but this will be confirmed in the TG chat and the times can be discussed then.
"},{"location":"sidebar/","title":"Tree","text":"Warning
Make sure you go through upgrade steps for your setup in a non-mainnet environment first!!
guild-deploy.sh (always double check syntax of the script with guild-deploy.sh -h). The scripts modified with user content (env, gLiveView.sh, topologyUpdater.sh, cnode.sh, etc) will be backed up before overwriting. The backed up files will be in the same folder as the original files, and will be named as ${filename}_bkp<timestamp>. More static files (genesis files or some of the scripts themselves) will not be backed up, as they're not expected to be modified.Remember
You are expected to provide appropriate environment-specific parameters (eg: custom top level folder [-p], alternate name for top level folder [-t], network flag [-n], etc) to the examples that pertain to your use case
Depending on node release, you may be able to simply perform an update-in-place of scripts and node, or for some cases, you may need to overwrite configs as well. Some Examples below:
mkdir \"$HOME/tmp\";cd \"$HOME/tmp\"\ncurl -sfS -o guild-deploy.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/guild-deploy.sh && chmod 700 guild-deploy.sh\n./guild-deploy.sh -s dl -b master -n mainnet -t cnode -p /opt/cardano\n\"${CNODE_HOME}\"/files folder if you'd like to compare/reuse previous version.mkdir \"$HOME/tmp\";cd \"$HOME/tmp\"\ncurl -sfS -o guild-deploy.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/guild-deploy.sh && chmod 700 guild-deploy.sh\n./guild-deploy.sh -s dlf -b master -n mainnet -t cnode -p /opt/cardano\nBeware
When upgrading node, depending on node versions (especially for major release) - you'd likely have to wait for node to revalidate/replay ledger. This can take a few hours. Please always plan ahead, do it first on a relay to ensure you've got \"${CNODE_HOME}/db\" folder ready to copy over (while source and target node have been shutdown) - prior to starting on upgrade on new machine. If mithril for target node version is ready, you can also use mithril-client to download snapshot instead of replaying, which may save you some time
\"${HOME}\"/.local/bin is part of your \\(PATH environment variable. If your shell does not auto-run bashrc, you may want to set it a call to \"\\)\"/.bashrc in your .profilesource \"${HOME}\"/.bashrc\necho \"${PATH}\"\nWe've found users often confuse between \\(PATH variable resolution between multiple shell sessions, systemd, etc. While if you only used this guide, the binaries should be in \"\\)/.local/bin\", you may have manually downloaded to another location before. To avoid this, you can edit the following files and uncomment and set the following variables to the appropriate paths as per your deployment (eg: CCLI=\"${HOME}\"/.local/bin/cardano-cli if following above):
The above should take care of tools and services. However, you might still have duplicate binaries in your $PATH (previous artifacts, re-build using old scripts, etc) - it is best that you remove any old binary files from alternate folders. You can do so by executing the below:
whereis bech32 cardano-address cardano-cli cardano-db-sync cardano-hw-cli cardano-node cardano-submit-api cncli ogmios\nFor some cases - you might have no values (eg: you may not use cardano-db-sync, cncli, ogmios and/or cardano-hw-cli. You need not take any actions for the binaries you do not use.
sudo systemctl status cnode. If the node shows as up, you can monitor logs via tail -100f \"${CNODE_HOME}/logs/node.json. If you're unable to start node using systemd itself, you can check the systemd output via sudo journalctl -xeu cnode.service and scroll back until you see a startup attempt with reason (typically this could be a couple of pages back). If nothing obvious (eg: showing files used as /files/config.json instead of $CNODE_HOME/files/config.json )comes up from there, you'd want to view node logs to see if there is something recorded in node.json instead.Let's say you accidentally did an update of the files that you didnt intend to and want to revert all your scripts to previous state. While you have the backups of the scripts created while doing in-place update, you can always make use of branch flag that most scripts that perform update-in-place provide (eg: for gLiveView.sh that you want to restore to 8.1.2 state), this would be -b node-8.1.2. The available tags that can be used can be visited here
Hope the guide above helps you with the migration, but again - we could've missed some edge cases. If so, please report via chat in Koios Discussions channel or open an issue on github. Please DO NOT make edits to the script content based on forum/alternate guide/channels, while done with best intentions - there have been solutions put online that modify files unnecessarily instead of correcting configs and disabling updates, such actions will only cause trouble for future updates.
"},{"location":"Appendix/RecoverByronWallet/","title":"Unofficial Instructions for recovering your Byron Era funds on the new Incentivized Shelley Testnet","text":""},{"location":"Appendix/RecoverByronWallet/#1-grab-and-install-haskell","title":"1. Grab and install Haskell","text":"curl -sSL https://get.haskellstack.org/ | sh\nnote: you must build from source as of today as there are changes that just got into master you need
git clone https://github.com/cardano-foundation/cardano-wallet.git\ncd cardano-wallet\nstack build --test --no-run-tests\nsudo apt install libssl-dev\nsudo apt-get install sqlite3 libsqlite3-dev \nsudo apt-get install libgmp3-dev \nsudo apt install libsystemd-dev\nget coffee... It takes awhile
"},{"location":"Appendix/RecoverByronWallet/#5-when-its-done-install-executables-to-your-path","title":"5. When its done, install executables to your path","text":"stack install\nGenerate your new mnemonics you will need below. Note that this generates 15 words as opposed to your byron era mnemnomics which were only 12 words.
cardano-wallet-jormungandr mnemonic generate\nyou can either open another terminal window or use screen or something. anyway, wherever you run this next command you won't be able to use anymore for a terminal until you stop the wallet
change --node-port 3001 to wherever you have your jormungandr rest interface running. for me it was 5001.. so
change --port 3002 to wherever you want to access the wallet interface at. If you have other things running avoid those ports. for most, 3002 should be free
just to future proof these instructions. genesis should be whatever genesis you are on.
cardano-wallet-jormungandr serve --node-port 3001 --port 3002 --genesis-block-hash e03547a7effaf05021b40dd762d5c4cf944b991144f1ad507ef792ae54603197\n--->in another window
replace foo, foo, foo with all your mnemnomics from the byron wallet you are restoring
Also, if you put your wallet on a different port than 3002, fix that too
curl -X POST -H \"Content-Type: application/json\" -d '{ \"name\": \"legacy_wallet\", \"mnemonic_sentence\": [\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\"], \"passphrase\": \"areallylongpassword\"}' http://localhost:3002/v2/byron-wallets\nRemember all those mnemnomics you made above.. put them here instead of all the foo's.
curl -X POST -H \"Content-Type: application/json\" -d '{ \"name\": \"pool_wallet\", \"mnemonic_sentence\": [\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\",\"foo\"], \"passphrase\": \"areallylongpasswordagain\"}' http://localhost:3002/v2/wallets\nNow you are ready to migrate your wallet. replace the <old wallet id> and <new wallet id> with the values you got above
curl -X POST -H \"Content-Type: application/json\" -d '{\"passphrase\": \"areallylongpassword\"}' http://localhost:3002/v2/byron-wallets/<old wallet id>/migrations/<new wallet id>\nFrom here we recommend you send them to a new address entirely owned and created by jcli or whatever method you have been using for the testnet process.
This technically may not be required. But a lot of us did it and we know it works for setting up pools and stuff.
send a small amount first just to make sure you are in control of the transaction and don't send your funds to la la land.
If you want to send to another address use the command below, but replace the address that you want to send it to, the amount, and your <new wallet id>
curl -X POST -H \"Content-Type: application/json\" -d '{\"payments\": [ { \"address\": \"<address to send to>\"\", \"amount\": { \"quantity\": 83333330000000, \"unit\": \"lovelace\" } } ], \"passphrase\": \"areallylongpasswordagain\"}' http://localhost:3002/v2/wallets/<new wallet id>/transactions\nEnsure the Pre-Requisites are in place before you proceed.
This is an easy-to-use script to automate setting up of monitoring tools. Tasks automates the following tasks: - Installs Prometheus, Node Exporter and Grafana Servers for your respective Linux architecture. - Configure Prometheus to connect to cardano node and node exporter jobs. - Provisions the installed prometheus server to be automatically available as data source in Grafana. - Provisions two of the common grafana dashboards used to monitor cardano-node by SkyLight and IOHK to be readily consumed from Grafana. - Deploy prometheus,node_exporter and grafana-server as systemd service on Linux. - Start and enable those services.
Note that securing prometheus/grafana servers via TLS encryption and other security best practices are out of scope for this document, and its mainly aimed to help you get started with monitoring without much fuss.
!> Ensure that you've opened the firewall port for grafana server (default used in this script is 5000)
"},{"location":"Appendix/monitoring/#download-setup_monsh","title":"Download setup_mon.sh","text":"If you have run guild-deploy.sh, you can skip this step. To download monitoring script, you can execute the commands below:
cd $CNODE_HOME/scripts\nwget https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/setup_mon.sh\nchmod 750 setup_mon.sh\nThe default selection may not always be usable for everyone. You can customise further environment variable settings by opening in editor (eg: vi setup_mon.sh ), and updating variables below to your liking:
#!/usr/bin/env bash\n# shellcheck disable=SC2209,SC2164\n\n######################################################################\n#### Environment Variables\n######################################################################\nCNODE_IP=127.0.0.1\nCNODE_PORT=12798\nGRAFANA_HOST=0.0.0.0\nGRAFANA_PORT=5000\nPROJ_PATH=/opt/cardano/monitoring\nPROM_HOST=127.0.0.1\nPROM_PORT=9090\nNEXP_PORT=$(( PROM_PORT + 1 ))\n````\n\n#### Set up Monitoring\n\nExecute setup_mon.sh with full path to destination folder you want to setup monitoring in. If you're following guild folder structure, you do not need to specify `-d`. Read the usage comments below before you run the actual script.\n\nNote that to deploy services as systemd, the script expect sudo access is available to the user running the script.\n\n``` bash\ncd $CNODE_HOME/scripts\n# To check Usage parameters:\n# ./setup_mon.sh -h\n#Usage: setup_mon.sh [-d directory] [-h hostname] [-p port]\n#Setup monitoring using Prometheus and Grafana for Cardano Node\n#-d directory Directory where you'd like to deploy the packages for prometheus , node exporter and grafana\n#-i IP/hostname IPv4 address or a FQDN/DNS name where your cardano-node (relay) is running (check for hasPrometheus in config.json; eg: 127.0.0.1 if same machine as cardano-node)\n#-p port Port at which your cardano-node is exporting stats (check for hasPrometheus in config.json; eg: 12798)\n./setup_mon.sh\n# \n# Downloading prometheus v2.18.1...\n# Downloading grafana v7.0.0...\n# Downloading exporter v0.18.1...\n# Downloading grafana dashboard(s)...\n# - SKYLight Monitoring Dashboard\n# - IOHK Monitoring Dashboard\n# \n# NOTE: Could not create directory as rdlrt, attempting sudo ..\n# NOTE: No worries, sudo worked !! Moving on ..\n# Configuring components\n# Registering Prometheus as datasource in Grafana..\n# Creating service files as root..\n# \n# =====================================================\n# Installation is completed\n# =====================================================\n# \n# - Prometheus (default): http://127.0.0.1:9090/metrics\n# Node metrics: http://127.0.0.1:12798\n# Node exp metrics: http://127.0.0.1:9091\n# - Grafana (default): http://0.0.0.0:5000\n# \n# \n# You need to do the following to configure grafana:\n# 0. The services should already be started, verify if you can login to grafana, and prometheus. If using 127.0.0.1 as IP, you can check via curl\n# 1. Login to grafana as admin/admin (http://0.0.0.0:5000)\n# 2. Add \"prometheus\" (all lowercase) datasource (http://127.0.0.1:9090)\n# 3. Create a new dashboard by importing dashboards (left plus sign).\n# - Sometimes, the individual panel's \"prometheus\" datasource needs to be refreshed.\n# \n# Enjoy...\n# \n# Cleaning up...\nYou should now be able to Login to grafana dashboard, using the public IP of your server, at port 5000. The initial credentials to login would be admin/admin, and you will be asked to update your password upon first login. Once logged on, you should be able to go to Manage > Dashboards and select the dashboard you'd like to view. Note that if you've just started the server, you might see graphs as empty, as initial interval for dashboards is 12 hours. You can change it to 5 minutes by looking at top right section of the page.
Thanks to Pal Dorogi for the original setup instructions used for modifying.
"},{"location":"Appendix/postgres/","title":"Sample Postgres Setup","text":"These deployment instructions used for reference while building cardano-db-sync tool, with the scope being ease of set up, and some tuning baselines for those who are new to Postgres DB. It is recommended to customise these as per your needs for Production builds.
Important
You'd find it pretty useful to set up ZFS on your system prior to setting up Postgres, to help with your IOPs throughput requirements. You can find sample install instructions here. You can set up your entire root mount to be on ZFS, or you can opt to mount a file as ZFS on \"${CNODE_HOME}\"
"},{"location":"Appendix/postgres/#install-postgresql-server","title":"Install PostgreSQL Server","text":"Execute commands below to set up Postgres Server
# Determine OS platform\nOS_ID=$( (grep -i ^ID_LIKE= /etc/os-release || grep -i ^ID= /etc/os-release) | cut -d= -f 2)\nDISTRO=$(grep -i ^NAME= /etc/os-release | cut -d= -f 2)\n\nif [ -z \"${OS_ID##*debian*}\" ]; then\n#Debian/Ubuntu\nwget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -\n RELEASE=$(lsb_release -cs)\necho \"deb [arch=amd64] http://apt.postgresql.org/pub/repos/apt/ ${RELEASE}\"-pgdg main | sudo tee /etc/apt/sources.list.d/pgdg.list\n sudo apt-get update\n sudo apt-get -y install postgresql-17 postgresql-server-dev-17 postgresql-contrib libghc-hdbc-postgresql-dev\n sudo systemctl enable postgresql\nelse\necho \"We have no automated procedures for this ${DISTRO} system\"\nfi\nBefore you start populating your DB instance using dbsync data, now might be a good time to put some thought on to baseline configuration of your postgres instance by editing /etc/postgresql/17/main/postgresql.conf. Typically, you might find a lot of common standard practices parameters available in tuning guides. For our consideration, it would be nice to start with some baselines - for which we will use inputs from example here, which would need to be customised further to your environment and resources.
In a typical Koios [gRest] setup, we use below for minimum viable specs (i.e. 64GB RAM, > 8 CPUs, >16K IOPs for ioping -q -S512M -L -c 10 -s8k . output when postgres data directory is on ZFS configured with max arc of 4GB), we find the below configuration to be the best common setup:
In addition to above, due to the nature of usage by dbsync (synching from node and restart traversing back to last saved ledger-state snapshot), we leverage data retention on blockchain - as we're not affected by loss of volatile information upon a restart of instance. Thus, we can relax some of the data retention and protection against corruption related settings, as those are IOPs/CPU Load Average impacts that the instance does not need to spend. We'd recommend setting 3 of those below in your /etc/postgresql/17/main/postgresql.conf:
Once your changes are done, ensure to restart postgres service using sudo systemctl restart postgresql.
Login to Postgres instance as superuser:
echo $(whoami)\n# <user>\nsudo su postgres\npsql\nNote the returned as the output of echo $(whoami) command. Replace all instance of in the documentation below. Execute the below in psql prompt. Replace and PasswordYouWant with your OS user (output of echo $(whoami) command executed above) and a password you'd like to authenticate to Postgres with:
CREATE ROLE <user> SUPERUSER LOGIN;\n\\q\nexit at shell to return to your user from postgres"},{"location":"Appendix/postgres/#verify-login-to-postgres-instance","title":"Verify Login to postgres instance","text":"export PGPASSFILE=$CNODE_HOME/priv/.pgpass\necho \"/var/run/postgresql:5432:cexplorer:*:*\" > $PGPASSFILE\nchmod 0600 $PGPASSFILE\npsql postgres\n# psql (17.0)\n# Type \"help\" for help.\n# \n# postgres=#\nImportant
An average pool operator may not require cardano-db-sync at all. Please verify if it is required for your use as mentioned here.
PGPASSFILE environment variable is set as per the instructions in the sample guide, for db-sync to be able to connect.$CNODE_HOME/files/db-sync-config.json to toggle ledger to disable (read dbsync documentation for it's effects).Execute the below to clone the cardano-db-sync repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/intersectmbo/cardano-db-sync\ncd cardano-db-sync\nYou can use the instructions below to build the latest release of cardano-db-sync.
git fetch --tags --all\ngit pull\n# Include the cardano-crypto-praos and libsodium components for db-sync\n# On CentOS 7 (GCC 4.8.5) we should also do\n# echo -e \"package cryptonite\\n flags: -use_target_attributes\" >> cabal.project.local\n# Replace tag against checkout if you do not want to build the latest released version\ngit checkout $(curl -sLf https://api.github.com/repos/intersectmbo/cardano-db-sync/releases/latest | jq -r .tag_name)\n# Use `-l` argument if you'd like to use system libsodium instead of IOG fork of libsodium while compiling\n$CNODE_HOME/scripts/cabal-build-all.sh\ncardano-db-sync binary into ~/.local/bin folder."},{"location":"Build/dbsync/#prepare-db-for-sync","title":"Prepare DB for sync","text":"Now that binaries are available, let's create our database (when going through breaking changes, you may need to use --recreatedb instead of --createdb used for the first time. Again, we expect that PGPASSFILE environment variable is already set (refer to the top of this guide for sample instructions):
cd ~/git/cardano-db-sync\n# scripts/postgresql-setup.sh --dropdb #if exists already, will fail if it doesnt - thats OK\nscripts/postgresql-setup.sh --createdb\n# Password:\n# Password:\n# All good!\nVerify you can see \"All good!\" as above!
"},{"location":"Build/dbsync/#create-symlink-to-schema-folder","title":"Create Symlink to schema folder","text":"DBSync instance requires the schema files from the git repository to be present and available to the dbsync instance. You can either clone the ~/git/cardano-db-sync/schema folder OR create a symlink to the folder and make it available to the startup command we will be using. We will use the latter in sample below:
ln -s ~/git/cardano-db-sync/schema $CNODE_HOME/guild-db/schema\nIf you're running a mainnet/preview/preprod instance of dbsync, you might want to consider use of dbsync snapshots as documented here. The snapshot files from IO for their default configs as of recent epoch are available via links in release notes. Note that the snapshots should only be used pertaining to their specific configs, if using configs from Koios - you'd want to look at snapshots here instead.
At high-level, this would involve steps as below (read and update paths as per your environment):
# Replace the actual link below with the latest one from release notes\ncurl -fL 'https://share.koios.rest/api/public/dl/xFdZDfM4/dbsync/mainnet-dbsyncsnap-latest.tgz' -o /tmp/dbsyncsnap.tgz\nrm -rf ${CNODE_HOME}/guild-db/ledger-state ; mkdir -p ${CNODE_HOME}/guild-db/ledger-state\ncd -; cd ~/git/cardano-db-sync\nscripts/postgresql-setup.sh --restore-snapshot /tmp/dbsyncsnap.tgz ${CNODE_HOME}/guild-db/ledger-state\n# The restore may take a while, please be patient and do not interrupt the restore process. Once restore is successful, you may delete the downloaded snapshot as below:\n# rm -f /tmp/dbsyncsnap.tgz\nIn order to verify that you can run dbsync, before making a start - you'd want to ensure that you can run it interactively once. To do so, try the commands below:
cd $CNODE_HOME/scripts\nexport PGPASSFILE=$CNODE_HOME/priv/.pgpass\n./dbsync.sh\nYou can monitor logs if needed via parallel session using tail -10f $CNODE_HOME/logs/dbsync.json. If there are no error, you would want to press Ctrl-C to stop the dbsync.sh execution and deploy it as a systemd service. To do so, use the commands below (the creation of file is done using sudo permissions, but you can always deploy it manually):
cd $CNODE_HOME/scripts\n./dbsync.sh -d\n# Deploying cnode-dbsync.service as systemd service..\n# cnode-dbsync.service deployed successfully!!\nNow to start dbsync instance, you can run sudo systemctl start cnode-dbsync
Note
Note that dbsync while syncs, it might defer creation of indexes/constraints to speed up initial catch up. Once relatively closer to tip, this will initiate creation of indexes - which can take a while in background. Thus, you might notice the query timings right after reaching to tip might not be as good.
"},{"location":"Build/dbsync/#update-dbsync","title":"Update DBSync","text":"Updating dbsync can have different tasks depending on the versions involved. We attempt to briefly explain the tasks involved:
sudo systemctl stop cnode-dbsync)Go to your git folder, pull and checkout to latest version as in example below (if you were to switch to 13.6.0.2):
cd ~/git/cardano-db-sync\ngit pull\ngit checkout 13.6.0.2\nIf going through major version update (eg: 13.x.x.x to 14.x.x.x), you might need to rebuild and resync db from scratch, you may still follow the section to restore using snapshot to save some time (as long as you use a compatible snapshot).
cardano-node version has changed (specifically if it's ledger-state schema is different), you'd also need to clear the ledger-state directory (eg: rm -rf $CNODE_HOME/guild-db/ledger-state)dbsync.sh starts up fine manually as described above. If it does, stop it and go ahead with startup of systemd service (i.e. sudo systemctl start cnode-dbsync)To validate, connect to your postgres instance and execute commands as per below:
export PGPASSFILE=$CNODE_HOME/priv/.pgpass\npsql cexplorer\nYou should be at the psql prompt, you can check the tables and verify they're populated:
\\dt\nselect * from meta;\nA sample output of the above two commands may look like below (the number of tables and names may vary between versions):
cexplorer=# \\dt\nList of relations\n Schema | Name | Type | Owner\n--------+---------------------------+-------+-------\n public | ada_pots | table | centos\n public | admin_user | table | centos\n public | block | table | centos\n public | delegation | table | centos\n public | delisted_pool | table | centos\n public | epoch | table | centos\n public | epoch_param | table | centos\n public | epoch_stake | table | centos\n public | ma_tx_mint | table | centos\n public | ma_tx_out | table | centos\n public | meta | table | centos\n public | orphaned_reward | table | centos\n public | param_proposal | table | centos\n public | pool_hash | table | centos\n public | pool_meta_data | table | centos\n public | pool_metadata | table | centos\n public | pool_metadata_fetch_error | table | centos\n public | pool_metadata_ref | table | centos\n public | pool_owner | table | centos\n public | pool_relay | table | centos\n public | pool_retire | table | centos\n public | pool_update | table | centos\n public | pot_transfer | table | centos\n public | reserve | table | centos\n public | reserved_ticker | table | centos\n public | reward | table | centos\n public | schema_version | table | centos\n public | slot_leader | table | centos\n public | stake_address | table | centos\n public | stake_deregistration | table | centos\n public | stake_registration | table | centos\n public | treasury | table | centos\n public | tx | table | centos\n public | tx_in | table | centos\n public | tx_metadata | table | centos\n public | tx_out | table | centos\n public | withdrawal | table | centos\n(37 rows)\n\n\n\nselect * from meta;\n id | start_time | network_name\n----+---------------------+--------------\n 1 | 2017-09-23 21:44:51 | mainnet\n(1 row)\nThis release adds support for cardano db sync 13.6.0.2, alongwith underlying components supporting Conway HF. The major chunk of work for this release is behind the scenes, with minor value additions to input/output schema.
"},{"location":"Build/grest-changelog/#new-endpoints-added","title":"New endpoints added:","text":"/block_tx_info and /tx_info - Fix for _bytecode flag set to false not setting all byte fields to null #306/block_tx_info and /tx_info - _scripts flag set to false will no longer suppress reference_inputs, collateral_inputs or collateral_outputs #306/proposal_voting_summary - Add new fields drep_abstain_votes_cast, pool_abstain_votes_cast and committee_abstain_votes_cast #303/drep_info, /drep_metadata - Rename url to meta_url and hash to meta_hash , keeping it consistent with other endpoints #306/tx_cbor - Add new fields block_hash, block_height, epoch_no, absolute_slot, tx_timestamp #306/pool_info - Add new fields reward_addr_delegated_drep and voting_power #306view column with own bech32 utility functions #302This is a finalised release that builds on 1.2.0a to provide support for CIP-129 and add a summary of votes for given proposal. The changes accordingly are primarily only targetting Governance endpoints. This will be the version used for mainnet upgrade as well. Please go through the changelogs below
/proposal_voting_summary - Get a summary of votes cast on specified governance action #300/commitee_votes - Will require _cc_hot_id which will accept committee member hot key formatted in bech32 as per CIP-0005/129 #300/voter_proposal_list - Will require _voter_id which will accept DRep/SPO/Committee member formatted in bech32 as per CIP-0005/129 #300/proposal_votes - Will require _proposal_id which will accept government proposal ID formatted in bech32 as per CIP-129 #300/drep_metadata , /drep_updates, - added column has_script which shows if given credential is a script hash #300/drep_votes , /proposal_list , /committee_info - added column proposal_id to show proposal action ID in accordance with CIP-129 #300/proposal_votes , - voter is renamed to voter_id and shows DRep/Pool/Committee member formatted in bech32 as per CIP-129 #300This release starts providing Conway support providing 14 new endpoints - primarily focusing on new governance data. Also, based on community requests/feedbacks - it introduces a few breaking changes for tx_info and block_tx_info endpoints. Please go through the changelogs below
/tx_cbor - Raw transaction CBOR against a transaction #298/drep_epoch_summary - Summary of voting power and DRep count for each epoch #298/drep_list - List of all active delegated representatives (DReps) #298/drep_info - Get detailed information about requested delegated representatives (DReps) #298/drep_metadata - List metadata for requested delegated representatives (DReps) #298/drep_updates - List of updates for requested (or all) delegated representatives (DReps) #298/drep_votes - List of all votes casted by requested delegated representative (DRep) #298/drep_delegators - List of all delegators to requested delegated representative (DRep) #298/committee_info - Information about active committee and its members #298/committee_votes - List of all votes casted by given committee member or collective #298/proposal_list - List of all governance proposals #298/voter_proposal_list - List of all governance proposals for specified DRep, SPO or Committee credential #298/proposal_votes - List of all votes cast on specified governance action #298/pool_votes - List of all votes casted by a pool #298/block_tx_info, /tx_info - collateral_tx_out -> asset_list - Outputs for collateral tx out are never created on-chain and thus, cannot be queried from ma_tx_out. Instead a rough description of assets involved are saved , which is now returned as info from ledger. This is returned as-is from dbsync and does not adhere to asset_list schema we use in other endpoints. #298/tx_info , /block_tx_info - These endpoints now require you to specify what all information you'd like to retrieve from a transaction, providing flags _inputs , _metadata, _assets , _withdrawals, _certs, _scripts, _bytecode, _governance #298/policy_asset_mints , /policy_asset_info, /asset_info - Will return latest mint transaction that has metadata (instead of latest mint transaction) details (excluding burn transactions) #298/account_info , /pool_info , /pool_list - Add deposit field to output for deposit associated with registration #298/account_info - Add delegated_drep field to the output #298/block_tx_info , /tx_info - Add treasury_deposit, voting_procedures and proposal_procedures to the output #298/epoch_params - Add various fields to epoch_params as per Conway protocol parameters #298/pool_metadata, /pool_relays - Remove pool_status field from output (it's already listed in pool_info and list) #298/pool_updates - owners is now a JSONB field instead of JSONB array #298asset_info_cache - first_mint_tx_id , first_mint_keys , last_mint_keys are not used/required #286last_mint_meta_tx_id field to asset_info_cache - to return latest asset that does have metadata #286pool_history_cache #289pool_stat instead of epoch_stake for pool_history_cache #294instant_reward table in dbsync moved to reward_restada_pots : deposit now split into three different types of depositsThis release is minor bugfix for data consistency changes behind the scenes. It has no impact to any of the API endpoints.
"},{"location":"Build/grest-changelog/#chores_3","title":"Chores:","text":"pool_info : Add a pool_delegators_list RPC that brings across only minimal information required by pool_info, thereby improving it's performance #281pool_history stats for latest epochs by restricting cache to current - 3 epoch while calculating the subsequent information using live status from the database #282account_info , account_info_cached and stake_distribution_cache.This release primarily focuses on backend performance fixes and work with dbsync 13.2.0.2 - while also, we have started preparing compatibility with upcoming koios lite release, to make it a seamless swap for specific endpoints without any impact to consumers. There are no breaking (impact to existing columns or inputs) changes with this release, but we have retired 2 deprecated endpoints that were almost unused on mainnet. Due to the amount of backend changes in queries, there is a chance that we may have missed some data accuracy checks, and - hence - would like to test in non-mainnet networks first before marking final release. Accordingly, any testing/reports of data inconsistency would be welcome.
"},{"location":"Build/grest-changelog/#new-endpoints-added_3","title":"New endpoints added:","text":"/asset_policy_mints - List of mint/burn count for all assets minted under a policy #269/block_tx_info - Equivalent of tx_info but uses blocks as inputs to fetch tx_info against all tx in the block[s] requested, also contains additional flags to control performance and output #255/cli_protocol_params - Return protocl-parameters as returned by cardano-cli from Koios servers #269/reserve_withdrawals , /treasury_withdrawals - Add earned_epoch and spendable_epoch fields #269/block - Add parent_hash field #263/account_list - Add stake_address_hex and script_hash fields #263/asset_list - Add script_hash field #263/asset_summary - Add addresses field #263/asset_addresses , /asset_nft_address and /policy_asset_addresses - Add stake_address field #262/script_utxos as it was incorrectly returning object instead of array for asset_list #269/tx_info - Add plutus_contract -> spends_input to plutus_contracts to point the input transaction being consumed by the script #269asset_address_list and asset_policy_info endpoints are now retired, as they were marked as deprecated in Koios 1.0.10 , and we have seen it's usage to be negligible (only a single hit in 48 hours on mainnet while marking this release). #269stake_distribution_new_accounts and stake_snapshot_cache cache, as we directly perform lookup on live tables for newly registered accounts #269epoch_sync_progress table #269consumed_by_tx_in_id references in SQL by consumed_by_tx_id #269pool_history_cache now breaks into populating 500 epochs at a time (on guildnet, this query used to run for hours against ~20K epochs) #269reward table into instant_reward #269stake_distribution_cache to ensure that epoch info cache was run for current - 1 epoch. #269grest.cip67_strip_label from text to bytea #269tx_in as it is no longer required #269pool_offline_data with off_chain_pool_data #269asset-txo-cache-update as the endpoints leveraging asset-txo will be moved to koios-liteblock, account_list, asset_list asset_token_registry from view to function #263asset_info_cache - ensure mint tx is only against a positive mint quantity #262tx_info - Fix spend_redeemers CTE Join condition #269This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (/api/v1). The major work with this release was to start making use of newer flags in dbsync which help performance of queries under new endpoints. Please ensure to check out the release notes for 1.1.0rc below. The list for this section is only a small addendum to 1.1.0rc:
asset_addresses and policy_asset_addresses #250pool_registrations and pool_retirements) #249This will be first major [breaking] release for Koios consumers in a while, and will be rolled out under new base prefix (/api/v1). The major work with this release was to start making use of newer flags in dbsync which help performance of queries under new endpoints. Also, you'd see quite a few new endpoint additions below, that'd be helping out with slightly lighter version of queries. To keep migration paths easier, we will ensure both v0 and v1 versions of the release is up for a month post release, before retiring v0.
/pool_registrations - List of all pool registrations initiated in the requested epoch #239/pool_retirements - List of all pool retirements initiated in the requested epoch #239/treasury_withdrawals - List of withdrawals made from treasury #239/reserve_withdrawals - List of withdrawals made from reserves (MIRs) #239/account_txs - Transactions associated with a given stake address #239/address_utxos - Get UTxO details for requested addresses #239/asset_utxos - Get UTxO details for requested assets #239/script_utxos - Get UTxO details for requested script hashes #239/utxo_info - Details for requested UTxO arrays #239/script_info - Information about a given script FROM script hashes #239/ogmios/ - Expose stateless ogmios endpoints #1690/account_utxos , /credential_utxos - Accept extended as an additional flag - which enables asset_list, reference_script and inline_datum to the output #239/block_txs - Flatten output with transaction details (tx_hash, epoch_no, block_height, block_time) instead of tx_hashes array #239/epoch_params - Update cost_models to JSON (upstream change in node) #239/account_assets , /address_assets - Flatten the output result (instead of asset_list array) making it easier to apply horizontal filtering based on any of the fields/account_utxos , /address_utxos, /asset_utxos , /script_utxos and /utxo_info to return same schema giving complete details about UTxOs involved, with few fields toggled based on extended input flag #239/pool_list - Add various details to the endpoint for each pool (pool_id_hex,active_epoch_no,margin,fixed_cost,pledge,reward_addr,owners,relays,ticker,meta_url,meta_hash,pool_status,retiring_epoch) - this should mean some of the requests to pool_info should no longer be required #239/pool_updates - In v0, pool_updates only provided pool registration updates, while pool_status corresponded to current status of pool. With v1, we will have registration as well as deregistration transactions, and each transaction will have update_type (enum of either registration or deregistration) instead of pool_status. As a side-effect, since a registration transaction only has retiring_epoch as metadata, all the other fields will show up as null for such a transaction #239/pool_metadata , /pool_relays - Add pool_status field to denote whether pool is retired #239/datum_info - Rename hash to datum_hash and add creation_tx_hash #239/native_script_list - Remove script column (as it has pretty large output better queried against script_info), add size and change type to text #239/plutus_script_list - Add type and size to output #239/asset_info - Add cip68_metadata JSONB field #239/pool_history - Add member_rewards #225/tx_utxos - No longer required as replaced by /utxo_info #239v1 from v0 #1690epoch_info_cache Remove protocol parameters, as they can be queried from live table. Accordingly update dependent queries #239consumed_by_tx_in_id column in tx_out from dbsync 13.1.1.3 across endpoints #239_last_active_stake_validated_epoch in active_stake_cache #222The release is effectively same as 1.0.10rc except with one minor modification below.
cs.[{\"key\":\"value\"}] in PostgREST #172This release primarily focuses on ability to support better DeFi projects alongwith some value addition for existing clients by bringing in 10 new endpoints (paired with 2 deprecations), and few additional optional input parameters , and some additional output columns to existing endpoints. The only breaking change/fix is for output returned for tx_info.
Also, dbsync 13.1.x.x has been released and is recommended to be used for this release
"},{"location":"Build/grest-changelog/#new-endpoints-added_5","title":"New endpoints added","text":"/asset_addresses - Equivalent of deprecated /asset_address_list #149/asset_nft_address - Returns address where the specified NFT sits on #149/account_utxos - Returns brief details on non-empty UTxOs associated with a given stake address #149/asset_info_bulk - Bulk version of /asset_info #142/asset_token_registry - Returns assets registered via token registry on github #145/credential_utxos - Returns UTxOs associated with a payment credential #149/param_updates - Returns list of parameter update proposals applied to the network #149/policy_asset_addresses - Returns addresses with quantity for each asset on a given policy #149/policy_asset_info - Equivalent of deprecated /asset_policy_info but with more details in output #149/policy_asset_list - Returns list of asset under the given policy (including supply) #142, #149/account_addresses - Add optional _first_only and _empty flags to show only first address with tx or to include empty addresses to output #149/epoch_info - Add optional _include_next_epoch field to show next epoch stats if available (eg: nonce, active stake) #143/account_assets , /address_assets , /address_info, /tx_info, /tx_utxos - Add decimals to output #142/policy_asset_info - Add minting_tx_hash, total_supply, mint_cnt, burn_cnt and creation_time fields to the output #149/tx_info - Change _invalid_before and _invalid_after to text field #141tx_info - Remove the field plutus_contracts > [array] > outputs as there is no logic to connect it to inputs spending #163/asset_address_list - Renamed to asset_addresses keeping naming line with other endpoints (old one still present, but will be deprecated in future release) #149/asset_policy_info - Renamed to policy_asset_info keeping naming line with other endpoints (old one still present, but will be deprecated in future release) #149/epoch_info, /epoch_params - Restrict output to current epoch #149/block_info - Use /previous_id field to show previous/next blocks (previously was using block_id/height) #145/asset_info/asset_policy_info - Fix mint tx data to be latest #141grest.asset_info_cache to hold mint/burn counts alongwith first/last mint tx/keys #142/pool_delegators output column latest_delegation_tx_hash #149authenticator user, whose default statement_timeout is set to 65s and update configs accordingly #1606This release is effectively same as 1.0.9rc below (please check out the notes accordingly), just with minor bug fix on setup-grest.sh itself.
This release candidate is non-breaking for existing methods and inputs, but breaking for output objects for endpoints. The aim with release candidate version is to allow folks couple of weeks to test, adapt their libraries before applying to mainnet.
"},{"location":"Build/grest-changelog/#new-endpoints-added_6","title":"New endpoints added","text":"datum_info - List of datum information for given datum hashesaccount_info_cached - Same as account_info, but serves cached information instead of live oneaddress_info, address_assets, account_assets, tx_info, asset_list asset_summary to align output asset_list object to return array of policy_id, asset_name, fingerprint (and quantity, minting_txs where applicable) #120asset_history - Fix metadata to wrap in array to refer to right object #122asset_txs - Add optional boolean parameter _history (default: false) to toggle between querying current UTxO set vs entire history for asset #122pool_history - fixed_cost, pool_fees, deleg_rewards, epoch_ros will be returned as 0 when null #122tx_info - plutus_contracts->outputs can be null #122guild-operators repository to koios-artifacts repository. This is to ensure that the updates made to scripts and other tooling do not have a dependency on Koios query versioning #122block_info - Use block_no instead of id to check for previous/next block hash #122This release is contains minor bug-fixes that were discovered in koios-1.0.7. No major changes to output for this one.
"},{"location":"Build/grest-changelog/#changes-for-api","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#new-endpoints-added_7","title":"New endpoints added","text":"tx_info and tx_metadata - Align metadata for JSON output format #1542blocks - Query Output aligned to specs (epoch => epoch_no)epoch_block_protocols - [ ** Specs only ** ] Fix Documentation schema , which was accidentally showing wrong outputpool_delegators_history - List all epochs instead of current, if no _epoch_no is specified #1545asset_info - Fix metadata aggregaton for minting transactions with multiple metadata keys #1543stake_distribution_new_accounts - Leftover reference for account_info which now accepts array, resulted in error to populate stake distribution cache for new accounts #1541grest-poll.sh - Remove query view section from polling script, and remove grestrpcs re-creation per hour (it's already updated when setup-grest.sh is run) , in preparation for #1545This release continues updates from koios-1.0.6 to further utilise stake-snapshot cache tables which would be useful for SPOs as well as reduce downtime post epoch transition. One largely requested feature to accept bulk inputs for many block/address/account endpoints is now complete. Additionally, koios instance providers are now recommended to use cardano-node 1.35.3 with dbsync 13.0.5.
"},{"location":"Build/grest-changelog/#changes-for-api_1","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#new-endpoints-added_8","title":"New endpoints added","text":"pool_delegators_history - Provides historical record for pool's delegators #1486pool_stake_snapshot - Provides mark, set and go snapshot values for pool being queried. #1489pool_delegators - No longer accepts _epoch_no as parameter, as it only returns live delegators. Additionally provides latest_delegation_hash in output. #1486tx_info - epoch => epoch_no #1494tx_info - Change collateral_outputs (array) to collateral_output (object) as collateral output is only singular in current implementation #1496address_info - Add inline_datum and reference_script to output #1500pool_info - Add sigma field to output #1511pool_updates - Add historical metadata information to output #1503_stake_address text becomes _stake_addresses text[]). The additional changes in output as below:block_txs - Now returns block_hash and array of tx_hashesaddress_info - Additional field address returned in outputaddress_assets - Now returns address and an array of assets JSONaccount_addresses - Accepts stake_addresses array and outputs stake_address and array of addressesaccount_assets - Accepts stake_addresses array and outputs stake_address and array of assets JSONaccount_history - Accepts stake_addresses array alongwith epoch_no integer and outputs stake_address and array of history JSONaccount_info - Accepts stake_addresses array and returns additional field stake_address to outputaccount_rewards - Now returns stake_address and an array of rewards JSONaccount_updates - Now returns stake_address and an array of updates JSONasset_info - Change minting_tx_metadata from array to object #1533account_addresses - Sort results by oldest address first #1538epoch_info_cache - Only update last_tx_id of previous epoch on epoch transition #1490 and #1502epoch_info_cache / stake_snapshot_cache - Store total snapshot stake to epoch stake cache, and active pool stake to stake snapshot cache #1485The backlog of items not being added to mainnet has been increasing due to delays with Vasil HFC event to Mainnet. As such we had to come up with a split update approach. The mainnet nodes are still not qualified to be Vasil-ready (in our opinion) for 1.35.x , but dbsync 13 can be used against node 1.34.1 fine. In order to cater for this split, we have added an intermediate koios-1.0.6m tag that brings dbsync updates while maintaining node-1.34.1.
"},{"location":"Build/grest-changelog/#changes-for-api_2","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#data-output-changes","title":"Data Output Changes","text":"pool_delegators - epoch_no => active_epoch_no #1454asset_history - Add block_time and metadata fields for all previous mint transactions #1468asset_info - Retain latest mint transaction instead of first (in line with most CIPs as well as pool metadata - latest valid meta being live) #1468/tip , /blocks, /block_info => block_time/genesis => systemStart/epoch_info => start_time, first_block_time, last_block_time, end_time/tx_info => tx_timestamp/asset_info => creation_timetx_info - Add Vasil data #1464collaterals => collateral_inputscollateral_outputs, reference_inputs to tx_infodatum_hash, inline_datum, reference_script to collateral input/outputs, reference inputs & inputs/outputs JSON.cost_model instead of cost_model_id referenceepoch_params - Update leftover lovelace references to text for consistency: #1484key_depositpool_depositmin_utxo_valuemin_pool_costcoins_per_utxo_sizeget-metrics.sh - Add active/idle connections to database #1459grest-poll.sh: Bump haproxy to 2.6.1 and set default value of API_STRUCT_DEFINITION to be dependent on network used. #1450grest.account_active_stake_cache - optimise code and delete historical view (beyond 4 epochs). [#1451(https://github.com/cardano-community/guild-operators/pull/1451)tx_metalabels - Move metalabels from view to RPC using lose indexscan (much better performance) #1474grest.stake_snapshot_cache - Fix rewards for new accounts #1476Since there have been a few deviations wrt Vasil for testnet and mainnet, this version only targets networks except Mainnet!
"},{"location":"Build/grest-changelog/#changes-for-api_3","title":"Changes for API","text":""},{"location":"Build/grest-changelog/#data-output-changes_1","title":"Data Output Changes","text":"/epoch_info - Add total_rewards and avg_block_reward for a given epoch #43/tip , /blocks, /block_info => block_time/genesis => systemStart/epoch_info => start_time, first_block_time, last_block_time, end_time/tx_info => tx_timestamp/asset_info => creation_time/blocks, /block_info => Add proto_major and proto_minor for a given block to output #55asset_registry_update.sh script to rely on commit hash instead of POSIX timestamps, and performance bump. #1428epoch_no, block_no to /address_txs, /credential_txs and /asset_txs endpoints. #1409/asset_txs, returning transactions as an array - allows for leveraging native PostgREST filtering. #1409/pool_info. #1414setup-grest.sh with -r (reset flag), as the delta registry records to insert depends on file (POSIX) timestamps. #1410grest-poll.sh. Important
gRest is an open source implementation of a query layer built over dbsync using PostgREST and HAProxy. The package is built as part of Koios team's efforts to unite community individual stream of work together and give back a more aligned structure to query dbsync and adopt standardisation to queries utilising open-source tooling as well as collaboration. In addition to these, there are also accessibility features to deploy rules for failover, do healthchecks, set up priorities, have ability to prevent DDoS attacks, provide timeouts, report tips for analysis over a longer period, etc - which can prove to be really useful when performing any analysis for instances.
Note
Note that the scripts below do allow for provisioning ogmios integration too, but Ogmios - currently - is not designed to provide advanced session management for a server-client architecture in absence of a middleware. Thus, the availability for ogmios from monitoring instance is restricted to avoid ability to DDoS an instance.
"},{"location":"Build/grest/#components","title":"Components","text":"PostgREST: An RPC JSON interface for any PostgreSQL database (in our case, database served via cardano-db-sync) to provide a RESTful Web Service. The endpoints of PostgREST in itself are essentially the table/functions defined in elected schema via grest config file. You can read more about advanced query syntax using PostgREST API here, but we will provide a simpler view using examples towards the end of the page. It is an easy alternative - with almost no overhead as it directly serves the underlying database as an API, as compared to Cardano GraphQL component (which may often have lags). Some of the other advantages of PostgREST over graphql based projects are also performance, being stateless, 0 overhead, support for JWT / native Postgres DB authentication against the Rest Interface as well.
HAProxy: An easy gateway proxy that automatically provides failover/basic DDoS protection, specify rules management for load balancing, setup multiple frontend/backends, provide easy means to have TLS enabled for public facing instances, etc. You may alter the settings for proxy layer as per your SecOps preferences. This component is optional (eg: if you prefer to expose your PostgREST server itself, you can do so using similar steps below).
To start with you'd want to ensure your current shell session has access to Postgres credentials, continuing from examples from the above mentioned Sample Postgres deployment guide.
cd $CNODE_HOME/priv\nPGPASSFILE=$CNODE_HOME/priv/.pgpass\npsql cexplorer\nEnsure that you can connect to your Postgres DB fine using above (quit from psql once validated using \\q). As part of guild-deploy.sh execution, you'd find setup-grest.sh file made available in ${CNODE_HOME}/scripts folder, which will help you automate installation of PostgREST, HAProxy as well as brings in latest queries/functions provided via Koios to your instances.
Warning
As of now, gRest services are in alpha stage - while can be utilised, please remember there may be breaking changes and every collaborator is expected to work with the team to keep their instances up-to-date using alpha branch.
Familiarise with the usage options for the setup script , the syntax can be viewed as below:
cd \"${CNODE_HOME}\"/scripts\n./setup-grest.sh -h\n#\n# Usage: setup-grest.sh [-f] [-i [p][r][m][c][d]] [-u] [-b <branch>]\n# \n# Install and setup haproxy, PostgREST, polling services and create systemd services for haproxy, postgREST and dbsync\n# \n# -f Force overwrite of all files including normally saved user config sections\n# -i Set-up Components individually. If this option is not specified, components will only be installed if found missing (eg: -i prcd)\n# p Install/Update PostgREST binaries by downloading latest release from github.\n# r (Re-)Install Reverse Proxy Monitoring Layer (haproxy) binaries and config\n# m Install/Update Monitoring agent scripts\n# c Overwrite haproxy, postgREST configs\n# d Overwrite systemd definitions\n# -u Skip update check for setup script itself\n# -q Run all DB Queries to update on postgres (includes creating grest schema, and re-creating views/genesis table/functions/triggers and setting up cron jobs)\n# -b Use alternate branch of scripts to download - only recommended for testing/development (Default: master)\n#\nTo run the setup overwriting all standard deployment tasks from a branch (eg: koios-1.0.9 branch), you may want to use:
./setup-grest.sh -f -i prmcd -r -q -b koios-1.0.9\nSimilarly - if you'd like to re-install all components and force overwrite all configs but not reset cache tables, you may run:
./setup-grest.sh -f -i prmcd -q\nAnother example could be to preserve your config, but only update queries using an alternate branch (eg: let's say you want to try the branch alpha prior to a tagged release). To do so, you may run:
./setup-grest.sh -q -b alpha\nPlease ensure to follow the on-screen instructions, if any (for example restarting deployed services, or updating configs to specify correct target postgres URLs/enable TLS/add peers etc in ${CNODE_HOME}/priv/grest.conf and ${CNODE_HOME}/files/haproxy.cfg).
The default ports used will make haproxy instance available at port 8053 or 8453 if TLS is enabled (you might want to enable firewall rule to open this port to services you would like to access). If you want to prevent unauthenticated access to grest schema, uncomment the jwt-secret and specify a custom secret-token.
Reminder
Once you've successfully deployed the grest instance, it will deploy certain cron jobs that will ensure the relevant cache tables are updated periodically. Until these have finished (especially on first run, it could take an hour or so on mainnet, your instance will likely not pass any tests from grest-poll.sh but that's expected.
In order to enable SSL on your haproxy, all you need to do is edit the file ${CNODE_HOME}/files/haproxy.cfg and update the frontend app section to uncomment ssl bind (and comment normal bind).
Info
If you're not familiar with how to configure TLS OR would not like to buy one, you can find tips on how to create a TLS certificate for free via LetsEncrypt using tutorials here. Once you do have a TLS Certificate generated, you need to chain the private key and full chain cert together in a file - /etc/ssl/server.pem - which can be then referenced as below:
frontend app\n #bind 0.0.0.0:8053\n ## If using SSL, comment line above and uncomment line below\n bind :8453 ssl crt /etc/ssl/server.pem no-sslv3\n http-request set-log-level silent\n acl srv_down nbsrv(grest_postgrest) eq 0\n acl is_wss hdr(Upgrade) -i websocket\n ...\nWith the setup, you also have a checkstatus.sh script, which will query the Postgres DB instance via haproxy (coming through postgREST), and only show an instance up if the latest block in your DB instance is within 180 seconds.
Important
If you'd like to participate in joining to the elastic cluster via Koios, please raise a PR request by editing topology files in this folder to do so!!
If you were using guild network, you could do a couple of very basic sanity checks as per below:
To query active stake for pool pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr in epoch 122, we can execute the below:
curl -d _pool_bech32=pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr -d _epoch_no=122 -s http://localhost:8053/rpc/pool_active_stake\n## {\"active_stake_sum\" : 19409732875}\nTo check latest owner key(s) for a given pool pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr, you can execute the below:
curl -d _pool_bech32=pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr -s http://localhost:8050/rpc/pool_owners\n## [{\"owner\" : \"stake_test1upx5p04dn3t6dvhfh27744su35vvasgaaq565jdxwlxfq5sdjwksw\"}, {\"owner\" : \"stake_test1uqak99cgtrtpean8wqwp7d9taaqkt9gkkxga05m5azcg27chnzfry\"}]\nYou may want to explore what all endpoints come out of the box, and test them out, to do so - refer to API documentation for OpenAPI3 documentation. Each endpoint has a pre-filled example for mainnet and connects by default to primary Koios endpoint, allowing you to test endpoints and if needed - grab the curl commands to start testing yourself against your local or remote instances.
If you're interested to participate in decentralised infrastructure by providing an instance, there are a few additional steps you'd need:
Enable ports for your HAProxy instance (default: 8053), gRest Exporter service (default: 8059) and (optionally) submit API instance (default: 8090) against the monitoring instance (do not need to open these ports to internet) of corresponding network.
Ensure that each of the service above is listening on your public IP address (for instance, submitapi.sh might need to be edited to change HOSTADDR to 0.0.0.0 and restarted).
Create a PR specifying connectivity information to your HAProxy port here.
Make sure to join the telegram discussions group to participate in any discussions, actions, polls for new-features, etc. Feel free to give a shout in the group in case you have trouble following any of the above
Reminder !!
Unless there is a very particular reason you want to compile (eg: running on non-popular OS flavor), you DO NOT \"need\" to build node binaries - guild-deploy.sh already provides an option to download pre-compiled binaries. Ensure the Pre-Requisites are in place before you proceed.
Execute the below to clone the cardano-node repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/intersectmbo/cardano-node\ncd cardano-node\nYou can use the instructions below to build the latest release of cardano-node.
git fetch --tags --recurse-submodules --all\ngit pull\n# Replace tag against checkout if you do not want to build the latest released version, we recommend using battle tested node versions - which may not always be latest\ngit checkout $(curl -sLf https://api.github.com/repos/intersectmbo/cardano-node/releases/latest | jq -r .tag_name)\n\n# Use `-l` argument if you'd like to use system libsodium instead of IOG fork of libsodium while compiling\n$CNODE_HOME/scripts/cabal-build-all.sh\nThe above would copy the binaries built into ~/.local/bin folder.
While certain folks might want to build the node themselves (could be due to OS/arch compatibility, trust factor or customisations), for most it might not make sense to build the node locally. Instead, you can download the binaries using cardano-node release notes, where-in you can find the download links for every version. This is already taken care of by guild-deploy.sh if you used the option to download binaries (you can always re-run with specific arguments if unsure).
Execute cardano-cli and cardano-node to verify output as below (the exact version and git rev should depend on your checkout tag on github repository):
cardano-cli version\n# cardano-cli 10.x.x - linux-x86_64 - ghc-8.10\n# git rev <...>\ncardano-node version\n# cardano-node 10.x.x - linux-x86_64 - ghc-8.10\n# git rev <...>\nBefore you go ahead with starting your node, you may want to update values for CNODE_PORT in $CNODE_HOME/scripts/env. Note that it is imperative for operational relays and pools to ensure that the port mentioned is opened via firewall to the destination your node is supposed to connect from. Update your network/firewall configuration accordingly. Future executions of guild-deploy.sh will preserve and not overwrite these values (or atleast back up if forced to overwrite).
CNODEBIN=\"${HOME}/.local/bin/cardano-node\"\nCCLI=\"${HOME}/.local/bin/cardano-cli\"\nCNODE_PORT=6000\nPOOL_NAME=\"GUILD\"\nImportant
POOL_NAME is the name of folder that you will use when registering pools and starting node in core mode. This folder would typically contain your hot.skey,vrf.skey and op.cert files required. If the mentioned files are absent (expected if this is a fresh install), the node will automatically start in a relay mode.
To test starting the node in interactive mode, we will make use of pre-built script cnode.sh. This script automatically determines whether to start the node as a relay or block producer (if the required pool keys are present in the $CNODE_HOME/priv/pool/<POOL_NAME> as mentioned above). If the <MITHRIL_DOWNLOAD> variable is set to 'Y' it will download the latest snapshot from a Mithril aggregator to speed up the blockchain synchronization. The script contains a user-defined variable CPU_CORES which determines the number of CPU cores the node will use upon start-up:
######################################\n# User Variables - Change as desired #\n# Common variables set in env file #\n######################################\n\n#CPU_CORES=4 # Number of CPU cores cardano-node process has access to (please don't set higher than physical core count, 4 recommended)\nNow let's test starting the node in interactive mode.
Note
At this stage, upon executing cnode.sh, you are expected to see the live config and a line ending with Listening on http://127.0.0.1:12798 - this is expected, as your logs are being written to $CNODE_HOME/logs/node.json . If so, you should be alright to return to your console by pressing Ctrl-C. The node will be started later using instructions below using systemd (Linux's service management). In case you receive any errors, please troubleshoot and fix those before proceeding.
cd \"${CNODE_HOME}\"/scripts\n./cnode.sh\nPress Ctrl-C to exit node and return to console.
"},{"location":"Build/node-cli/#modify-the-nodes-config-files","title":"Modify the node's config files","text":"Now that you've tested the basic node operation, you might want to customise your config files (assuming you are in top-level folder , i.e. cd \"${CNODE_HOME}\") :
files/config.json : This file contains the logging configurations (tracers of to tune logging, paths for other genesis config files, address/ports on which the prometheus/EKG monitoring will listen, etc). Unless running more than one node on same machine (not recommended), you should be alright to use most of this file as-is. You might - however - want to double-check PeerSharing in this file, if using a relay node where you'd like connecting peers (marked as \"advertise\": \"true\" in topology.json) to be shared , you may turn this setting to true.
files/topology.json : This file tells your node how to connect to other nodes (especially initially to start synching). You would want to update this file as below:
localRoots > accessPoints section to include your local nodes that you want persistent connection against (eg: this could be your BP and own relay nodes) against definition where trustable is set to true.advertise to true for that peer group. You do NOT want to do that on BPlocalRoots > valency (valency is the same as hotValency, not yet replaced since the example in cardano-node-wiki repo still suggests valency) to number of connections from your localRoots that you always want to keep active connection to for that node.publicRoots section as well as localRoots > accessPoints as desired, tho defaults populated should work fine. On mainnet, we did add a few additional nodes to help add more redundancy for initial sync.useLedgerAfterSlot tells the node to establish networking with nodes from defined peers to sync the node initially until reaching an absolute slot number, after which - it can start attempting to connect to peers registered as pool relays on the network. You may want this number to be relatively recent (eg: not have it 50 epochs old).Important
On BP, You'd want to set useLedgerAfterSlot to -1 for your Block Producing (Core) node - thereby, telling your Core node to remain in non-P2P mode, and ensure PeerSharing is to false.
The resultant topology file on a relay could look something like below:
{\n\"bootstrapPeers\": [\n{\n\"address\": \"backbone.cardano.iog.io\",\n\"port\": 3001\n},\n{\n\"address\": \"backbone.mainnet.emurgornd.com\",\n\"port\": 3001\n},\n{\n\"address\": \"backbone.mainnet.cardanofoundation.org\",\n\"port\": 3001\n}\n],\n\"localRoots\": [\n{\n\"accessPoints\": [\n{\"address\": \"xx.xx.xx.xx\", \"port\": 6000 , \"description\": \"Core\"},\n{\"address\": \"zz.zz.zz.zz\", \"port\": 6000 , \"description\": \"Relay2\"}\n],\n\"advertise\": false,\n\"trustable\": true,\n\"hotValency\": 2\n},\n{\n\"accessPoints\": [\n{\"address\": \"node-dus.poolunder.com\", \"port\": 6900, \"pool\": \"UNDR\", \"location\": \"EU/DE/Dusseldorf\" },\n{\"address\": \"node-syd.poolunder.com\", \"port\": 6900, \"pool\": \"UNDR\", \"location\": \"OC/AU/Sydney\" },\n{\"address\": \"194.36.145.157\", \"port\": 6000, \"pool\": \"RDLRT\", \"location\": \"EU/DE/Baden\" },\n{\"address\": \"95.216.38.251\", \"port\": 6000, \"pool\": \"RDLRT\", \"location\": \"EU/FI/Helsinki\" },\n{\"address\": \"148.72.153.168\", \"port\": 16000, \"pool\": \"AAA\", \"location\": \"NA/US/StLouis\" },\n{\"address\": \"78.47.99.41\", \"port\": 6000, \"pool\": \"AAA\", \"location\": \"EU/DE/Nuremberg\" },\n{\"address\": \"relay1-pub.ahlnet.nu\", \"port\": 2111, \"pool\": \"AHL\", \"location\": \"EU/SE/Malmo\" },\n{\"address\": \"relay2-pub.ahlnet.nu\", \"port\": 2111, \"pool\": \"AHL\", \"location\": \"EU/SE/Malmo\" },\n{\"address\": \"relay1.clio.one\", \"port\": 6010, \"pool\": \"CLIO\", \"location\": \"EU/IT/Milan\" },\n{\"address\": \"relay2.clio.one\", \"port\": 6010, \"pool\": \"CLIO\", \"location\": \"EU/IT/Bozlano\" },\n{\"address\": \"relay3.clio.one\", \"port\": 6010, \"pool\": \"CLIO\", \"location\": \"EU/IT/Bozlano\" }\n],\n\"advertise\": false,\n\"trustable\": false,\n\"hotValency\": 5,\n\"warmValency\": 10\n}\n],\n\"publicRoots\": [\n{\n\"accessPoints\": [],\n\"advertise\": false\n}\n],\n\"useLedgerAfterSlot\": 128908821\n}\nSimilarly, a typical topology file on a Core could look something like below:
{\n\"bootstrapPeers\": [],\n\"localRoots\": [\n{\n\"accessPoints\": [\n{\"address\": \"yy.yy.yy.yy\", \"port\": 6000, \"description\": \"Relay1\"},\n{\"address\": \"zz.zz.zz.zz\", \"port\": 6000, \"description\": \"Relay2\"}\n],\n\"advertise\": false,\n\"trustable\": true,\n\"hotValency\": 2\n}\n],\n\"publicRoots\": [\n{\n\"accessPoints\": [],\n\"advertise\": false\n}\n],\n\"useLedgerAfterSlot\": -1\n}\nOnce above two files are updated, since you modified the file manually - there is always a chance of human errors (eg: missing comma/quotes). Thus, we would recommend you to start the node interactively once again before proceeding.
cd \"${CNODE_HOME}\"/scripts\n./cnode.sh\nAs before, ensure you do not have any errors in the console. To stop the node, hit Ctrl-C - we will start the node as systemd later in the document.
"},{"location":"Build/node-cli/#start-the-submit-api","title":"Start the submit-api","text":"Note
An average pool operator may not require cardano-submit-api at all. Please verify if it is required for your use as mentioned here. If - however - you do run submit-api for accepting sizeable transaction load, you would want to override the default MEMPOOL_BYTES by uncommenting it in cnode.sh.
cardano-submit-api is one of the binaries built as part of cardano-node repository and allows you to submit transactions over a Web API. To run this service interactively, you can use the pre-built script below (submitapi.sh). Make sure to update submitapi.sh script to change listen IP or Port that you'd want to make this service available on.
cd $CNODE_HOME/scripts\n./submitapi.sh\nTo stop the process, hit Ctrl-C
"},{"location":"Build/node-cli/#systemd","title":"Run as systemd service","text":"The preferred way to run the node (and submit-api) is through a service manager like systemd. This section explains how to setup a systemd service file.
1. Deploy as a systemd service Execute the below command to deploy your node as a systemd service (from the respective scripts folder):
cd $CNODE_HOME/scripts\n./cnode.sh -d\n# Deploying cnode.service as systemd service..\n# cnode.service deployed successfully!!\n\n./submitapi.sh -d\n# Deploying cnode-submit-api.service as systemd service..\n# cnode-submit-api deployed successfully!!\n2. Start the service Run below commands to enable automatic start of service on startup and start it.
sudo systemctl start cnode.service\nsudo systemctl start cnode-submit-api.service\n3. Check status and stop/start commands Replace status with stop/start/restart depending on what action to take.
sudo systemctl status cnode.service\nsudo systemctl status cnode-submit-api.service\nImportant
In case you see the node exit unsuccessfully upon checking status, please verify you've followed the transition process correctly as documented below, and that you do not have another instance of node already running. It would help to check your system logs, you can also check journalctl -f -u cnode to examine startup attempt for services, and scroll up until you see output for node startup attempt) for any errors while starting node.
You can use gLiveView to monitor your node that was started as a systemd service.
cd $CNODE_HOME/scripts\n./gLiveView.sh\nImportant
In the Cardano multi-asset era, this project helps you create and submit metadata describing your assets, storing them off-chain.
"},{"location":"Build/offchain-metadata-tools/#download-pre-built-binaries","title":"Download pre-built binaries","text":"Go to input-output-hk/offchain-metadata-tools to download the binaries and place in a directory specified by PATH, e.g. $HOME/.local/bin/.
An alternative to pre-built binaries - instructions describe how to build the token-metadata-creator tool but the offchain-metadata-tools repository contains other tools as well. Build the ones needed for your installation.
Execute the below to clone the offchain-metadata-tools repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/input-output-hk/offchain-metadata-tools.git\ncd offchain-metadata-tools/token-metadata-creator\nYou can use the instructions below to build token-metadata-creator, same steps can be executed in future to update the binaries (replacing appropriate tag) as well.
git fetch --tags --all\ngit pull\n# Replace master with appropriate tag if you'd like to avoid compiling against master\ngit checkout master\n$CNODE_HOME/scripts/cabal-build-all.sh\n~/.local/bin folder."},{"location":"Build/offchain-metadata-tools/#verify","title":"Verify","text":"Verify that the tool is executable from anywhere by running:
token-metadata-creator -h\n!> - An average pool operator may not require cardano-wallet at all. Please verify if it is required for your use as mentioned here.
Ensure the Pre-Requisites are in place before you proceed.
"},{"location":"Build/wallet/#build-instructions","title":"Build Instructions","text":"Follow instructions below for building the cardano-wallet binary:
"},{"location":"Build/wallet/#clone-the-repository","title":"Clone the repository","text":"Execute the below to clone the cardano-wallet repository to $HOME/git folder on your system:
cd ~/git\ngit clone https://github.com/cardano-foundation/cardano-wallet\ncd cardano-wallet\nYou can use the instructions below to build the latest release of cardano-wallet.
!> - Note that the latest release of cardano-wallet may not work with the latest release of cardano-node. Please check the compatibility of each cardano-wallet release yourself in the official docs, e.g. https://github.com/cardano-foundation/cardano-wallet/releases/latest.
git fetch --tags --all\ngit pull\n# Replace tag against checkout if you do not want to build the latest released version\ngit checkout $(curl -s https://api.github.com/repos/cardano-foundation/cardano-wallet/releases/latest | jq -r .tag_name)\n$CNODE_HOME/scripts/cabal-build-all.sh\nThe above would copy the binaries into ~/.local/bin folder.
You can run the below to connect to a cardano-node instance that is expected to be already running and the wallet will start syncing.
cardano-wallet serve /\n --node-socket $CNODE_HOME/sockets/node.socket /\n --mainnet / # if using the testnet flag you also need to specify the testnet shelley-genesis.json file\n--database $CNODE_HOME/priv/wallet\ncardano-wallet network information\nOk.\n{\n\"network_tip\": {\n\"time\": \"2021-06-01T17:31:05Z\",\n\"epoch_number\": 269,\n\"absolute_slot_number\": 31002374,\n\"slot_number\": 157574\n},\n\"node_era\": \"mary\",\n\"node_tip\": {\n\"height\": {\n\"quantity\": 5795127,\n\"unit\": \"block\"\n},\n\"time\": \"2021-06-01T17:31:00Z\",\n\"epoch_number\": 269,\n\"absolute_slot_number\": 31002369,\n\"slot_number\": 157569\n},\n\"sync_progress\": {\n\"status\": \"ready\"\n},\n\"next_epoch\": {\n\"epoch_start_time\": \"2021-06-04T21:44:51Z\",\n\"epoch_number\": 270\n}\n}\nIf you're creating a new wallet, you'd first want to generate a mnemonic for use (see below):
cardano-wallet recovery-phrase generate\n# false brother typical saddle settle phrase foster sauce ask sunset firm gate service render burger\ncardano-wallet wallet create from-recovery-phrase MyWalletName\nPlease enter a 15\u201324 word recovery phrase: false brother typical saddle settle phrase foster sauce ask sunset firm gate service render burger\n(Enter a blank line if you do not wish to use a second factor.)\nPlease enter a 9\u201312 word second factor:\nPlease enter a passphrase: **********\nEnter the passphrase a second time: **********\nOk.\n{\n ...\n}\nMithril Networks provide the ability to download and bootstrap cardano nodes via snapshots of the the Cardano blockchain. This is a great way to speed up the process of syncing a new node, especially for stake pool operators. The tools provided by Guild Operators are designed to facilitate the ease of use in setting up and managing the:
The env file contains a new environment variable MITHRIL_DOWNLOAD that when enabled allows the cnode.sh script to automatically download the latest Mithril snapshot if the local db directory is empty. This is useful for new nodes that need to be bootstrapped with the latest snapshot to avoid synchronizing the entire blockchain from scratch. While also providing a high level of trust that the snapshot is valid since it is signed by multiple pool operators.
The architecture for Mithril Networks is described in detail at Mithril network architecture by CF/IOHK. However the architecture suggested and supported by the Guild Operators tools is not identical to the upstream documentation in that we provide a more simplified approach to the setup and management of the Mithril Network components and tools that allow setting up a Squid Mithril relays and an Nginx loadbalancer (aka sidecar) local to the Mithril signer. The Nginx sidecar provides the ability to loadbalance requests to multiple Squid based Mithril Relays running on each of the SPO's Cardano Relay nodes.
"},{"location":"Mithril/mithril-overview/#single-relay-architecture","title":"Single Relay Architecture","text":"For SPO's who only have a single Cardano relay node, an Squid based Mithril relay can be run on the same node as the Cardano relay. This can be used by the Mithril signer to submit the snapshot signatures to the Mithril Aggregator.
"},{"location":"Mithril/mithril-overview/#multi-relay-architecture","title":"Multi Relay Architecture","text":"For SPO's who have multiple Cardano relay nodes, a Nginx relay sidecar can be run on the Block Producer and load balance requests over mutliple Cardano relay nodes, each running its own Nginx Mithril relay to pass the signature along to the Mithril aggregator. This can be used to avoid a single point of failure in case a Relay server is offline for any reason. This provides high availability for the Mithril signer through multiple relays as long as the local Nginx Mithril relay is running on the same server as the Cardano Block Producer node.
"},{"location":"Mithril/mithril-overview/#installation","title":"Installation","text":"The installation of the Mithril tools is automated via guild-deploy.sh. To participate in a Mithril network include the -s m flag which will install the Mithril Client and Mithril Signer release binaries to \"${HOME}\"/.local/bin.
guild-deploy.sh -s m\nThe Mithril client is used to download a snapshot of the Cardano blockchain from a Mithril Aggregator. The snapshot is then used to bootstrap a new Cardano node. The Mithril client can be used to download the latest snapshot, list all available snapshots, or show details of a specific snapshot.
To bootstrap a Cardano node using the Mithril client, follow these steps:
Setup the Cardano Node: Use the guild tools to setup the Cardano node, either by building the binaries or using pre-compiled binaries. Follow the instructions in the guild-operators documentation.
Create the Mithril environment file: Run the script with the environment setup command. This will create a new mithril.env file with all the necessary environment variables for the Mithril client.
./mithril-client.sh environment setup\nsnapshot download command. This snapshot contains the latest state of the Cardano blockchain db from a Mithril Aggregator../mithril-client.sh snapshot download\nThe Mithril signer is used to participate in the creation of stake based signatures of snapshots. The Mithril signer can be used to sign a snapshots. The signed snapshot is then submitted to a Mithril Aggregator, via a Squid based Mithril Relay.
The first step to participate in the Mithril network is to deploy your Squid based Mithril Relays. The Mithril relay is used to provide a private and highly available network for submitting the snapshots to a Mithril Aggregator.
"},{"location":"Mithril/mithril-overview/#deploying-the-squid-mithril-relay","title":"Deploying the Squid Mithril Relay","text":"To deploy your Squid based Mithril Relays with your Cardano relay node, follow these steps:
Deploy the Squid Mithril Relay: Run the mithril-relay.sh script:
Use the -d flag to deploy the Squid Mithril Relay.
./mithril-relay.sh -d\n\nInstalling squid proxy\nEnter the IP address of your Block Producer: 1.2.3.4\nEnter the relays listening port (press Enter to use default 3132):\nUsing port 3132 for relays listening port.\nCreate the appropriate firewall rule: sudo ufw allow from 1.2.3.4 to any port 3132 proto tcp\n sudo systemctl enable --now squid\nDeploy the Mithril Signer: Run the mithril-signer.sh script:
Use the -e flag to update the mithril.env file with the Mithril Signer environment variables.
Optionally provide the relays listening port when prompted to use a port.
./mithril-signer.sh -e\n Enter the IP address of the relay endpoint: 4.5.6.7\n Enter the port of the relay endpoint (press Enter to use default 3132):\n Using RELAY_ENDPOINT=4.5.6.7:3132 for the Mithril signer relay endpoint.\nUse the -d flag to deploy the Mithril Signer.
./mithril-signer.sh -d\n Creating cnode-mithril-signer systemd service environment file..\n Mithril signer service successfully deployed\nEnable the Systemd service to start the Mithril Signer on boot.
sudo systemctl enable cnode-mithril-signer\nDeploy the Nginx sidecar loadbalancer: Run the mithril-relay.sh script:
Use the -l flag to deploy the Nginx Mithril Relay.
Create the appropriate firewall rule to allow traffic from your Block Producer to the Mithril Relay(s).
./mithril-relay.sh -l\n\nInstalling nginx load balancer\nEnter the IP address of a relay: 4.5.6.7\nAre there more relays? (y/n) y\nEnter the IP address of a relay: 8.9.10.11\nAre there more relays? (y/n) n\nEnter the IP address of the load balancer (press Enter to use default 127.0.0.1):\nUsing IP address 127.0.0.1 for the load balancer configuration.\nEnter the relays listening port (press Enter to use default 3132):\nUsing port 3132 for relays listening port.\nStarting Mithril relay sidecar (nginx load balancer)\nEnable the Systemd Nginx Mithril Relay service to start on boot.
sudo systemctl enable --now nginx\nDeploy the Mithril Signer: Run the mithril-signer.sh script:
Use the -e flag to update the mithril.env file with the Mithril Signer environment variables.
Optionally provide the loadbalancer listening port when prompted to use a port.
./mithril-signer.sh -e\n Enter the IP address of the relay endpoint: 127.0.0.1\n Enter the port of the relay endpoint (press Enter to use default 3132):\n Using RELAY_ENDPOINT=127.0.0.1:3132 for the Mithril signer relay endpoint.\nUse the -d flag to deploy the Mithril Signer.
./mithril-signer.sh -d\n Creating cnode-mithril-signer systemd service environment file..\n Mithril signer service successfully deployed\nEnable the Systemd service to start the Mithril Signer on boot.
sudo systemctl enable cnode-mithril-signer\nReminder !!
Ensure the Pre-Requisites are in place before you proceed.
blockPerf.sh is a script to monitor the network propagation of new blocks as seen by the local cardano-node.
Although blockPerf can also run on the block producer, it makes the most sense to run it on the upstream relays. There it waits for each new block announced to the relay over the network by its remote peers.
It looks for the delay times that result
You can view this data locally as a console stream, or run it as a systemd service in background.
BlockPerf also sends this data to the TopologyUpdater server, so that there is a possibility to compare this data (similar to sendtip to pooltool). As a contributing operator you get the possibility to see how your own relays compare to other nodes regarding receive quality, delay times and thus performance.
There is no connection or constraint between the TopologyUpdater Relay subscription and the BlockPerf analysis. BlockPerf is even designed to work outside the cnTools suite.
The results of these data are a good basis to make optimizations and to evaluate which changes were useful or might by required to improve the performance compared to other relay nodes.
"},{"location":"Scripts/blockperf/#installation","title":"Installation","text":"The script is best run as a background process. This can be accomplished in many ways but the preferred method is to run it as a systemd service. A terminal multiplexer like tmux or screen could also be used but not covered here.
"},{"location":"Scripts/blockperf/#run-as-service","title":"Run as service","text":"Use the deploy-as-systemd.sh script to create a systemd unit file. In this setup the script is started in \"service\" mode. Error/Warn level log output is handled by journald. journalctl -f -u cnode-tu-blockperf.service can be used to check service output (follow mode).
Outside the cnTools environment call blockPerf.sh -d to install it as a systemd service.
If you run blockPerf local in the console (scripts/blockPerf.sh) , immediately after the appearance of a new block it shows where it came from, how many slots away from the previous block it was, and how many milliseconds the individual steps took.
Block:.... 6860534\n Slot..... 52833850 (+59s)\n ......... 2022-02-09 09:49:01\n Header... 2022-02-09 09:49:02,780 (+1780 ms)\n Request.. 2022-02-09 09:49:02,780 (+0 ms)\n Block.... 2022-02-09 09:49:02,830 (+50 ms)\n Adopted.. 2022-02-09 09:49:02,900 (+70 ms)\n Size..... 79976 bytes\n delay.... 1.819971868 sec\n From..... 104.xxx.xxx.61:3001\n\nBlock:.... 6860535\n Slot..... 52833857 (+7s)\n ......... 2022-02-09 09:49:08\n Header... 2022-02-09 09:49:08,960 (+960 ms)\n Request.. 2022-02-09 09:49:08,970 (+10 ms)\n Block.... 2022-02-09 09:49:09,020 (+50 ms)\n Adopted.. 2022-02-09 09:49:09,090 (+70 ms)\n Size..... 64950 bytes\n delay.... 1.028341023 sec\n From..... 34.xxx.xxx.15:4001\nA further aim of the blockPerf project is to use the data that individual nodes send to the central TopologyUpdater database to produce graphical visualisations and evaluations that provide the participating node operators with useful insights into their performance compared to all others.
"},{"location":"Scripts/cncli/","title":"CNCLI","text":"Reminder !!
Ensure the Pre-Requisites are in place before you proceed.
cncli.sh is a script to download and deploy CNCLI created and maintained by Andrew Westberg. It's a community-based CLI tool written in RUST for low-level cardano-node communication. Usage is optional and no script is dependent on it. The main features include:
gLiveView for peer analysis if available. sqlite database. firstSlotOfNextEpoch - (3 * k / f)).cncli.sh script's main functions, sync, leaderlog, validate and PoolTool sendslots/sendtip are not meant to be run manually, but instead deployed as systemd services that run in the background to do the block scraping and validation automatically. Additional commands exist for manual execution to initiate the sqlite db, filling the blocklog DB with all blocks created by the pool known to the blockchain, migration of old cntoolsBlockCollector JSON blocklog, and re-validation of blocks and leaderlogs. See usage output below for a complete list of available commands.
The script works in tandem with Log Monitor to provide faster adopted status but mainly to catch slots the node is leader for but are unable to create a block for. These are marked as invalid. Blocklog will however work fine without the logMonitor service and CNCLI is able to handle everything except catching invalid blocks.
guild-deploy.sh with guild-deploy.sh -s c to download and install RUST and CNCLI. IOG fork of libsodium required by CNCLI is automatically compiled by CNCLI build process. If a previous installation is found, RUST and CNCLI will be updated to the latest version.deploy-as-systemd.sh to deploy the systemd services that handle all the work in the background. Six systemd services in total are deployed whereof four are related to CNCLI. See above for the different purposes they serve.If you want to disable some of the deployed services, run sudo systemctl disable <service>
cnode.service (main cardano-node launcher)
cnode-cncli-sync.servicecnode-cncli-leaderlog.servicecnode-cncli-validate.servicecnode-cncli-ptsendtip.servicecnode-cncli-ptsendslots.servicecnode-logmonitor.service (see Log Monitor)You can override the values in the script at the User Variables section shown below. POOL_ID, POOL_VRF_SKEY and POOL_VRF_VKEY should automatically be detected if POOL_NAME is set in the common env file and can be left commented. PT_API_KEY and POOL_TICKER need to be set in the script if PoolTool sendtip/sendslots are to be used before starting the services. For the rest of the commented values, if the defaults do not provide the right values, uncomment and make adjustments.
#POOL_ID=\"\" # Automatically detected if POOL_NAME is set in env. Required for leaderlog calculation & pooltool sendtip, lower-case hex pool id\n#POOL_VRF_SKEY=\"\" # Automatically detected if POOL_NAME is set in env. Required for leaderlog calculation, path to pool's vrf.skey file\n#POOL_VRF_VKEY=\"\" # Automatically detected if POOL_NAME is set in env. Required for block validation, path to pool's vrf.vkey file\n#PT_API_KEY=\"\" # POOLTOOL sendtip: set API key, e.g \"a47811d3-0008-4ecd-9f3e-9c22bdb7c82d\"\n#POOL_TICKER=\"\" # POOLTOOL sendtip: set the pools ticker, e.g. \"TCKR\"\n#PT_HOST=\"127.0.0.1\" # POOLTOOL sendtip: connect to a remote node, preferably block producer (default localhost)\n#PT_PORT=\"${CNODE_PORT}\" # POOLTOOL sendtip: port of node to connect to (default is CNODE_PORT from the env file)\n#CNCLI_DIR=\"${CNODE_HOME}/guild-db/cncli\" # path to the directory for cncli sqlite db\n#SLEEP_RATE=60 # CNCLI leaderlog/validate: time to wait until next check (in seconds)\n#CONFIRM_SLOT_CNT=600 # CNCLI validate: require at least these many slots to have passed before validating\n#CONFIRM_BLOCK_CNT=15 # CNCLI validate: require at least these many blocks on top of minted before validating\n#TIMEOUT_LEDGER_STATE=300 # CNCLI leaderlog: timeout in seconds for ledger-state query\n#BATCH_AUTO_UPDATE=N # Set to Y to automatically update the script if a new version is available without user interaction\nServices are controlled by sudo systemctl <status|start|stop|restart> <service name> All services are configured as child services to cnode.service and as such, when an action is taken against this service it's replicated to all child services. E.g running sudo systemctl start cnode.service will also start all child services.
Log output is handled by journald. journalctl -f -u <service> can be used to check service output (follow mode). Other logging configurations are not covered here.
Recommended workflow to get started with CNCLI blocklog.
$CNODE_HOME/scripts/cncli.sh migrate <path> where is the location to the directory containing all blocks_.json files. sudo systemctl start cnode-cncli-sync.service (starts leaderlog, validate & ptsendslots automatically)sudo systemctl start cnode-logmonitor.servicesudo systemctl start cnode-cncli-ptsendtip.service (optional but recommended)sudo systemctl restart cnode.service$CNODE_HOME/scripts/cncli.sh initUsage: cncli.sh [operation <sub arg>]\nScript to run CNCLI, best launched through systemd deployed by 'deploy-as-systemd.sh'\n\nsync Start CNCLI chainsync process that connects to cardano-node to sync blocks stored in SQLite DB (deployed as service)\nleaderlog One-time leader schedule calculation for current epoch, then continuously monitors and calculates schedule for coming epochs, 1.5 days before epoch boundary on the mainnet (deployed as service)\n force Manually force leaderlog calculation and overwrite even if already done, exits after leaderlog is calculated\nvalidate Continuously monitor and confirm that the blocks made actually was accepted and adopted by chain (deployed as service)\n all One-time re-validation of all blocks in blocklog db\n epoch One-time re-validation of blocks in blocklog db for the specified epoch \nptsendtip Send node tip to PoolTool for network analysis and to show that your node is alive and well with a green badge (deployed as service)\nptsendslots Securely sends PoolTool the number of slots you have assigned for an epoch and validates the correctness of your past epochs (deployed as service)\ninit One-time initialization adding all minted and confirmed blocks to blocklog\nmigrate One-time migration from old blocklog (cntoolsBlockCollector) to new format (post cncli)\n path Path to the old cntoolsBlockCollector blocklog folder holding json files with blocks created\nBest and easiest viewed in CNTools and gLiveView but the blocklog database is a SQLite DB so if you are comfortable with SQL, the sqlite3 command can be used to query the DB.
Block status
- Leader : Scheduled to make block at this slot\n- Ideal : Expected/Ideal number of blocks assigned based on active stake (sigma)\n- Luck : Leader slots assigned vs ideal slots for this epoch\n- Adopted : Block created successfully\n- Confirmed : Block created validated to be on-chain with the certainty set in `cncli.sh` for `CONFIRM_BLOCK_CNT`\n- Missed : Scheduled at slot but no record of it in CNCLI DB and no other pool has made a block for this slot\n- Ghosted : Block created but marked as orphaned and no other pool has made a valid block for this slot -> height battle or block propagation issue\n- Stolen : Another pool has a valid block registered on-chain for the same slot\n- Invalid : Pool failed to create block, base64 encoded error message can be decoded with `echo <base64 hash> | base64 -d | jq -r`\nOpen CNTools and select [b] Blocks to open the block viewer. Either select Epoch and enter the epoch you want to see a detailed view for or choose Summary to display blocks for last x epochs.
If the node was elected to create blocks in the selected epoch it could look something like this:
Summary >> BLOCKS\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nCurrent epoch: 96\n\n+--------+---------------------------+----------------------+--------------------------------------+\n| Epoch | Leader | Ideal | Luck | Adopted | Confirmed | Missed | Ghosted | Stolen | Invalid |\n+--------+---------------------------+----------------------+--------------------------------------+\n| 96 | 34 | 31.66 | 107.39% | 18 | 18 | 0 | 0 | 0 | 0 |\n| 95 | 32 | 30.57 | 104.68% | 32 | 32 | 0 | 0 | 0 | 0 |\n+--------+---------------------------+----------------------+--------------------------------------+\n\n[h] Home | [b] Block View | [i] Info | [*] Refresh\n >> BLOCKS\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nCurrent epoch: 96\n\n+---------------------------+----------------------+--------------------------------------+\n| Leader | Ideal | Luck | Adopted | Confirmed | Missed | Ghosted | Stolen | Invalid |\n+---------------------------+----------------------+--------------------------------------+\n| 34 | 31.66 | 107.39% | 18 | 18 | 0 | 0 | 0 | 0 |\n+---------------------------+----------------------+--------------------------------------+\n\n+-----+------------+----------+---------------------+--------------------------+-------+-------------------------------------------------------------------+\n| # | Status | Block | Slot | SlotInEpoch | Scheduled At | Size | Hash |\n+-----+------------+----------+---------------------+--------------------------+-------+-------------------------------------------------------------------+\n| 1 | confirmed | 2043444 | 11142827 | 40427 | 2020-11-16 08:34:03 CET | 3 | ec216d3fb01e4a3cc3e85305145a31875d9561fa3bbcc6d0ee8297236dbb4115 |\n| 2 | confirmed | 2044321 | 11165082 | 62682 | 2020-11-16 14:44:58 CET | 3 | b75c33a5bbe49a74e4b4cc5df4474398bfb10ed39531fc65ec2acc51f89ddce5 |\n| 3 | confirmed | 2044397 | 11166970 | 64570 | 2020-11-16 15:16:26 CET | 3 | c1ea37fd72543779b6dab46e3e29e0e422784b5fd6188f828ace9eabcc87088f |\n| 4 | confirmed | 2044879 | 11178909 | 76509 | 2020-11-16 18:35:25 CET | 3 | 35a116cec80c5dc295415e4fc8e6435c562b14a5d6833027006c988706c60307 |\n| 5 | confirmed | 2046965 | 11232557 | 130157 | 2020-11-17 09:29:33 CET | 3 | d566e5a1f6a3d78811acab4ae3bdcee6aa42717364f9afecd6cac5093559f466 |\n| 6 | confirmed | 2047101 | 11235675 | 133275 | 2020-11-17 10:21:31 CET | 3 | 3a638e01f70ea1c4a660fe4e6333272e6c61b11cf84dc8a5a107b414d1e057eb |\n| 7 | confirmed | 2047221 | 11238453 | 136053 | 2020-11-17 11:07:49 CET | 3 | 843336f132961b94276603707751cdb9a1c2528b97100819ce47bc317af0a2d6 |\n| 8 | confirmed | 2048692 | 11273507 | 171107 | 2020-11-17 20:52:03 CET | 3 | 9b3eb79fe07e8ebae163870c21ba30460e689b23768d2e5f8e7118c572c4df36 |\n| 9 | confirmed | 2049058 | 11282619 | 180219 | 2020-11-17 23:23:55 CET | 3 | 643396ea9a1a2b6c66bb83bdc589fa19c8ae728d1f1181aab82e8dfe508d430a |\n| 10 | confirmed | 2049321 | 11289237 | 186837 | 2020-11-18 01:14:13 CET | 3 | d93d305a955f40b2298247d44e4bc27fe9e3d1486ef3ef3e73b235b25247ccd7 |\n| 11 | confirmed | 2049747 | 11299205 | 196805 | 2020-11-18 04:00:21 CET | 3 | 19a43deb5014b14760c3e564b41027c5ee50e0a252abddbfcac90c8f56dc0245 |\n| 12 | confirmed | 2050415 | 11316075 | 213675 | 2020-11-18 08:41:31 CET | 3 | dd2cb47653f3bfb3ccc8ffe76906e07d96f1384bafd57a872ddbab3b352403e3 |\n| 13 | confirmed | 2050505 | 11318274 | 215874 | 2020-11-18 09:18:10 CET | 3 | deb834bc42360f8d39eefc5856bb6d7cabb6b04170c842dcbe7e9efdf9dbd2e1 |\n| 14 | confirmed | 2050613 | 11320754 | 218354 | 2020-11-18 09:59:30 CET | 3 | bf094f6fde8e8c29f568a253201e4b92b078e9a1cad60706285e236a91ec95ff |\n| 15 | confirmed | 2050807 | 11325239 | 222839 | 2020-11-18 11:14:15 CET | 3 | 21f904346ba0fd2bb41afaae7d35977cb929d1d9727887f541782576fc6a62c9 |\n| 16 | confirmed | 2050997 | 11330062 | 227662 | 2020-11-18 12:34:38 CET | 3 | 109799d686fe3cad13b156a2d446a544fde2bf5d0e8f157f688f1dc30f35e912 |\n| 17 | confirmed | 2051286 | 11336791 | 234391 | 2020-11-18 14:26:47 CET | 3 | bb1beca7a1d849059110e3d7dc49ecf07b47970af2294fe73555ddfefb9561a8 |\n| 18 | confirmed | 2051734 | 11348498 | 246098 | 2020-11-18 17:41:54 CET | 3 | 87940b53c2342999c1ba4e185038cda3d8382891a16878a865f5114f540683de |\n| 19 | leader | - | 11382001 | 279601 | 2020-11-19 03:00:17 CET | - | - |\n| 20 | leader | - | 11419959 | 317559 | 2020-11-19 13:32:55 CET | - | - |\n| 21 | leader | - | 11433174 | 330774 | 2020-11-19 17:13:10 CET | - | - |\n| 22 | leader | - | 11434241 | 331841 | 2020-11-19 17:30:57 CET | - | - |\n| 23 | leader | - | 11435289 | 332889 | 2020-11-19 17:48:25 CET | - | - |\n| 24 | leader | - | 11440314 | 337914 | 2020-11-19 19:12:10 CET | - | - |\n| 25 | leader | - | 11442361 | 339961 | 2020-11-19 19:46:17 CET | - | - |\n| 26 | leader | - | 11443861 | 341461 | 2020-11-19 20:11:17 CET | - | - |\n| 27 | leader | - | 11446997 | 344597 | 2020-11-19 21:03:33 CET | - | - |\n| 28 | leader | - | 11453110 | 350710 | 2020-11-19 22:45:26 CET | - | - |\n| 29 | leader | - | 11455323 | 352923 | 2020-11-19 23:22:19 CET | - | - |\n| 30 | leader | - | 11505987 | 403587 | 2020-11-20 13:26:43 CET | - | - |\n| 31 | leader | - | 11514983 | 412583 | 2020-11-20 15:56:39 CET | - | - |\n| 32 | leader | - | 11516010 | 413610 | 2020-11-20 16:13:46 CET | - | - |\n| 33 | leader | - | 11518958 | 416558 | 2020-11-20 17:02:54 CET | - | - |\n| 34 | leader | - | 11533254 | 430854 | 2020-11-20 21:01:10 CET | - | - |\n+-----+------------+----------+---------------------+--------------------------+-------+-------------------------------------------------------------------+\nCurrently shows a block summary for current epoch. For full block details use CNTools for now. Invalid, missing, ghosted and stolen blocks only shown in case of a non-zero value.
\u2502--------------------------------------------------------------\u2502\n\u2502 BLOCKS Leader | Ideal | Luck | Adopted | Confirmed \u2502\n\u2502 24 27.42 87.53% 1 1 \u2502\n\u2502 08:07:57 until leader XXXXXXXXX.....................\u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nAll notable changes to this tool will be documented in this file.
Whenever you're updating between versions where format/hash of keys have changed , or you're changing networks - it is recommended to Backup your Wallet and Pool folders before you proceed with launching cntools on a fresh network.
The format is based on Keep a Changelog, and this adheres to Semantic Versioning.
"},{"location":"Scripts/cntools-changelog/#1331-2024-11-25","title":"[13.3.1] - 2024-11-25","text":""},{"location":"Scripts/cntools-changelog/#fixed","title":"Fixed","text":"light mode support for all governance functions.test_koios call from cntools.library to cntools.shdialog by default, it is an optional component - and no longer installed by default.--whole-utxo flag, as it returns all address and will not accept --address--whole-utxo flag when query UTxO, as required by cardano-cli 1.28, to keep behaviour same as before.AdvancedThough mostly unchanged in the user interface, this is a major update with most of the code re-written/touched in the back-end. Only the most noticeable changes added to changelog.
"},{"location":"Scripts/cntools-changelog/#added_15","title":"Added","text":"--cold-verification-key-file instead of --verification-key-fileThis is a major release with a lot of changes. It is highly recommended that you familiarise yourself with the usage for Hybrid or Online v/s Offline mode on a testnet environment before doing it on production. Please visit https://cardano-community.github.io/guild-operators/upgrade for details.
"},{"location":"Scripts/cntools-changelog/#added_18","title":"Added","text":"cardano-address and bech32 in yout $PATH to use this feature (available if you rebuild cardano-node using updated cabal-build-all.sh), reusing guide from @ilap.srm) when available when deleting files.,) in user input for sending ADA and pledge/cost at pool registration to make it easier to count the zeroscardano-node 1.19.0, please upgrade if you're not using this version.Pool >> Show now moved to its own menu option This is to de-clutter and because it takes time to parse this data from ledger-statePool >> Delegators removed.pool >> show stake distribution showing up as always 0.prereqs.sh -t) fix for internal update--output-format hex when extracting pool ID in hex formatWallet >> Encrypt as these are re-generated from keys and need to be writableFunds >> Withdraw for base address as this is used to pay the withdraw transaction feePool >> Show delegator rewards parsing from ledger-statemainnet_candidate, and add second argument (g) to run prereqs against guild network[c] to [Esc]Wallet >> Show2.1.1 included a change to env file and thus require a major version bump.Pool >> ShowPool >> Show(stake + reward) is below pledge (single-owner only for now)Pool >> ShowPool >> New to Pool >> Register.Wallet >> List Not a registered wallet on chain information from Wallet listingPool >> ShowImportant
Familiarize yourself with the Online workflow of creating wallets and pools on the Preview/Preprod/Guild network first. You can then move on to test the Offline Workflow. The Offline workflow means that the private keys never touch the Online node. When comfortable with both the online and offline CNTools workflow, it's time to deploy what you learnt on the mainnet.
This chapter describes some common use-cases for wallet and pool creation when running CNTools in Online mode. CNTools contains much more functionality not described here.
Create WalletA wallet is needed for pledge and to pay for pool registration fee.
[w] Wallet and you will be presented with the following menu: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Wallet Management\n\n ) New - create a new wallet\n ) Import - import a Daedalus/Yoroi 24/25 mnemonic or Ledger/Trezor HW wallet\n ) Register - register a wallet on chain\n ) De-Register - De-Register (retire) a registered wallet\n ) List - list all available wallets in a compact view\n ) Show - show detailed view of a specific wallet\n ) Remove - remove a wallet\n ) Decrypt - remove write protection and decrypt wallet\n ) Encrypt - encrypt wallet keys and make all files immutable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Select Wallet Operation\n\n [n] New\n [i] Import\n [r] Register\n [z] De-Register\n [l] List\n [s] Show\n [x] Remove\n [d] Decrypt\n [e] Encrypt\n [h] Home\n[n] New to create a new wallet. [i] Import can also be used to import a Daedalus/Yoroi based 15 or 24 word wallet seed ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET >> NEW\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nName of new wallet: Test\n\nNew Wallet : Test\nAddress : addr_test1qpq5qjr774cyc6kxcwp060k4t4hwp42q43v35lmcg3gcycu5uwdwld5yr8m8fgn7su955zf5qahtrgljqfjfa4nr8jfsj4alxk\nEnterprise Address : addr_test1vpq5qjr774cyc6kxcwp060k4t4hwp42q43v35lmcg3gcyccuxhdka\n\nYou can now send and receive Ada using the above addresses.\nNote that Enterprise Address will not take part in staking.\nWallet will be automatically registered on chain if you\nchoose to delegate or pledge wallet when registering a stake pool.\nThe Import feature of CNTools is originally based on this guide from Ilap.
If you would like to use Import function to import a Daedalus/Yoroi based 15 or 24 word wallet seed, please ensure that cardano-address and bech32 bineries are available in your $PATH environment variable:
bech32 --version\n1.1.0\n\ncardano-address --version\n3.5.0\nIf the version is not as per above, please run the latest guild-deploy.sh from here and rebuild cardano-node as instructed here.
To import a Daedalus/Yoroi wallet to CNTools, open CNTools and select the [w] Wallet option, and then select the [i] Import, the following menu will appear:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET >> IMPORT\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Wallet Import\n\n ) Mnemonic - Daedalus/Yoroi 24 or 25 word mnemonic\n ) HW Wallet - Ledger/Trezor hardware wallet\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Select Wallet operation\n\n [m] Mnemonic\n [w] HW Wallet\n [h] Home\nNote
You can import Hardware wallet using [w] HW Wallet above, but please note that before you are able to use hardware wallet in CNTools, you need to ensure you can detect your hardware device at OS level using cardano-hw-cli
Select the wallet you want to import, for Daedalus / Yoroi wallets select [m] Mnemonic:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> WALLET >> IMPORT >> MNEMONIC\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nName of imported wallet: TEST\n\n24 or 15 word mnemonic(space separated):\nCreate the necessary pool keys.
[p] Pool ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Pool Management\n\n ) New - create a new pool\n ) Register - register created pool on chain using a stake wallet (pledge wallet)\n ) Modify - change pool parameters and register updated pool values on chain\n ) Retire - de-register stake pool from chain in specified epoch\n ) List - a compact list view of available local pools\n ) Show - detailed view of specified pool\n ) Rotate - rotate pool KES keys\n ) Decrypt - remove write protection and decrypt pool\n ) Encrypt - encrypt pool cold keys and make all files immutable\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Select Pool Operation\n\n [n] New\n [r] Register\n [m] Modify\n [x] Retire\n [l] List\n [s] Show\n [o] Rotate\n [d] Decrypt\n [e] Encrypt\n [h] Home\n[n] New to create a new pool ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL >> NEW\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nPool Name: TEST\n\nPool: TEST\nID (hex) : 8d5a3510f18ce241115da38a1b2419ed82d308599c16e98caea1b4c0\nID (bech32) : pool134dr2y833n3yzy2a5w9pkfqeakpdxzzenstwnr9w5x6vqtnclue\nRegister the pool on-chain.
[p] Pool [r] Register Make sure you set your pledge low enough to insure your funds in your wallet will cover pledge plus pool registration fees.
Test in our case. As this is a newly created wallet, you will be prompted to continue with wallet registration. When complete and if successful, both wallet and pool will be registered on-chain.It will look something like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL >> REGISTER\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nOnline mode - The default mode to use if all keys are available\n\nHybrid mode - 1) Go through the steps to build a transaction file\n 2) Copy the built tx file to an offline node\n 3) Sign it using 'Sign Tx' with keys on offline node\n (CNTools started in offline mode '-o' without node connection)\n 4) Copy the signed tx file back to the online node and submit using 'Submit Tx'\n\nSelected value: [o] Online\n\n# Select pool\nSelected pool: TEST\n\n# Pool Parameters\npress enter to use default value\n\nPledge (in Ada, default: 50,000):\nMargin (in %, default: 7.5):\nCost (in Ada, minimum: 340, default: 340):\n\n# Pool Metadata\n\nEnter Pool's JSON URL to host metadata file - URL length should be less than 64 chars (default: https://foo.bat/poolmeta.json):\n\nEnter Pool's Name (default: TEST):\nEnter Pool's Ticker , should be between 3-5 characters (default: TEST):\nEnter Pool's Description (default: No Description):\nEnter Pool's Homepage (default: https://foo.com):\n\nOptionally set an extended metadata URL?\nSelected value: [n] No\n{\n \"name\": \"TEST\",\n \"ticker\": \"TEST\",\n \"description\": \"No Description\",\n \"homepage\": \"https://foo.com\",\n \"nonce\": \"1613146429\"\n}\n\nPlease host file /opt/cardano/guild/priv/pool/TEST/poolmeta.json as-is at https://foo.bat/poolmeta.json\n\n# Pool Relay Registration\nSelected value: [d] A or AAAA DNS record (single)\nEnter relays's DNS record, only A or AAAA DNS records: relay.foo.com\nEnter relays's port: 6000\nAdd more relay entries?\nSelected value: [n] No\n\n# Select main owner/pledge wallet (normal CLI wallet)\nSelected wallet: Test (100,000.000000 Ada)\nWallet Test3 not registered on chain\n\nWaiting for new block to be created (timeout = 600 slots, 600s)\nINFO: press any key to cancel and return (won't stop transaction)\n\nOwner #1 : Test added!\n\nRegister a multi-owner pool (you need to have stake.vkey of any additional owner in a seperate wallet folder under $CNODE_HOME/priv/wallet)?\nSelected value: [n] No\n\nUse a separate rewards wallet from main owner?\nSelected value: [n] No\n\nWaiting for new block to be created (timeout = 600 slots, 600s)\nINFO: press any key to cancel and return (won't stop transaction)\n\nPool TEST successfully registered!\nOwner #1 : Test\nReward Wallet : Test\nPledge : 50,000 Ada\nMargin : 7.5 %\nCost : 340 Ada\n\nUncomment and set value for POOL_NAME in ./env with 'TEST'\n\nINFO: Total balance in 1 owner/pledge wallet(s) are: 99,497.996518 Ada\nPOOL_NAME in ./env with 'TEST' (in our case, the POOL_NAME is TEST). The cnode.sh script will automatically detect whether the files required to run as a block producing node are present in the $CNODE_HOME/priv/pool/<POOL_NAME> directory.The node runs with an operational certificate, generated using the KES hot key. For security reasons, the protocol asks to re-generate (or rotate) your KES key once reaching expiry. On mainnet, this expiry is in 62 cycles of 18 hours (thus, to ask for rotation quarterly), after which your node will not be able to forge valid blocks unless rotated. To be able to rotate KES keys, your cold keys files (cold.skey, cold.vkey and cold.counter) need to be present on the machine where you run CNTools to rotate your KES key.
To Rotate KES keys and generate the operational certificate - op.cert.
From the main menu select [p] Pool
[o] Rotate The output should look like:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> POOL >> ROTATE KES\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSelect pool to rotate KES keys on\nSelected pool: TEST\n\nPool KES keys successfully updated\nNew KES start period : 240\nKES keys will expire : 302 - 2021-09-04 11:24:31 UTC\n\nRestart your pool node for changes to take effect\n\npress any key to return to home menu\ncardano-node. If deployed as a systemd service as shown here, you can run sudo systemctl restart cnode.You can use gLiveView - the output at the top should say > Cardano Node - (Core - Guild).
Alternatively, you can check the node logs in $CNODE_HOME/logs/ to see whether the node is performing leadership checks (TraceStartLeadershipCheck, TraceNodeIsNotLeader, etc.)
Important
Koios CNTools is like a swiss army knife for pool operators to simplify typical operations regarding their wallet keys and pool management. Please note that this tool only aims to simplify usual tasks for its users, but it should NOT act as an excuse to skip understanding how to manually work through things or basics of Linux operations. The skills highlighted on the home page are paramount for a stake pool operator, and so is the understanding of configuration files and network. Please ensure you've read and understood the disclaimers before proceeding.
Visit the Changelog section to see progress and current release.
"},{"location":"Scripts/cntools/#overview","title":"Overview","text":"The tool consist of three files.
cntools.sh - the main script to launch cntools.cntools.library - internal script with helper functions.In addition to the above files, there is also a dependency on the common env file. CNTools connects to your node through the configuration in the env file located in the same directory as the script. Customize env and cntools.sh files to your needs.
Additionally, CNTools can integrate and enable optional functionalities based on external components:
cncli.sh is a companion script with optional functionalities to run on the core node (block producer) such as monitoring created blocks, calculating leader schedules and block validation.logMonitor.sh is another companion script meant to be run together with the cncli.sh script to give a more complete picture.See CNCLI and Log Monitor sections for more details.
Koios CNTools can operate in following modes:
-n - The local node is used to query the blockchain for needed data. This is the default mode when you start CNTools without parameters.-l - Koios query layer is used and removes the need for a local node deployment. This mode is both quicker and lighter on resources but comes with a third party dependency.-o - Launches CNTools with a limited set of features. This mode does not require access to cardano-node or access to an internet connection. It is mainly used to create Wallet/Pool and access Transaction >> Sign to sign an offline transaction file created in Hybrid mode.-a - Exposes a new Advanced menu, which allows users to manage (create/mint/burn) new assets.In addition to above mentioned runtime arguments to launch CNTools in different modes, it can also be persisted by editing User Variables section within cntools.sh script.
The update functionality is provided from within CNTools. In case of breaking changes, please follow the prompts post-upgrade. If stuck, it's always best to re-run the latest guild-deploy.sh before proceeding.
If you have not updated in a while, it is possible that you might come from a release with breaking changes. If so, please be sure to check out the upgrade instructions.
"},{"location":"Scripts/cntools/#navigation","title":"Navigation","text":"The scripts menu supports both arrow key navigation and shortcut key selection. The character within the square brackets is the shortcut to press for quick navigation. For other selections like wallet and pool menu that don't contain shortcuts, there is a third way to navigate. Key pressed is compared to the first character of the menu option and if there is a match the selection jumps to this location. A handy way to quickly navigate a large menu.
"},{"location":"Scripts/cntools/#hardware-wallet","title":"Hardware Wallet","text":"CNTools includes hardware wallet support since version 7.0.0 through Vacuumlabs cardano-hw-cli application. Initialize and update firmware/app on the device to the latest version before usage following the manufacturer instructions.
To enable hardware support run guild-deploy.sh -s w. This downloads and installs Vacuumlabs cardano-hw-cli including udev configuration. When a new version of Vacuumlabs cardano-hw-cli is released, run guild-deploy.sh -s w again to update. For additional runtime options, run guild-deploy.sh -h.
Trezor Bridge for your system before trying to use your Trezor device in CNTools. You can find the latest version of the bridge at https://wallet.trezor.io/#/bridgeCNTools can be run in online and offline mode. At a very high level, for working with offline devices, remember that you need to use CNTools in an online node to generate a staging transaction for the desired type of transaction, and then move the staging transaction to an offline node to sign (authorize) using the signing keys on your offline node - and then bring back the signed transaction to the online node for submission to the chain.
For the offline workflow, all the wallet and pool keys should be kept on the offline node. The backup function in CNTools has an option to create a backup without private keys (sensitive signing keys) to be transferred to online node. All other files are included in the backup to be transferred to the online node.
Keys excluded from backup when created without private keys: Wallet - payment.skey, stake.skey Pool - cold.skey
Note that setting up an offline server requires good SysOps background (you need to be aware of how to set up your server with offline mirror repository, how to transfer files across and be fairly familiar with the disk layout presented in the documentation). The guild-deploy.sh in its current state is not expected to run on an offline machine. Essentially, you simply need the cardano-cli, bech32, cardano-address binaries in your $PATH, OS level dependency packages [jq, coreutils, pkgconfig, gcc-c++ and bc ], and perhaps a copy from your online cnode directory (to ensure you have the right genesis/config files on your offline server). We strongly recommend you to familiarise yourself with the workflow on the preview / preprod / guild networks first, before attempting on mainnet.
Example workflow for creating a wallet and pool:
sequenceDiagram Note over Offline: Create/Import a wallet Note over Offline: Create a new pool Note over Offline: Rotate KES keys to generate op.cert Note over Offline: Create a backup w/o private keys Offline->>Online: Transfer backup to online node Note over Online: Fund the wallet base address with enough Ada Note over Online: Register wallet using ' Wallet \u00bb Register ' in hybrid mode Online->>Offline: Transfer built tx file back to offline node Note over Offline: Use ' Transaction >> Sign ' with payment.skey from wallet to sign transaction Offline->>Online: Transfer signed tx back to online node Note over Online: Use ' Transaction >> Submit ' to send signed transaction to blockchain Note over Online: Register pool in hybrid mode loop Offline-->Online: Repeat steps to sign and submit built pool registration transaction end Note over Online: Verify that pool was successfully registered with ' Pool \u00bb Show ' Online modeTo start CNTools in Online (advanced) Mode, execute the script from the $CNODE_HOME/scripts/ directory:
cd $CNODE_HOME/scripts\n./cntools.sh -a\nYou should get a screen that looks something like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> Koios CNTools vX.X.X - Guild - CONNECTED <<\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Main Menu Telegram Announcement / Support channel: t.me/CardanoKoios/9759\n\n ) Wallet - create, show, remove and protect wallets\n ) Funds - send, withdraw and delegate\n ) Pool - pool creation and management\n ) Transaction - Sign and Submit a cold transaction (hybrid/offline mode)\n ) Blocks - show core node leader schedule & block production statistics\n ) Backup - backup & restore of wallet/pool/config\n ) Advanced - Developer and advanced features: metadata, multi-assets, ...\n ) Refresh - reload home screen content\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Epoch 276 - 3d 19:08:27 until next\n What would you like to do? Node Sync: 12 :)\n\n [w] Wallet\n [f] Funds\n [p] Pool\n [t] Transaction\n [b] Blocks\n [u] Update\n [z] Backup & Restore\n [a] Advanced\n [r] Refresh\n [q] Quit\nTo start CNTools in Offline Mode, execute the script from the $CNODE_HOME/scripts/ directory using the -o flag:
cd $CNODE_HOME/scripts\n./cntools.sh -o\nThe main menu header should let you know that node is started in offline mode:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n >> Koios CNTools vX.X.X - Guild - OFFLINE <<\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Main Menu Telegram Announcement / Support channel: t.me/CardanoKoios/9759\n\n ) Wallet - create, show, remove and protect wallets\n ) Funds - send, withdraw and delegate\n ) Pool - pool creation and management\n ) Transaction - Sign and Submit a cold transaction (hybrid/offline mode)\n\n ) Backup - backup & restore of wallet/pool/config\n\n ) Refresh - reload home screen content\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n Epoch 276 - 3d 19:03:46 until next\n What would you like to do?\n\n [w] Wallet\n [f] Funds\n [p] Pool\n [t] Transaction\n [z] Backup & Restore\n [r] Refresh\n [q] Quit\nA common environment file called env is sourced by most scripts in the Guild Operators repository. This file holds common variables and functions needed by other scripts. There are several benefits to this, not having to specify duplicate settings and being able to reuse functions decreasing the risk of misconfiguration and inconsistency.
env file is downloaded together with the rest of the scripts when Pre-Requisites if followed and located in the $CNODE_HOME/scripts/ directory. The file is also automatically downloaded/updated by some of the individual scripts if missing, like cntools.sh, gLiveView.sh and topologyUpdater.sh. All custom changes in User Variables section are untouched on updates unless a forced overwrite is selected when running guild-deploy.sh.
Most variables can be left commented to use the automatically detected or default value. But there are some that need to be set as explained below.
CNODE_PORT - This is the most important variable and needs to be set. Used when launching the node through cnode.sh and to identify the correct process of the node.CNODE_HOME - The root directory of the Cardano node holding all the files needed. Can be left commented if guild-deploy.sh has been run as this variable is then exported and added as a system environment variable.POOL_NAME - If the node is to be started as a block producer by cnode.sh this variable needs to be uncommented and set. This is the name given to the pool in CNTools (not ticker), i.e. the pool directory name under $CNODE_HOME/priv/pool/<POOL_NAME>Take your time and look through the different variables and their explanations and decide if you need/want to change the default setting. For a default deployment using guild-deploy.sh, the CNODE_PORT (all installs) and POOL_NAME (only block producer) should be the only variables needed to be set.
######################################\n# User Variables - Change as desired #\n# Leave as is if unsure #\n######################################\n\n#CCLI=\"${HOME}/.local/bin/cardano-cli\" # Override automatic detection of path to cardano-cli executable\n#CNCLI=\"${HOME}/.local/bin/cncli\" # Override automatic detection of path to cncli executable (https://github.com/AndrewWestberg/cncli)\n#CNODE_HOME=\"/opt/cardano/cnode\" # Override default CNODE_HOME path (defaults to /opt/cardano/cnode)\nCNODE_PORT=6000 # Set node port\n#CONFIG=\"${CNODE_HOME}/files/config.json\" # Override automatic detection of node config path\n#SOCKET=\"${CNODE_HOME}/sockets/node.socket\" # Override automatic detection of path to socket\n#TOPOLOGY=\"${CNODE_HOME}/files/topology.json\" # Override default topology.json path\n#LOG_DIR=\"${CNODE_HOME}/logs\" # Folder where your logs will be sent to (must pre-exist)\n#DB_DIR=\"${CNODE_HOME}/db\" # Folder to store the cardano-node blockchain db\n#UPDATE_CHECK=\"Y\" # Check for updates to scripts, it will still be prompted before proceeding (Y|N).\n#TMP_DIR=\"/tmp/cnode\" # Folder to hold temporary files in the various scripts, each script might create additional subfolders\n#EKG_HOST=127.0.0.1 # Set node EKG host IP\n#EKG_PORT=12788 # Override automatic detection of node EKG port\n#PROM_HOST=127.0.0.1 # Set node Prometheus host IP\n#PROM_PORT=12798 # Override automatic detection of node Prometheus port\n#EKG_TIMEOUT=3 # Maximum time in seconds that you allow EKG request to take before aborting (node metrics)\n#CURL_TIMEOUT=10 # Maximum time in seconds that you allow curl file download to take before aborting (GitHub update process)\n#BLOCKLOG_DIR=\"${CNODE_HOME}/guild-db/blocklog\" # Override default directory used to store block data for core node\n#BLOCKLOG_TZ=\"UTC\" # TimeZone to use when displaying blocklog - https://en.wikipedia.org/wiki/List_of_tz_database_time_zones\n#SHELLEY_TRANS_EPOCH=208 # Override automatic detection of shelley epoch start, e.g 208 for mainnet\n#TG_BOT_TOKEN=\"\" # Uncomment and set to enable telegramSend function. To create your own BOT-token and Chat-Id follow guide at:\n#TG_CHAT_ID=\"\" # https://cardano-community.github.io/guild-operators/Scripts/sendalerts\n#USE_EKG=\"N\" # Use EKG metrics from the node instead of Promethus. Promethus metrics(default) should yield slightly better performance\n#TIMEOUT_LEDGER_STATE=300 # Timeout in seconds for querying and dumping ledger-state\n#IP_VERSION=4 # The IP version to use for push and fetch, valid options: 4 | 6 | mix (Default: 4)\n\n#WALLET_FOLDER=\"${CNODE_HOME}/priv/wallet\" # Root folder for Wallets\n#POOL_FOLDER=\"${CNODE_HOME}/priv/pool\" # Root folder for Pools\n# Each wallet and pool has a friendly name and subfolder containing all related keys, certificates, ...\n#POOL_NAME=\"\" # Set the pool's name to run node as a core node (the name, NOT the ticker, ie folder name)\n\n#WALLET_PAY_VK_FILENAME=\"payment.vkey\" # Standardized names for all wallet related files\n#WALLET_PAY_SK_FILENAME=\"payment.skey\"\n#WALLET_HW_PAY_SK_FILENAME=\"payment.hwsfile\"\n#WALLET_PAY_ADDR_FILENAME=\"payment.addr\"\n#WALLET_BASE_ADDR_FILENAME=\"base.addr\"\n#WALLET_STAKE_VK_FILENAME=\"stake.vkey\"\n#WALLET_STAKE_SK_FILENAME=\"stake.skey\"\n#WALLET_HW_STAKE_SK_FILENAME=\"stake.hwsfile\"\n#WALLET_STAKE_ADDR_FILENAME=\"reward.addr\"\n#WALLET_STAKE_CERT_FILENAME=\"stake.cert\"\n#WALLET_STAKE_DEREG_FILENAME=\"stake.dereg\"\n#WALLET_DELEGCERT_FILENAME=\"delegation.cert\"\n\n#POOL_ID_FILENAME=\"pool.id\" # Standardized names for all pool related files\n#POOL_HOTKEY_VK_FILENAME=\"hot.vkey\"\n#POOL_HOTKEY_SK_FILENAME=\"hot.skey\"\n#POOL_COLDKEY_VK_FILENAME=\"cold.vkey\"\n#POOL_COLDKEY_SK_FILENAME=\"cold.skey\"\n#POOL_OPCERT_COUNTER_FILENAME=\"cold.counter\"\n#POOL_OPCERT_FILENAME=\"op.cert\"\n#POOL_VRF_VK_FILENAME=\"vrf.vkey\"\n#POOL_VRF_SK_FILENAME=\"vrf.skey\"\n#POOL_CONFIG_FILENAME=\"pool.config\"\n#POOL_REGCERT_FILENAME=\"pool.cert\"\n#POOL_CURRENT_KES_START=\"kes.start\"\n#POOL_DEREGCERT_FILENAME=\"pool.dereg\"\n\n#ASSET_FOLDER=\"${CNODE_HOME}/priv/asset\" # Root folder for Multi-Assets containing minted assets and subfolders for Policy IDs\n#ASSET_POLICY_VK_FILENAME=\"policy.vkey\" # Standardized names for all multi-asset related files\n#ASSET_POLICY_SK_FILENAME=\"policy.skey\"\n#ASSET_POLICY_SCRIPT_FILENAME=\"policy.script\" # File extension '.script' mandatory\n#ASSET_POLICY_ID_FILENAME=\"policy.id\"\nReminder !!
Ensure the Pre-Requisites are in place before you proceed.
Koios gLiveView is a local monitoring tool to use in addition to remote monitoring tools like Prometheus/Grafana, Zabbix or IOG's RTView. This is especially useful when moving to a systemd deployment - if you haven't done so already - as it offers an intuitive UI to monitor the node status.
"},{"location":"Scripts/gliveview/#configuration-startup","title":"Configuration & Startup","text":"For most setups, it's enough to set CNODE_PORT in the env file. The rest of the variables should automatically be detected. If required, modify User Variables in env and gLiveView.sh to suit your environment (if the environment is customised). This should lead you to a stage where you can now start running ./gLiveView.sh in the folder you downloaded the script (the default location would be $CNODE_HOME/scripts). Note that the script is smart enough to automatically detect when you're running as a Core or Relay and will show fields accordingly.
The tool can be run in legacy mode with only standard ASCII characters for terminals with trouble displaying the box-drawing characters. Run ./gLiveView.sh -h to show available command-line parameters or permanently set it directly in script.
Note !!
Keeping gLiveView to it's intent of being a dashboard and not a full-fledged monitoring tool, we intend to keep most relevant information for a node operator in a minimalistic dashboard, accordingly - gLiveView runs by default in compact mode. One can enable verbose mode by pressing 'v' to unhide additional fields.
A sample output from both core and relay together with peer analysis:
Core Relay Peer Analysis "},{"location":"Scripts/gliveview/#upper-main-section","title":"Upper main section","text":"Displays live metrics from cardano-node gathered through the nodes EKG/Prometheus(env setting) endpoint.
activeSlotsCoeff). A slot on MainNet happens every 1 second(slotLength), thus the max chain density can be calculated as slotLength/activeSlotsCoeff = 5%. Normally, the value should fluctuate around this value. starting|sync xx.x% or if close to reference tip, the tip difference Tip (ref) - Tip (node) to see how far of the tip (diff value) the node is. With current parameters a slot diff up to 40 from reference tip is considered good but it should usually stay below 30. It's perfectly normal to see big differences in slots between blocks. It's the built in randomness at play. To see if a node is really healthy and staying on tip you would need to compare the tip between multiple nodes. Cold peers indicate the number of inactive but known peers to the node.Warm peers tell how many established connections the node has.Hot peers how many established connections are actually active.Bi-Dir(bidirectional) and Uni-Dir(unidirectional) indicate how the handshake protocol negotiated the connection. The connection between p2p nodes will always be bidirectional, but it will be unidirectional between p2p nodes and non-p2p nodes. Duplex shows the connections that are actually used in both directions, only bidirectional connections have this potential.If the node is run as a core, identified by the 'forge-about-to-lead' parameter, a second core section is displayed.
Missed slot checks - A value that show if the node have missed slots for attempting leadership checks (as absolute value and percentage since node startup). !!! info \"Missed Slot Leadership Check\"
Note that while this counter should ideally be close to zero, you would often see a higher value if the node is busy (e.g. paused for garbage collection or busy with reward calculations). A consistently high percentage of missed slots would need further investigation (assistance for troubleshooting can be seeked here ), as in extremely remote cases - it can overlap with a slot that your node could be a leader for.
Blocks - If CNCLI is activated to store blocks created in a blocklog DB, data from this blocklog is displayed. See linked CNCLI documentation for details regarding the different block metrics. If CNCLI is not deployed, block metrics displayed are taken from node metrics and show blocks created by the node since node start.
A manual peer analysis can be triggered by key press p. A latency test will be done on incoming and outgoing connections to the node.
Note
Note that with P2P enabled, an incoming/outgoing connection can be reused for bi-directional traffic. There isnt a way to distinctly identify the P2P peer's direction yet for a given IP.
Outgoing connections(peers in topology file), ping type used is done in this order: 1. cncli - If available, this gives the most accurate measure as it checks the entire handshake process against the remote peer. 2. ss - Sends a TCP SYN package to ping the remote peer on the cardano-node port. Should give ~100% success rate. 2. tcptraceroute - Same as ss. 3. ping - fallback method using ICMP ping against IP. Will only work if firewall of remote peer accept ICMP traffic.
For incoming connections, only ICMP ping is used as remote peer port is unknown. It's not uncommon to see many undetermined peers for incoming connections as it's a good security practice to disable ICMP in firewall.
Once the analysis is finished, it will display the RTTs (return-trip times) for the peers and group them in ranges 0-50, 50-100, 100-200, 200<. The analysis is NOT live. Press [h] Home to go back to default view or [i] Info to show in-script help text. Up and Down arrow keys is used to select incoming or outgoing detailed list of IPs and their RTT value. Left (<) and Right (>) arrow keys can be used to navigate the pages in the selected list.
In case you run into trouble while running the script, you might want to edit env & gLiveView.sh and look at User Variables section. You can override the values if the automatic detection do not provide the right information, but we would appreciate if you could also notify us by raising an issue against the GitHub repository:
gLiveView.sh
######################################\n# User Variables - Change as desired #\n######################################\n\nNODE_NAME=\"Cardano Node\" # Change your node's name prefix here, keep at or below 19 characters!\nREFRESH_RATE=2 # How often (in seconds) to refresh the view (additional time for processing and output may slow it down)\nLEGACY_MODE=false # (true|false) If enabled unicode box-drawing characters will be replaced by standard ASCII characters\nRETRIES=3 # How many attempts to connect to running Cardano node before erroring out and quitting\nPEER_LIST_CNT=6 # Number of peers to show on each in/out page in peer analysis view\nTHEME=\"dark\" # dark = suited for terminals with a dark background\n# light = suited for terminals with a bright background\nENABLE_IP_GEOLOCATION=\"Y\" # Enable IP geolocation on outgoing and incoming connections using ip-api.com\nTo claim rewards earned during the Incentivized TestNet the private and public keys from ITN must be converted to Shelley stake keys. A script called itnRewards.sh has been created to guide you through the process of converting the keys and to create a CNTools compatible wallet from were the rewards can be withdrawn.
jcli account in ITN was ed25519_sk (not extended), you can run the itnRewards.sh script providing the name for the CNTools wallet and ITN owner public/secret keys that were used to register your pool as below. cd $CNODE_HOME/scripts\n./itnRewards.sh MyITNWallet ~/jormu/account/priv/owner.sk ~/jormu/account/priv/owner.pk\nFUNDS >> WITHDRAW to move rewards to the base address of walletDisclaimer
Currently this is to protect the existing pools from the ITN who already have a delegator base against spoofing - to avoid scammers building on results of ITN from known pools. There would be a solution in the future for Mainnet nodes too - but it doesn't apply to those in its current form.
"},{"location":"Scripts/itnwitness/#concept","title":"Concept","text":"Due to the expected ticker spoofing attack for pools that were famous during ITN, some of the community members have proposed an interim solution to verify the legitimacy of a pool for delegators. You can check the high-level workflow below:
graph TB A(\"ITN Owner skey (ed25519/ed25519e) ..\") --x C([\"jcli key sign ..\"]) B(\"Haskell Pool ID (pool.id) ..\") --x C C --x D(\"Signature key, (pool.sig) ..\") E(\"ITN Owner vkey (ed25519_pk) ..\") --x F(\"Extended Metadata JSON (poolmeta_extended.json) ..\") D --x F F --x G(\"Pool Meta JSON (poolmeta.json) ..\") ;"},{"location":"Scripts/itnwitness/#steps","title":"Steps","text":"The actual implementation is pretty straightforward, we will keep it brisk - as we assume ones participating are fairly familiar with jcli usage.
mainnet_pool.id)owner_skey) as per below: jcli key sign --secret-key ~/jormu/account/priv/owner.sk $CNODE_HOME/priv/pool/TEST/pool.id --output mainnet_pool.sig\ncat mainnet_pool.sig\n# ed25519_sig1sn32v3z...d72rg7rc6gs\n{\n\"itn\": {\n\"owner\": \"ed25519_pk1...\",\n\"witness\": \"ed25519_sig1...\"\n}\n}\nIf the process is approved to appear for wallets, we may consider providing easier alternatives. If any queries about the process, or any additions please create a git issue/PR against guild repository - to capture common queries and update instructions/help text where appropriate.
"},{"location":"Scripts/itnwitness/#sample-output-of-json-files-generated","title":"Sample output of JSON files generated","text":"Metadata JSON used for registering pool (one that will be hosted URL used to define pool, eg: https://hosting.site/poolmeta.json)
{\n\"name\":\"Test\",\n\"ticker\":\"TEST\",\n\"description\":\"For demo purposes only\",\n\"homepage\":\"https://hosting.site\",\n\"nonce\":\"1595816423\",\n\"extended\":\"https://hosting.site/poolmeta_extended.json\"\n}\nExtended Metadata JSON used for hosting additional metadata (hosted at URL referred in extended field above, thus - eg : https://hosting.site/poolmeta_extended.json)
{\n\"itn\": {\n\"owner\": \"ed25519_pk1...\",\n\"witness\": \"ed25519_sig1...\"\n}\n}\nReminder !!
Ensure the Pre-Requisites are in place before you proceed.
logMonitor.sh is a general purpose JSON log monitoring script for traces created by cardano-node. Currently, it looks for traces related to leader slots and block creation but other uses could be added in the future.
For the core node (block producer) the logMonitor.sh script can be run to monitor the JSON log file created by cardano-node for traces related to leader slots and block creation.
For optimal coverage, it's best run together with CNCLI scripts as they provide different functionalities. Together, they create a complete picture of blocks assigned, created, validated or invalidated due to node issues.
"},{"location":"Scripts/logmonitor/#installation","title":"Installation","text":"The script is best run as a background process. This can be accomplished in many ways but the preferred method is to run it as a systemd service. A terminal multiplexer like tmux or screen could also be used but not covered here.
Use the deploy-as-systemd.sh script to create a systemd unit file (deployed together with CNCLI). Log output is handled by journald. journalctl -f -u cnode-logmonitor.service can be used to check service output (follow mode). Other logging configurations are not covered here.
Best viewed in CNTools or gLiveView. See CNCLI for example output.
"},{"location":"Scripts/mithril-client/","title":"Client","text":"mithril-client.sh is a script to manage the Mithril client, a tool used to set up the Mithril client environment and manage downloading Mithril snapshots and stake distributions. The main features include:
mithril.env file with all the necessary environment variables for the Mithril client.Usage: bash [-u] <command> <subcommand> [<sub arg>]\nA script to run Cardano Mithril Client\n\n-u Skip script update check overriding UPDATE_CHECK value in env (must be first argument to script)\n\nCommands:\nenvironment Manage mithril environment file\n setup Setup mithril environment file\n override Override default variable in the mithril environment file\n update Update mithril environment file\ncardano-db Interact with Cardano DB\n download Download Cardano DB from Mithril snapshot\n snapshot Interact with Mithril snapshots\n list List available Mithril snapshots\n json List availble Mithril snapshots in JSON format\n show Show details of a Mithril snapshot\n json Show details of a Mithril snapshot in JSON format\nstake-distribution Interact with Mithril stake distributions\n download Download latest stake distribution\n list List available stake distributions\n json Output latest Mithril snapshot in JSON format\nTo prepare a relay or block producer node, you should follow these steps:
environment setup command. This will create a new mithril.env file with all the necessary environment variables for the Mithril client../mithril-client.sh environment setup\nsnapshot download command. This snapshot contains the latest state of the Cardano blockchain db from a Mithril Aggregator../mithril-client.sh cardano-db download\nYou can investigate the available snapshots by using the snapshot list and snapshot show commands:
snapshot list command. Add json at the end to get the output in JSON format../mithril-client.sh cardano-dbsnapshot list\n./mithril-client.sh cardano-dbsnapshot list json\nsnapshot show <DIGEST> command, where <DIGEST> is the digest of the snapshot. Add json at the end to get the output in JSON format../mithril-client.sh cardano-dbsnapshot show <DIGEST>\n./mithril-client.sh cardano-dbsnapshot show <DIGEST> json\n./mithril-client.sh cardano-dbsnapshot show json <DIGEST>\nYou can manage stake distributions by using the stake-distribution download and stake-distribution list commands:
stake-distribution download command../mithril-client.sh stake-distribution download\nstake-distribution list command. Add json at the end to get the output in JSON format../mithril-client.sh stake-distribution list\n./mithril-client.sh stake-distribution list json\nmithril-relay.sh is a bash script for deployment of Squid Mithril Relays and a Nginx loadbalancer. It provides functionalities such as:
bash [-d] [-l] [-u] [-h]\nA script to setup Cardano Mithril relays\n\n-d Install squid and configure as a relay\n-l Install nginx and configure as a load balancer\n-u Skip update check\n-h Show this help text\nThe mithril-relay.sh script is a bash script for managing the Mithril Relay Server. It provides functionalities such as installing and configuring Squid as a relay, installing and configuring Nginx as a load balancer.
The script uses the following environment variable:
RELAY_LISTENING_PORT: The port on which the relay server listens.The script parses command line options and performs the corresponding actions based on the options provided. If the -d option is provided, it installs Squid and configures it as a relay. If the -l option is provided, it installs Nginx and configures it as a load balancer. If no options are provided, it displays the usage message.
mithril-signer.sh is a bash script for managing the Mithril Signer Server. It provides functionalities such as deploying the server as a systemd service and updating the environment file to contain variables specific to the Mithril Signer.
Usage: bash [-d] [-D] [-e] [-k] [-r] [-s] [-u] [-h]\nA script to setup, run and verify Cardano Mithril Signer\n\n-d Deploy mithril-signer as a systemd service\n-D Run mithril-signer as a daemon\n-e Update mithril environment file\n-k Stop signer using SIGINT\n-r Verify signer registration\n-s Verify signer signature\n-u Skip update check\n-h Show this help text\nThis script is a bash script for managing the Mithril Signer Server. It provides functionalities such as deploying the server as a systemd service, updating the environment file, and running the server.
"},{"location":"Scripts/mithril-signer/#environment-variables","title":"Environment Variables","text":"The script uses several environment variables, some of which are:
MITHRILBIN: Path for mithril-signer binary, if not in $PATH.HOSTADDR: Default Listen IP/Hostname for Mithril Signer Server.POOL_NAME: The name of the pool.NETWORK_NAME: The name of the network.The script parses command line options, sources the environment file, sets default values, and performs basic sanity checks. It then checks if the -d or -u options were specified and performs the corresponding actions. If no options were specified, it runs the Mithril Signer Server.
!> Ensure the Pre-Requisites are in place before you proceed.
This section describes the ways in which CNTools can send important messages to the operator.
"},{"location":"Scripts/sendalerts/#telegram-alerts","title":"Telegram alerts","text":"If known but unwanted errors occur on your node, or if characteristic values indicate an unusual status , CNTools can send you Telegram alert messages.
To do this, you first have to activate your own bot and link it to your own Telegram user. Here is an explanation of how this works:
Open Telegram and search for \"botfather\".
Write him your wish: /newbot.
Define a name for your bot, such as cntools_[POOLNAME]_alerts.
Botfather will confirm the creation of your bot by giving you the unique bot access token. Keep it safe and private.
Now send at least one direct message to your new bot.
Open this URL in your browser by using your own, just created bot access token:
https://api.telegram.org/bot<your-access-token>/getUpdates\nresult.message.chat.id. This chat id should be a large integer number.This is all you need to enable your Telegram alerts in the scripts/env file - uncomment and add the chat ID to the TG_CHAT_ID user variable in the env file:
...\nTG_CHAT_ID=\"<YOUR_TG_CHAT_ID>\"\n... \nReminder !!
The topologyUpdater shell script must be executed on the relay node as a cronjob exactly every 60 minutes. After 4 consecutive requests (3 hours) the node is considered a new relay node in listed in the topology file. If the node is turned off, it's automatically delisted after 3 hours.
"},{"location":"Scripts/topologyupdater/#download","title":"Download and Configure","text":"If you have run guild-deploy.sh, this should already be available in your scripts folder and make this step unnecessary.
Before the updater can make a valid request to the central topology service, it must query the current tip/blockNo from the well-synced local node. It connects to your node through the configuration in the script as well as the common env configuration file. Customize these files for your needs.
To download topologyUpdater.sh manually, you can execute the commands below and test executing Topology Updater once (it's OK if first execution gives back an error):
cd $CNODE_HOME/scripts\ncurl -s -o topologyUpdater.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/topologyUpdater.sh\ncurl -s -o env https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/env\nchmod 750 topologyUpdater.sh\n./topologyUpdater.sh\nOut of the box, the scripts might come with some assumptions, that may or may not be valid for your environment. One of the common changes as an SPO would be to the complete CUSTOM_PEERS section as below to include your local relays/BP nodes (described in the How do I add my own nodes section), and any additional peers you'd like to be always available at minimum. Please do take time to update the variables in User Variables section in env & topologyUpdater.sh:
### topologyUpdater.sh\n\n######################################\n# User Variables - Change as desired #\n######################################\n\nCNODE_HOSTNAME=\"CHANGE ME\" # (Optional) Must resolve to the IP you are requesting from\nCNODE_VALENCY=1 # (Optional) for multi-IP hostnames\nMAX_PEERS=15 # Maximum number of peers to return on successful fetch\n#CUSTOM_PEERS=\"None\" # Additional custom peers to (IP,port[,valency]) to add to your target topology.json\n# eg: \"10.0.0.1,3001|10.0.0.2,3002|relays.mydomain.com,3003,3\"\n#BATCH_AUTO_UPDATE=N # Set to Y to automatically update the script if a new version is available without user interaction\nAny customisations you add above, will be saved across future guild-deploy.sh executions, unless you specify the -f flag to overwrite completely.
systemd service The script can be deployed as a background service in different ways but the recommended and easiest way if guild-deploy.sh was used, is to utilize the deploy-as-systemd.sh script to setup and schedule the execution. This will deploy both push & fetch service files as well as timers for a scheduled 60 min node alive message and cnode restart at the user set interval (default: 24 hours) when running the deploy script.
cnode-tu-push.service : pushes a node alive message to Topology Updater APIcnode-tu-push.timer : schedules the push service to execute once every hourcnode-tu-fetch.service : fetches a fresh topology file before the cnode.service file is started/restartedcnode-tu-restart.service : handles the restart of cardano-node (cnode.sh)cnode-tu-restart.timer : schedules the cardano-node restart service, default every 24hsystemctl list-timers can be used to to check the push and restart service schedule.
crontab job Another way to deploy the topologyUpdater.sh script is as a crontab job. Add the script to be executed once per hour at a minute of your choice (eg xx:25 o'clock in the example below). The example below will handle both the fetch and push in a single call to the script once an hour. In addition to the below crontab job for topologyUpdater, it's expected that you also add a scheduled restart of the relay node to pick up a fresh topology file fetched by topologyUpdater script with relays that are alive and well.
25 * * * * /opt/cardano/cnode/scripts/topologyUpdater.sh\nYou can check the last result of push message in logs/topologyUpdater_lastresult.json. If deployed as systemd service, use sudo journalctl -u <service> to check output from service.
If one of the parameters is outside the allowed ranges, invalid or missing the returned JSON will tell you what needs to be fixed.
Don't try to execute the script more often than once per hour. It's completely useless and may lead to a temporary blacklisting.
"},{"location":"Scripts/topologyupdater/#why-does-my-topology-file-only-contain-iog-peers","title":"Why does my topology file only contain IOG peers?","text":"Each subscribed node (4 consecutive requests) is allowed to fetch a subset of other nodes to prove loyalty/stability of the relay. Until reaching this point, your fetch calls will only return IOG peers combined with any custom peers added in USER VARIABLES section of topologyUpdater.sh script
The engineers of cardano-node network stack suggested to use around 20 peers. More peers create unnecessary and unwanted system load and delays.
In its default setting, topologyUpdater returns a list of 15 remote peers.
Note that the change in topology is only effective upon restart of your node. Make sure you account for some scheduled restarts on your relays, to help onboard newer relays onto the network (as described in the systemd section).
"},{"location":"Scripts/topologyupdater/#how-do-i-add-my-own-relaysstatic-nodes-in-addition-to-dynamic-list-generated-by-topologyupdater","title":"How do I add my own relays/static nodes in addition to dynamic list generated by topologyUpdater?","text":"Most of the Stake Pool Operators may have few preferences (own relays, close friends, etc) that they would like to add to their topology by default. This is where the CUSTOM_PEERS variable in topologyUpdater.sh comes in. You can add a list of peers in the format of: hostname/IP:port[:valency] here and the output topology.json formed will already include the custom peers that you supplied. Every custom peer is defined in the form [address]:[port] and optional :[valency] (if not specified, the valency defaults to 1). Multiple custom peers are separated by |. An example of a valid CUSTOM_PEERS variable would be:
CUSTOM_PEERS=\"foo.bar.io,3001,2|198.175.21.197,6001|36.233.3.89,6000\n2)."},{"location":"Scripts/topologyupdater/#how-are-the-peers-for-my-topology-file-selected","title":"How are the peers for my topology file selected?","text":"We calculate the distance on the Earth's surface from your node's IP to all subscribed peers. We then order the peers by distance (closest first) and start by selecting one peer. We then skip some, pick the next, skip, pick, skip, pick ... until we reach the end of the list (furthest away). The number of skipped records is calculated in a way to have the desired number of peers at the end.
Every requesting node has its personal distance to all other nodes.
We assume this should result in a well-distributed and interconnected peering network.
"},{"location":"docker/build/","title":"Build","text":""},{"location":"docker/build/#intro","title":"Intro","text":"\ud83d\udca1 Docker containers are the fastest way to run a Cardano node in both \"Relay\" and \"Block-Producing\" (Pool) mode.
"},{"location":"docker/build/#how-to-build","title":"How to build","text":"docker build -t cardanocommunity/cardano-node:latest - < dockerfile_bin\nWith Powershell on Windows, you can run docker by typing the following command:
Get-Content dockerfile_bin | docker build -t guild-operators/cardano-node:latest -\nDocker Tips
Docker Official Docs
"},{"location":"docker/docker/","title":"Overview","text":"Running your own Cardano node has never been so fast and easy.
But first, a kind reminder to the security aspects of running docker containers.
"},{"location":"docker/docker/#external-resources","title":"External resources","text":"Modular docker images based on Debian.
Based on the Guild's work the Cardano Node image is built in a single stage: -> dockerfile_bin
guild-deploy.sh to:If you prefer to build the images your own than you can check:
The dockerfiles are located in ./files/docker/
Node Ports Wallet Ports Flavor Node (6000) Wallet (8090) Debian Prometheus (12798) Prometheus (12798) EKG (12781)"},{"location":"docker/run/","title":"Run","text":""},{"location":"docker/run/#os-requirements","title":"OS Requirements","text":"docker-ce installed - Get Docker.Note
1) --entrypoint=bash # This option won't start the node's container but only the OS running (the node software wont actually start, you'll need to manually execute entrypoint.sh ), ready to get in (trough the command docker exec -it < container name or hash > /bin/bash) and play/explore around with it in command line mode. 2) all guild tools env variable can be used to start a new container using custom values by using the \"-e\" option. 3) CPU and RAM and Shared Memory allocation option for the container can be used when you start the container (i.e. --shm-size or --memory or --cpus official docker resource docs) 4) --env MITHRIL_DOWNLOAD=Y # This option will allow Mithril client to download the latest Mithril snapshot of the blockchain when the container starts and does not have a copy of the blockchain yet. This is useful when you want to start a new node from scratch and don't want to wait for the node to sync from the network. This option is not currently available for the guild network. 5) --env ENTRYPOINT_PROCESS=mithril-signer.sh # This option will allow the container to start the Mithril signer process instead of the node process. This is useful when you want to run a Mithril signer node and have the container setup the configuration files based on the NETWORK environment varaible.
docker run --init -dit\n--name <YourCName>\n--security-opt=no-new-privileges\n-e NETWORK=mainnet\n-v <your_custom_path>:/opt/cardano/cnode/priv\n-v <your_custom_db_path>:/opt/cardano/cnode/db\ncardanocommunity/cardano-node\ndocker run --init -dit\n--name <YourCName>\n--security-opt=no-new-privileges\n-e NETWORK=mainnet\n-p 6000:6000\n-v <your_custom_path>:/opt/cardano/cnode/priv\n-v <your_custom_db_path>:/opt/cardano/cnode/db\ncardanocommunity/cardano-node\ndocker run --init -dit\n--name <YourCName>\n--security-opt=no-new-privileges\n-e NETWORK=mainnet\n-e CONFIG=/opt/cardano/cnode/priv/<your own configuration files>.yml\n-p 6000:6000\n-v <your_custom_path>:/opt/cardano/cnode/priv\n-v <your_custom_db_path>:/opt/cardano/cnode/db\ncardanocommunity/cardano-node\nOn the security front, Docker developers are faced with different types of security attacks such as:
Docker containers are now being exploited to covertly mine for cryptocurrency, marking a shift from ransomware to cryptocurrency malware. As with all things in security, also Docker security is a moving target \u2014 so it\u2019s helpful to have access to up-to-date information, including experience-based best practices, for securing your containerized environments.
"},{"location":"docker/security/#here-below-some-key-concepts","title":"Here below some key concepts:","text":"Use a Third-Party Security Tool Docker allows you to use containers from untrusted public repositories, which increases the need to scrutinize whether the container was created securely and whether it is free of any corrupt or malicious files. For this, use a multi-purpose security tool that gives extensive dev-to-production security controls.(keep reading below)
Manage Vulnerability It is best to have a sound vulnerability management program that has multiple checks throughout the container lifecycle. Vulnerability management should incorporate quality gates to detect access issues and weaknesses for a potential exploit from dev-to-production environments.
Monitor and Audit Container Activity It is vital to monitor the container ecosystem and detect suspicious activity. Container monitoring activities provide real-time reports that can help you react promptly to a security breach.
Enable Docker Content Trust Docker Content Trustis a new feature incorporated into Docker 1.8. It is disabled by default, but once enabled, allows you to verify the integrity, authenticity, and publication date of all Docker images from the Docker Hub Registry.
Use Docker Bench for Security You should consider Docker Bench for Security as your must-use script. Once the script is run, you will notice a lot of information regarding configuration best practices for deploying Docker containers that can be used to further secure your Docker server and containers.
Resource Utilization To reduce performance impacts and denial-of-service attacks, it is a good practice to implement limits on the system resources that the containers can consume. If, for example, a web server is compromised, it helps to limit the impact to the other processes that are running on a host.
RBAC RBAC is role-based access control. If you have multiple users accessing you enviroment, this is a must-have. It can be quite expensive to implement but portainer makes it super easy.
Guild tips:
NEVER NEVER NEVER expose Docker API publicly!!! (disabled by default)
Keep Docker Host Up-to-date
Reverse ProxyDocker Socket OwnershipRun Docker Containers as RootUse Trusted Docker ImagesUse Privileged Mode Carefully (This is usually done by adding --privileged you can use --security-opt=no-new-privileges instead)Some more general tips:
\"--cap-drop ALL\"DOCKER_OPTS= \"--iptables=false\"With this quick guide you will be able to run a cardano node in seconds and also have the powerfull Koios SPO scripts built-in.
"},{"location":"docker/tips/#how-to-operate-interactively-within-the-container","title":"How to operate interactively within the container","text":"Once executed the container as a deamon with attached tty you are then able to enter the container by using the flag -dit .
While if you have a hook within the container console, use the following command (change CN with your container name):
docker exec -it CN bash This command will bring you within the container bash env ready to use the Koios tools.
"},{"location":"docker/tips/#docker-flags-explained","title":"Docker flags explained","text":"\"docker build\" options explained:\n -t : option is to \"tag\" the image you can name the image as you prefer as long as you maintain the references between dockerfiles.\n\n\"docker run\" options explained:\n -d : for detach the container\n -i : interactive enabled -t : terminal session enabled\n -e : set an Env Variable\n -p : set exposed ports (by default if not specified the ports will be reachable only internally)\n--hostname : Container's hostname\n --name : Container's name\ndocker run --init -itd \n-name Relay # Optional (recommended for quick access): set a name for your newly created container.\n-p 9000:6000 # Optional: to expose the internal container's port (6000) to the host <IP> port 9000\n-e NETWORK=mainnet # Mandatory: mainnet / preprod / guild-mainnet / guild\n--security-opt=no-new-privileges # Option to prevent privilege escalations\n-v <YourNetPath>:/opt/cardano/cnode/sockets # Optional: useful to share the node socket with other containers\n-v <YourCfgPath>:/opt/cardano/cnode/priv # Optional: if used has to contain all the sensitive keys needed to run a node as core\n-v <YourDBbk>:/opt/cardano/cnode/db # Optional: if not set a fresh DB will be downloaded from scratch\ncardanocommunity/cardano-node:latest # Mandatory: image to run\nNote
To be able to use the CNTools encryption key feature you need to manually change in \"cntools.config\" ENABLE_CHATTR to \"true\" and not use the --security-opt=no-new-privileges docker run option.
The docker container has an optional backup and restore functionality that can be used to backup the /opt/cardano/cnode/db directory. To have the backup persist longer than the countainer, the backup directory should be mounted as a volume.
[!NOTE] The backup and restore functionality is disabled by default.
[!WARNING] Make sure adequate space exists on the host as the backup will double the space consumed by the database.
"},{"location":"docker/tips/#creating-a-backup","title":"Creating a Backup","text":"When the container is started with the ENABLE_BACKUP environment variable set to Y the container will automatically create a backup in the /opt/cardano/cnode/backup/$NETWORK-db directory. The backup will be created when the container is started and if the backup directory is smaller than the db directory.
When the container is started with the ENABLE_RESTORE environment variable set to Y the container will automatically restore the latest backup from the /opt/cardano/cnode/backup/$NETWORK-db directory. The database will be restored when the container is started and if the backup directory is larger than the db directory.
The container now includes a static copy of each network's configuration files (Mainnet, Preprod, Preview, Sanchonet, and Guild networks). The NETWORK environment variable passed into the container determines which configuration files are copied into $CNODE_HOME/files.
The UPDATE_CHECK environment variable controls whether the container updates these configuration files from GitHub before starting. By default, the container has the environment variable set to UPDATE_CHECK=N, meaning the container uses the configuration files it was built with. This can be overriden either persistently or dynamically.
To always update the configuration files from GitHub, set the UPDATE_CHECK environment variable when creating the container by using the --env option, for example --env UPDATE_CHECK=Y.
To always update the configuration files from a specific GitHub account, set the G_ACCOUNT environment variable when creating the container by using the --env option, for example --env G_ACCOUNT=gh-fork-user.
[!NOTE] There is no way to change the environment variable of an already running container. To rollback the configuration files and scripts stop and remove the container and start it without setting the environment variable.
"},{"location":"docker/tips/#dynamically-updating-configuration-files","title":"Dynamically updating configuration files","text":"Set an environment file during create/run using --env-file=file, for example --env-file=/opt/cardano/cnode/.env.
UPDATE_CHECK is not defined in the environment file, the container will use the built-in configs.UPDATE_CHECK=Y is defined in the environment file the container will update configs and scripts from the cardano-community GitHub repository.G_ACCOUNT is defined in the environment file, the container will update configs and scripts from the GitHub repository of the specified account.To rollback the configuration files to the built-in versions, remove the UPDATE_CHECK=Y or set it to UPDATE_CHECK=N in the environment file. The static configuration files in the container will be used, however the scripts will remain updated. If you want both the configuration files and scripts to be rolled back, you will need to stop and remove the container and create a new one.
Run the Docker Image GitHub Action to build and push images to the ghcr.io registry.
G_ACCOUNT will be inherited from the GITHUB_REPOSITORY_OWNER.ghcr.io.ghcr.io registry as long as the Testing workflow remains checked.