Koios 1.1.1rc

Koios 1.1.1rc

v1.1.1rc is a release candidate, but will still be open for public testing for a couple of weeks (to include any fixes we missed to check to be added to the API).

This release primarily focuses on backend performance fixes and work with updates from node-8.7.3 and dbsync 13.2.0.1 - we have also started preparing compatibility with upcoming koios lite release, to make a seamless swap for specific endpoints without any impact to consumers. There are no breaking (impact to existing columns or inputs) changes with this release, but we dp have retired 2 deprecated endpoints that were almost unused on mainnet.

Due to the amount of backend changes in queries, there is a chance that we may have missed some data accuracy checks, and - hence - would like to test in non-mainnet networks first before marking final release. Accordingly, any testing/reports of data inconsistency would be welcome.

For instance providers, since there is major version bump of underlying infrastructure components (you need to re-create DB when updating to 13.2.0.1 if coming from 13.1.x.x) - expect some downtime to work through the update when you get to it.

API Docs and Schedule for release:

The schedule for monitoring layers to apply update , as well as live documentation can be browsed as per table below:

Network	Date	URL	Specs Release	gRest Instance Release	DBSync	Node
GuildNet	17-03-2024 22:00 UTC	https://guild.koios.rest/	v1.1.1rc	koios-1.1.1rc	13.2.0.2	8.9.2
Preview	20-03-2024 22:00 UTC	https://preview.koios.rest/	v1.1.1rc	koios-1.1.1rc	13.2.0.2	8.9.2
PreProd	20-03-2024 22:00 UTC	https://preprod.koios.rest/	v1.1.1rc	koios-1.1.1rc	13.2.0.2	8.9.2
Mainnet*	10-05-2024 22:00 UTC	https://api.koios.rest/	v1.1.1	koios-1.1.1	13.2.0.2	8.9.2

*For mainnet , we will go for finalised release (instead of v1.1.1rc, but that hasnt been tagged yet - will depend on outcomes of testing)

---

Changes for API

Expand

New endpoints added:

• /asset_policy_mints - List of mint/burn count for all assets minted under a policy #269
• /block_tx_info - Equivalent of tx_info but uses blocks as inputs to fetch tx_info against all tx in the block[s] requested, also contains additional flags to control performance and output #255

Data (Output only) Changes:

• /reserve_withdrawals , /treasury_withdrawals - Add earned_epoch and spendable_epoch fields #269
• /block - Add parent_hash field #263
• /account_list - Add stake_address_hex and script_hash fields #263
• /asset_list - Add script_hash field #263
• /asset_summary - Add addresses field #263
• /asset_addresses , /asset_nft_address and /policy_asset_addresses - Add stake_address field #262

Deprecations:

• None

Retirements:

• asset_address_list and asset_policy_info endpoints are now retired, as they were marked as deprecated in Koios 1.0.10 , and we have seen it's usage to be negligible (only a single hit in 48 hours on mainnet while marking this release). #269

Chores:

• Retire stake_distribution_new_accounts and stake_snapshot_cache cache, as we directly perform lookup on live tables for newly registered accounts #269
• Active stake cache no longer reads the logs, but instead relies on newly added epoch_sync_progress table #269
• Reduce asset_info_cache rollback lookup from 1000 to 250 #269
• Replace consumed_by_tx_in_id references in SQL by consumed_by_tx_id #269
• pool_history_cache now breaks into populating 500 epochs at a time (on guildnet, this query used to run for hours against ~20K epochs) #269
• Accomodate splitting of reward table into instant_reward #269
• Add a check in stake_distribution_cache to ensure that epoch info cache was run for current - 1 epoch. #269
• Change return type for internal function grest.cip67_strip_label from text to bytea #269
• Remove any references to tx_in as it is no longer required #269
• Remove references to pool_offline_data with off_chain_pool_data #269
• Disable running asset-txo-cache-update as the endpoints leveraging asset-txo will be moved to koios-lite
• Convert block, account_list, asset_list asset_token_registry from view to function #263
• asset_info_cache - ensure mint tx is only against a positive mint quantity #262

Full Changelog can be found here for Specs , and here for scripts and SQL queries.

---

Instructions for Instance Providers

Expand

If you're setting up as a new instance provider, you can find instructions here. The tag to use for setup-grest.sh would be koios-1.1.1rc - for which you can visit the SQL queries used here.

If you're an existing instance provider using koios-1.0.x, the update process would be:

1. Update your node, submitapi, dbsync and ogmios binaries. You can use the sample below as reference (change network and top folder according to your environment):

./guild-deploy.sh -n $NETWORK -t cnode -b koios-1.1.1rc -s plfdo

Note that the use of f above is to force overwrite your configs , as newer node includes ConwayGenesis placeholder keys. The above command will overwrite your config.json, topology.json as well as user variables, and save them original ones backups, you'd want to copy your user variables into the files modified and re-add port customisations to config.json or topology peering.

2. Shutdown your node and dbsync services - and start node back up (replace cnode as per your environment in instructions):

sudo systemctl stop cnode cnode-dbsync
sudo rm -f /etc/cron.d/cnode-*
sudo systemctl start cnode

Expect node to take a few hours to startup, you can short-circuit the process by copying $CNODE_HOME/db folder from already upgraded node - while both source and destination node folders were against node being in shutdown state (or use mithril-client or use https://csnapshots.io). You can also check node startup status by viewing node logs (tail -100f $CNODE_HOME/logs/node0.json)

3. While you Wait for your node to be in sync (you can monitor via gLiveView), you can proceed with dropping current dbsync DB and restoring from snapshot. To do so, use the below (execute commands one by one - verifying output):

We'd recommend using Postgres 15 and above, but it's up to individuals.

cd ~/git/cardano-db-sync
git fetch
git checkout 13.2.0.1
./scripts/postgresql-scripts.sh --recreatedb
rm -rf ${CNODE_HOME}/guild-db/ledger-state ; mkdir -p ${CNODE_HOME}/guild-db/ledger-state
ln -s ~/git/cardano-db-sync/schema ${CNODE_HOME}/guild-db/schema 
cd $CNODE_HOME/scripts ; ./dbsync.sh -d

If you're on mainnet, you'd likely want to restore from snapshot:

wget https://snapshots.koios.rest/dbsync-13.2-mainnet/db-sync-snapshot-schema-13.2-block-10045937-aarch64.tgz -O /tmp/dbsyncsnap.tgz
scripts/postgresql-setup.sh --restore-snapshot /tmp/dbsyncsnap.tgz ${CNODE_HOME}/guild-db/ledger-state

Once the above is complete, you can start dbsync by sudo systemctl start cnode-dbsync. Wait for dbsync to finish synching, you can tail logs via tail -10f $CNODE_HOME/logs/dbsync.json - on mainnet in particular, this could take 2 days).

5. With dbsync updated and on tip, you'd now be ready to go to the next step - updating koios. You'd want to update all components and reset your grest schema, sample command below:

./setup-grest.sh -b koios-1.1.1rc -r -i prmcd
sudo systemctl restart cnode-postgrest cnode-haproxy

Due to difference in script provisioning versions, it is possible that you might have trouble starting cnode-postgrest - if so, check and ensure that you can access $CNODE_HOME/priv/grest.conf file as authenticator user - you can use command as test : sudo -u authenticator cat /opt/cardano/gnode/priv/grest.conf

Since we're again re-building cache, expect the cache to take a bit (few minutes on non-mainnet, while a few hours on mainnet).

---

Other Notes

Expand

Bug Reports/Feature Requests

Everyone is invited to create future feature requests here to help us triage issues.

Support/Discussions Group

Please ensure to join the Koios Support/Discussions group.
We're open to have meetings to work together as well as discuss future additions, feel free to ask

Contributors

We are blessed to have community who contribute in various ways, be it by serving community instances, library implementations, testing, or feedbacks. Special thanks to all the contributors that have been helping us improve the query layer along the way
@Scitz0 , @rdlrt , @redoracle, @hodlonaut, @reqlez , @huths0lo , @chadle-git , @gufmar , @dostrelith678, @ray-wallet, @edridudi , @mkungla , @xray-robot , @nothingalike, @safestak-keith , @QuixoteSystems, @abdelkrimdev, @cardano-apexpool , @AlexDochioiu , @DCOneCrypto , @PatrickTobler , @agaffney , @rcmorano , @gitmachtl, @Crypto2099, @robinboening, @fallen-icarus , @M4rc0Russ0 , @HT-Moh , @ducpm2303 , @michele-nuzzi , @maxee , @Jack-0