Make upgrade status of the network easier accessible for everyone

[GuybrushX]

TL;DR

I would like to have the possibility to make the upgrade status of nodes available to everyone whenever a new upgrade is available and before it went live to get an overview about which nodes have staged the upgrade and how smooth the upgrade will go.

-> this feature could be implemented via an "upgrade status smart contract". The current upgrade process could add the possibility of publishing the upgrade status to the smart contract
-> that way the data is automatically available to tools like https://cspr.live or anybody else by querying the blockchain
-> it could be shown as a "Reliability Score" to incentivize node operators to use that smart contract
-> delegators could check the status of their validator on their own
-> node operators who didn't stage the upgrade yet could be contacted by others before anybody is losing rewards and reputation/trust (if contact data is available)

The current situation & much more details

The casper-node software which is needed to run the Casper Network provides a REST-API endpoint to get the status of an individual node. That includes the following details:

{
  "api_version": "1.4.15",
  "chainspec_name": "casper",
  "starting_state_root_hash": "eeec3c4eb17181647ed047ee9a983c633564681b5c40d6626e485f28b3d2500f",
  "peers": [
    {
      "node_id": "tls:1234..4321",
      "address": "1.2.3.4:35000"
    }
  ],
  "last_added_block_info": {
    "hash": "41d1d21ac65bc67769f4dfe6019cd3a39e6d380390707e01b2493e56cc19d505",
    "timestamp": "2023-08-26T16:41:39.584Z",
    "era_id": 10518,
    "height": 2006626,
    "state_root_hash": "51d75ba004fc9bd4791090541ef3c949d09900ac34cc6eaed02d38b70b032c21",
    "creator": "01ebaebffebe63ee6e35b88697dd9d5bfab23dac47cbd61a45efc8ea8d80ec9c38"
  },
  "our_public_signing_key": "<PublicKey>",
  "round_length": null,
  "next_upgrade": {
    "activation_point": 10553,
    "protocol_version": "1.5.2"
  },
  "build_version": "1.4.15-039d438f2-casper-mainnet",
  "uptime": "1day 15h 24m 26s 141ms"
}

It includes all kind of useful information, esp. the "next_upgrade" part, which can be used to check if a node has an upcoming upgrade staged or not. With the upcoming 1.5.2 upgrade there will be even more information available like how far the sync is already and which blocks are available on a node if the new SyncToTTL was used etc.

For security reasons node operators can disable that REST-API or block access from the internet.

What is bad about this?

If the endpoint is not accessible I'm right now not aware of a reliable way:

• to see which node operator has staged an upgrade already or not
• it's also not possible for developers to check if the node they want to use has enough historical data or if it's still syncing etc.
• delegators have to contact node operators directly (if they have contact details which not all provide) to get the information if the next upgrade was staged already
• nodes without the upgrade staged are discovered only during or after the upgrade when they get equivocated

Can and should this be improved?

I think so!

If that information would somehow be available network-wide than tools like https://cspr.live and other monitoring solutions could show that information to delegators and all other users of the blockchain. It would also be possible to spot nodes which haven't the upgrade staged yet and if contact details are available the node operator could be informed by others. In a perfect world that shouldn't happen but past upgrades show that there are always node operators who are missing upgrades.

Possible solutions

We have to differentiate between 2 things:

1. Node status for upgrades and before the upgrade is happening
2. General information about the node status like current block height, sync status etc. which will change very frequently -> this is not covered by this discussion

After quite some "discussions" with ChatGPT we came up with the following ideas:

1. Implement a smart contract which can be written to whenever a node upgrade was staged
2. Add the upgrade info to be part of the existing gossip protocol (and maybe also use a smart contract for it?)
3. Pushing upgrade data to any centralized entity like it is done right now already from many but not all node operators which makes it unreliable

My favorite is #1 because of the following reasons:

• no code changes and intensive testing needed for casper-node because it stays untouched
• data is stored in a decentralized way
• data is automatically accessible to everyone
• sites like https://cspr.live could add another metric like "Trust Score" or "Reliability" and could take an average of the last 3, 5, 10 upgrades and if they have been missed or staged in time and before the upgrade era was reached
• deploys using that smart contract can also be tracked by the SSE endpoint

-> I believe that the "Reliability" score could even be added right now already by checking which node was not earning rewards anymore after the last upgrades. Similar functionality exists already and is shown as "Performance" so it should be not too hard to implement it already.

The "Reliability" score also adds to the existing reputation system to incentivize node operators to participate and publish their upgrade status.

How it could be implemented:

• it could become part of the existing upgrade process to make that an easy process
• A timestamp of the upgrade process will exist as well because it will be a deploy to the network and/or added to the upgrade status but I would NOT use that as a metric on how late or early an upgrade was staged. As long as the upgrade isn't staged after its activation it doesn't really matter when an upgrade is staged and it can be very valid to not be the first because of potential bugs with new upgrade scripts
• the publishing of the upgrade status should require either confirmation or be another final step after it was checked and confirmed by the node operator that the upgrade was staged successfully and is waiting for activation -> the upgrade process could have a bug or any other unforeseen problems could occur. That way node operators also can actively decide if they want to publish the upgrade status or not.

Using the smart contract would provide information before the upgrade happens and to foresee if and how smooth the upgrade will be and if either delegators or other node operators need to either move their stake and/or try to inform the node operators which didn't stage the upgrade yes.

Modifying the gossip protocol should be avoided since it makes things much more complicated I guess. I think it makes sense to get feedback of core developers here if that would make sense or not.

Using a centralized service depends too much on a centralized entity which could be down, needs maintenance, and adds costs to run additional infrastructure for it. The blockchain exists already and is built to run 24/7.

Possible issues

The smart contract has a few downsides as well:

• it needs to be designed, developed, and implemented in the upgrade process
• it adds a bit more "pressure" to the node operators and the network which should be permissionless

I think this downsides are pretty small and the benefits are more important which makes it worth to discuss and implement it.

Opinions? Feedback?

[KillianH]

I'm not sure that a smart contract would work. You could still invoke the smart contract by yourself so you couldn't trust what's stored in the smart contract. The CNM tool built by Joe isn't 'centralized' it's a representation of the state of the network that is itself decentralized. If node operators don't want to open their rest API with a request per second setting that is totally safe up to them. Likewise for the account info smart contract.

Validators shouldn't be pressured to do an update just because there's an upgrade. They should stage it if they're well informed and accept the changes included in the update.

[GuybrushX]

Cheating the system

Yeah, cheating the system is an issue as always BUT this can be detected easily when the node loses on rewards or gets equivocated because it was reported by the node operator that the upgrade was staged but it was not and the node lost rewards because of that - that's basically the part where I've mentioned that the current "Performance" metric does a similar check already, just for the last 360 eras instead of the specific upgrade eras. In fact I think this check could easily be implemented already - even for the past regardless of the system contract.

Joe's CNM tool

Joe's tool is indeed super centralized since it is run by a single person and probably a single server. AFAIK it was a temporary solution and became more permanent than planned. It also "forces" everyone to either run another public service (the status REST-API) or add firewall exceptions, update them on changes and you even need to know that this exists. Also all collected data is stored centrally and isn't directly accessible by anyone else besides via the website or via web scraping, I also think there is no API -> how to implement this in a custom tool or on https://cspr.live? How to guarantee uptime and availability?

Agreeing on upgrades

I totally agree that upgrades should be checked before as much as possible by node operators and also by delegators in the long-run and not blindly installed. I assume here that this was done and otherwise a new discussion would start by node operators who don't agree on a specific change.

I also think that in "a perfect or decentralized world" essential changes are discussed before they get implemented and before an upgrade is made available with an activation time of less than 1 week in the future. That saves development time and gives enough time to discussions for all node operators and developers.

Pressure on node operators

Besides that node operators are pressured already to implement the Account Info Standard or the ranking on https://cspr.live is highly impacted - even the biggest validator (by total stake) is on position ~65 instead of #1 because of this. Also the Performance metric exists already and has an impact on the reputation of node operators in the public.

Small addition

Maybe it makes sense to also provide the option of adding an (optional) reason when an upgrade status is published? Either a pre-defined list of values like "I accept all new changes" or "I don't agree - needs further discussion first". Not sure about the benefits - it's also easy to forget all possible cases/reasons. A custom message adds overhead again etc. -> it makes things more complicated for sure - I doubt that it's worth it/useful in this case but since Casper supports upgradable smart contracts I think this could be added later on.

[sacherjj]

Currently the next_upgrade indicates that config files exist in /etc/casper/[protocol version] for a protocol version more than the current one. It parses the chainspec.toml to find the activation point. This could be completely bogus or the config.toml created could be bad and immediately crash the node. I do not recall if it required an associated /var/lib/casper/bin/[protocol version] to exist.

In other words, the presence of that folder does not assure that the node will continue to function at the upgrade point. However, that is also true with the current system, so someone irrelevant to improving the notification.

I don't believe the smart_contract is a valid method of storing and accessing this. The only port that is required to operate is node to node communication (default 35000). "Perfect" upgrade notification would required some level of status gossiping between nodes and each node to hold latest status information from other nodes. This would be ephemeral and should not take up much local storage, but update rate would determine how much network traffic is used.

It is possible that this could be used for handoff of nodes if we automate swapping between hot and cold nodes, using the same key. This is something I will be writing up in a CEP.

So if each node had a complete picture of the view of all, we should be able to use any node for update (and other network status) information.

[Charlotte-br560]

Implementing an "upgrade status smart contract" seems like a great idea. It would provide transparency to node operators and delegators, enhancing the reliability of network upgrades. Integrating it into the existing upgrade process minimizes complexity. The proposed "Reliability Score" adds an incentive for participation.