[Consensus] Consensus Onboarding & Knowledge Transfer

[Olshansk]

Objective

This is an umbrella ticket to capture references and discussion around knowledge transfer of the Consensus module. The primary focus is to onboard a new team member to co-own the module.

1. Origin Document

• Development Guides - Make sure to start here
• Consensus module source code - Make sure to look here
• V1 Consensus Specification - Make sure to read here

2. Knowledge Transfer FAQ

Where is the consensus module today?

We've got a basic version of 3-phase Hotstuff w/ deterministic round robin leader election driving the entire blockchain

What's the (#1) biggest known missing gap?

State sync is not implemented yet. The most basic version of it is in review in #812 and #874.

Note that this implementation uses the FSM (Finite State Machine) but we have decided to remove it as it is adding more complexity than simplicity to the codebase. We plan to make an ADR for this and likely remove it in #821.

What's the (#2) second biggest known missing gap?

The libraries for VRF leader election are in place but have not been integrated into the rest of the codebase.

We use a basic round-robin approach right now.

What's the (#3) third biggest known missing gap?

We haven't implemented penalties double signing yet, or any mechanism related to "evidence" in Tendermint.

There's also a ton of opportunity for this like inactivity leaks and implementing other penalties since we're building our own L1 ;)

What's an (???) unknown gap?

Our block & signature validation process. It's not documented well and there are not sufficient tests in place.

Points of contact for related ongoing work

Who is currently building p2p discovery, churn and gossip?

@bryanchriswhite 🌐

Who is currently building state important, export, rollback, validation and guaranteeing efficient data integrity?

@dylanlott 💾

Who is currently thinking about IBC and related light client implementations?

@h5law 🌲

Who is building & maintaining the infrastructure for all of this?

@okdas 🏗️

Who is dependant on this to enable E2E relays?

@adshmh 🚄

3. HotPOKT

The following is a screenshot from another thread (p2p related) we had just to show how hotstuff linear complexity works and how it relates to other types of gossip.

4. Goals

• Get the consensus module to a TestNet ready stage
• Leverage the perspective of someone new in order to update the tooling & documentation
• Identify and resolve low-hanging fruit
• Complete a few starter tasks and one major task
• Determine the missing gaps and define a path forward for HotPOKT
• Identify a co-owner and additional Point of Contact (POC) for the V1 Consensus module
• Outline & document follow-up work & tasks to build & scale the Consensus module

4.1 Starter Tasks

• [Consensus] When starting LocalNet from scratch, the first TriggerNextView leads to height 2 and not 1 #441
• [Consensus] Pacemaker - MinimumBlockTime Configuration #705
• [Consensus] Remove the dependency of state sync on the FSM #821
  • Blocked by [WIP] Block by Block Synchronous Sate Sync #812 and [E2E Test] Utilities for State Sync Test #874

4.2 Required Deliverables

• [Consensus] Leader Election - Productionize the sortition algorithm #324
• [WIP][Consensus] Leader Election - Round Robin (secondary) #650
• [Consensus] Penalize validators for double signing #432
• [DOCUMENT] Add Block Validation Process Documentation #469
• Create a ticket to scope, track & implement inactivity leaks
• Create tickets to scope future implementations / improvements of state sync

4.3 Stretch Deliverables

• [CONSENSUS] Add Quorum Certificate verification tests #615
• [Consensus] Fast Sync - Block Chunks #669
• [CONSENSUS] Fix (pseudo)random ranges in TestSortition #470
• [Consensus] BLS Signature Aggregation #109
• [CONSENSUS] Generate asynchronous event channel for state synchronization  #618

4.4 Non-Deliverables

• [SPIKE] State Sync Explorations w/ LibP2P and w/o the FSM #816
• [CONSENSUS] Super Validators #538
• [TESTING] Add byzantine scenario tests to Consensus module #473
• [Consensus] Make interaction among pacemaker and consensus modules asynchronous #428
• [Consensus] Common bus may be a potential attack surface and performance bottleneck #415
• [Consensus] Twins: BFT Systems Made Robust #108
• [Consensus] BLS Signature Aggregation #109

4.5 Non-goals

• Getting the consensus module to a MainNet-ready stage
• Updating/re-writing the Consensus specification; though this will need to be done in the future

---

Creator: @Olshansk
Co-Owners: @red-0ne

[Olshansk]

@red-0ne Friendly reminder that if you're ever blocked on something or waiting on someone, this should capture the first 3 months of onboarding in terms of goals & deliverables.