feat: finality gadget server (#13)

* docs: add readme

* feat: scaffold initial services

* docs: update readme

* chore: remove old comments

* feat: add mutex for db write operations

* feat: convert db write to tx with rollback

* chore: rename main -> demo

* fix: update verifier for sdk configs

* feat: run server inside verifier

* fix: remove hardcode sql script

* fix: sql to handle query non locally stored blocks

* Revert "fix: sql to handle query non locally stored blocks"

This reverts commit 033ba5a.

* fix: remove incorrect error check

* debug: add test messages

* fix: handle latest finalised block 0 exception

* fix: log errors in demo process

* fix: move server port and pg conn into configs

* feat: add `processNBlocks` for testing, improve block handling on restart

* fix: handle non-existent local block

* chore: remove debugging logs

* chore: fix lint errors

* feat: refactor to bbolt db

* fix: throw if block height is 0

* docs: update verifier readme

* fix: gitignore .db

* feat: create cli daemon and config file

* feat: improve logging

* chore: consolidate verifier code

* fix: lint errors

* fix: handle server error

* fix: gracefully handle server shutdown

* fix: gracefully shutdown verifier

* fix: remove ProcessNBlocks

* fix: handle stop error

* fix: handle non-existent latest consecutively finalized block

* chore: rm deprecated pg handler

* chore: fix formatting

* fix: remove .db

* chore: use relative path for db

* docs: update readme

* chore: fix formatting

* chore: clean unused deps

* feat: remove duplicate tracker for consecutively finalized block

* chore: restructure verifier as part of sdk

* fix: remove isFinalized bool from db type

* feat: refactor http -> grpc and rename verifier -> server

* feat: refactor http -> grpc and rename verifier -> server

* fix: deps

* fix: remove grpc server ping to fix name conflict

* debug: add logging

* debug: add more logging and bug fixes

* debug: add err logging for rpc client

* debug: use fmt for logging

* Revert "debug: use fmt for logging"

This reverts commit fc7ba0f.

* Revert "debug: add err logging for rpc client"

This reverts commit 910f07b.

* Revert "debug: add more logging and bug fixes"

This reverts commit faab534.

* Revert "debug: add logging"

This reverts commit 370fce1.

* feat: add method to delete local DB

* fix: delete db path

* feat: allow cancel ProcessBlocks with ctx

* chore: remove print

* update package name (#11)

* fix: update package name

* fix: block status queries

* feat: rename daemon and add build instructions

* fix: always persist DB state

* chore: rename finality gadget

* feat: normalise block hash for querying

* test: add db unit tests

* feat: refactor repo and merge finalitygadget <> sdk

* test: regenerate mocks

* chore: cleanup code and standardise client format

* test: regenerate mocks

* feat: use zap logger

* docs: update readme

* feat: add getBlockByHash route to db

* feat: add get block routes to finalitygadget

* chore: consolidate errors

* test: add fg unit tests

* feat: add logger

* fix: lint errors

* docs: add logger to readme

* restore expected_clients

* remove unused quit chan

* feat: simplify grpc insertBlock response

* chore: config.toml.example

* chore: rename fns for consistency

* nit: fix comment

* remove DeleteDB api

* chore: rename fns for consistency

* remove InsertBlock from grpc

* nit: fix comments

* chore: rename fns for consistency

---------

Co-authored-by: lesterli <[email protected]>

.gitignore

@@ -1,2 +1,6 @@
 **/target/
-.DS_Store
+.DS_Store
+**/.env
+**/*.db
+*.db
+config.toml

Makefile

@@ -2,6 +2,8 @@
 
 MOCKS_DIR=./testutil/mocks
 
+OPFGD_PKG := github.com/babylonlabs-io/finality-gadget/cmd/opfgd
+
 mock-gen:
 	go install go.uber.org/mock/mockgen@latest
 	mockgen -source=sdk/client/expected_clients.go -package mocks -destination $(MOCKS_DIR)/expected_clients_mock.go
@@ -11,4 +13,7 @@ test:
 	go test -race ./... -v
 
 lint:
-	golangci-lint run
+	golangci-lint run
+
+install:
+	go install -trimpath $(OPFGD_PKG)

README.md

@@ -1,19 +1,86 @@
 # Babylon Finality Gadget
 
-We proposed a [Babylon finality gadget](https://github.com/ethereum-optimism/specs/discussions/218) for OP-stack chains. The finality gadget depends on the EOTS data in a CosmWasm contract deployed on the Babylon chain. 
+The Babylon Finality Gadget is a program that can be run by users of OP stack L2s to track consecutive L2 block quorum and query the BTC-finalised status of blocks.
 
-We have modified the OP-stack codebase to use the SDK in this codebase for additional finalty checks.
+See our [proposal](https://github.com/ethereum-optimism/specs/discussions/218) on Optimism for more details.
 
-In the future, we will also move the CosmWasm contract code here.
+## Modules
 
-## Dependencies
+- `cmd` : entry point for `opfgd` finality gadget daemon
+- `finalitygadget` : top-level umbrella module that exposes query methods and coordinates calls to other clients
+- `client` : grpc client to query the finality gadget
+- `server` : grpc server for the finality gadget
+- `proto` : protobuf definitions for the grpc server
+- `config` : configs for the finality gadget
+- `btcclient` : wrapper around Bitcoin RPC client
+- `bbnclient` : wrapper around Babylon RPC client
+- `ethl2client` : wrapper around OP stack L2 ETH RPC client
+- `cwclient` : client to query CosmWasm smart contract deployed on BabylonChain
+- `db` : handler for local database to store finalized block state
+- `types` : common types
+- `log` : custom logger
+- `testutil` : test utilities and helpers
 
-The SDK requires a BTC RPC client defined in https://github.com/btcsuite/btcd/tree/master/rpcclient. We wrap it in our own BTC Client to make it easier to use.
+## Instructions
 
-## Usages
+### Download and configuration
 
-To run tests
+To get started, clone the repository.
 
+```bash
+git clone https://github.com/babylonlabs-io/finality-gadget.git
 ```
+
+Copy the `config.toml.example` file to `config.toml`:
+
+```bash
+cp config.toml.example config.toml
+```
+
+Configure the `config.toml` file with the following parameters:
+
+```toml
+L2RPCHost = # RPC URL of OP stack L2 chain
+BitcoinRPCHost = # Bitcoin RPC URL
+DBFilePath = # Path to local bbolt DB file
+FGContractAddress = # Babylon finality gadget contract address
+BBNChainID = # Babylon chain id
+BBNRPCAddress = # Babylon RPC host URL
+GRPCServerPort = # Port to run the gRPC server on
+PollInterval = # Interval to poll for new L2 blocks
+```
+
+### Building and installing the binary
+
+At the top-level directory of the project
+
+```bash
+make install
+```
+
+The above command will build and install the `opfgd` binary to
+`$GOPATH/bin`.
+
+If your shell cannot find the installed binaries, make sure `$GOPATH/bin` is in
+the `$PATH` of your shell. Usually these commands will do the job
+
+```bash
+export PATH=$HOME/go/bin:$PATH
+echo 'export PATH=$HOME/go/bin:$PATH' >> ~/.profile
+```
+
+### Running the daemon
+
+To start the daemon, run:
+
+```bash
+opfgd start --cfg config.toml
+```
+
+### Running tests
+
+To run tests:
+
+```bash
 make test
 ```

sdk/bbnclient/bbnclient.go → bbnclient/bbnclient.go

@@ -4,15 +4,29 @@ import (
 	"math"
 
 	"github.com/babylonlabs-io/babylon/client/query"
-	"github.com/babylonlabs-io/babylon/x/btcstaking/types"
+	bbntypes "github.com/babylonlabs-io/babylon/x/btcstaking/types"
 	sdkquerytypes "github.com/cosmos/cosmos-sdk/types/query"
 )
 
-type Client struct {
+type BabylonClient struct {
 	*query.QueryClient
 }
 
-func (bbnClient *Client) QueryAllFpBtcPubKeys(consumerId string) ([]string, error) {
+//////////////////////////////
+// CONSTRUCTOR
+//////////////////////////////
+
+func NewBabylonClient(queryClient *query.QueryClient) *BabylonClient {
+	return &BabylonClient{
+		QueryClient: queryClient,
+	}
+}
+
+//////////////////////////////
+// METHODS
+//////////////////////////////
+
+func (bbnClient *BabylonClient) QueryAllFpBtcPubKeys(consumerId string) ([]string, error) {
 	pagination := &sdkquerytypes.PageRequest{}
 	resp, err := bbnClient.QueryClient.QueryConsumerFinalityProviders(consumerId, pagination)
 	if err != nil {
@@ -27,7 +41,7 @@ func (bbnClient *Client) QueryAllFpBtcPubKeys(consumerId string) ([]string, erro
 	return pkArr, nil
 }
 
-func (bbnClient *Client) QueryFpPower(fpPubkeyHex string, btcHeight uint64) (uint64, error) {
+func (bbnClient *BabylonClient) QueryFpPower(fpPubkeyHex string, btcHeight uint64) (uint64, error) {
 	totalPower := uint64(0)
 	pagination := &sdkquerytypes.PageRequest{}
 	// queries the BTCStaking module for all delegations of a finality provider
@@ -58,7 +72,7 @@ func (bbnClient *Client) QueryFpPower(fpPubkeyHex string, btcHeight uint64) (uin
 	return totalPower, nil
 }
 
-func (bbnClient *Client) QueryMultiFpPower(
+func (bbnClient *BabylonClient) QueryMultiFpPower(
 	fpPubkeyHexList []string,
 	btcHeight uint64,
 ) (map[string]uint64, error) {
@@ -76,7 +90,7 @@ func (bbnClient *Client) QueryMultiFpPower(
 }
 
 // QueryEarliestActiveDelBtcHeight returns the earliest active BTC staking height
-func (bbnClient *Client) QueryEarliestActiveDelBtcHeight(fpPkHexList []string) (uint64, error) {
+func (bbnClient *BabylonClient) QueryEarliestActiveDelBtcHeight(fpPkHexList []string) (uint64, error) {
 	allFpEarliestDelBtcHeight := uint64(math.MaxUint64)
 
 	for _, fpPkHex := range fpPkHexList {
@@ -92,7 +106,7 @@ func (bbnClient *Client) QueryEarliestActiveDelBtcHeight(fpPkHexList []string) (
 	return allFpEarliestDelBtcHeight, nil
 }
 
-func (bbnClient *Client) QueryFpEarliestActiveDelBtcHeight(fpPubkeyHex string) (uint64, error) {
+func (bbnClient *BabylonClient) QueryFpEarliestActiveDelBtcHeight(fpPubkeyHex string) (uint64, error) {
 	pagination := &sdkquerytypes.PageRequest{
 		Limit: 100,
 	}
@@ -144,14 +158,69 @@ func (bbnClient *Client) QueryFpEarliestActiveDelBtcHeight(fpPubkeyHex string) (
 	return earliestBtcHeight, nil
 }
 
+//////////////////////////////
+// INTERNAL
+//////////////////////////////
+
+// we implemented exact logic as in GetStatus
+// https://github.com/babylonlabs-io/babylon-private/blob/3d8f190c9b0c0795f6546806e3b8582de716cd60/x/btcstaking/types/btc_delegation.go#L90-L111
+func (bbnClient *BabylonClient) isDelegationActive(
+	btcDel *bbntypes.BTCDelegationResponse,
+	btcHeight uint64,
+) (bool, error) {
+	btccheckpointParams, err := bbnClient.QueryClient.BTCCheckpointParams()
+	if err != nil {
+		return false, err
+	}
+	btcstakingParams, err := bbnClient.QueryClient.BTCStakingParams()
+	if err != nil {
+		return false, err
+	}
+	kValue := btccheckpointParams.GetParams().BtcConfirmationDepth
+	wValue := btccheckpointParams.GetParams().CheckpointFinalizationTimeout
+	covQuorum := btcstakingParams.GetParams().CovenantQuorum
+	ud := btcDel.UndelegationResponse
+
+	if len(ud.GetDelegatorUnbondingSigHex()) > 0 {
+		return false, nil
+	}
+
+	// k is not involved in the `GetStatus` logic as Babylon will accept a BTC delegation request
+	// only when staking tx is k-deep on BTC.
+	//
+	// But the msg handler performs both checks 1) ensure staking tx is k-deep, and 2) ensure the
+	// staking tx's timelock has at least w BTC blocks left.
+	// (https://github.com/babylonlabs-io/babylon-private/blob/3d8f190c9b0c0795f6546806e3b8582de716cd60/x/btcstaking/keeper/msg_server.go#L283-L292)
+	//
+	// So after the msg handler accepts BTC delegation the 1st check is no longer needed
+	// the k-value check is added per
+	//
+	// So in our case, we need to check both to ensure the delegation is active
+	if btcHeight < btcDel.StartHeight+kValue || btcHeight+wValue > btcDel.EndHeight {
+		return false, nil
+	}
+
+	if uint32(len(btcDel.CovenantSigs)) < covQuorum {
+		return false, nil
+	}
+	if len(ud.CovenantUnbondingSigList) < int(covQuorum) {
+		return false, nil
+	}
+	if len(ud.CovenantSlashingSigs) < int(covQuorum) {
+		return false, nil
+	}
+
+	return true, nil
+}
+
 // The active delegation needs to satisfy:
 // 1) the staking tx is k-deep in Bitcoin, i.e., start_height + k
 // 2) it receives a quorum number of covenant committee signatures
 //
 // return math.MaxUint64 if the delegation is not active
 //
 // Note: the delegation can be unbounded and that's totally fine and shouldn't affect when the chain was activated
-func getDelFirstActiveHeight(btcDel *types.BTCDelegationResponse, latestBtcHeight, kValue uint64, covQuorum uint32) uint64 {
+func getDelFirstActiveHeight(btcDel *bbntypes.BTCDelegationResponse, latestBtcHeight, kValue uint64, covQuorum uint32) uint64 {
 	activationHeight := btcDel.StartHeight + kValue
 	// not activated yet
 	if latestBtcHeight < activationHeight || uint32(len(btcDel.CovenantSigs)) < covQuorum {

sdk/btcclient/client.go → btcclient/btcclient.go

@@ -10,30 +10,38 @@ import (
 	"go.uber.org/zap"
 )
 
-type BTCClient struct {
+type BitcoinClient struct {
 	client *rpcclient.Client
 	logger *zap.Logger
 	cfg    *BTCConfig
 }
 
-func NewBTCClient(cfg *BTCConfig, logger *zap.Logger) (*BTCClient, error) {
+//////////////////////////////
+// CONSTRUCTOR
+//////////////////////////////
+
+func NewBitcoinClient(cfg *BTCConfig, logger *zap.Logger) (*BitcoinClient, error) {
 	c, err := rpcclient.New(cfg.ToConnConfig(), nil)
 	if err != nil {
 		return nil, err
 	}
 
-	return &BTCClient{
+	return &BitcoinClient{
 		client: c,
 		logger: logger,
 		cfg:    cfg,
 	}, nil
 }
 
+//////////////////////////////
+// METHODS
+//////////////////////////////
+
 type BlockCountResponse struct {
 	count int64
 }
 
-func (c *BTCClient) GetBlockCount() (uint64, error) {
+func (c *BitcoinClient) GetBlockCount() (uint64, error) {
 	callForBlockCount := func() (*BlockCountResponse, error) {
 		count, err := c.client.GetBlockCount()
 		if err != nil {
@@ -51,7 +59,7 @@ func (c *BTCClient) GetBlockCount() (uint64, error) {
 	return uint64(blockCount.count), nil
 }
 
-func (c *BTCClient) GetBlockHashByHeight(height uint64) (*chainhash.Hash, error) {
+func (c *BitcoinClient) GetBlockHashByHeight(height uint64) (*chainhash.Hash, error) {
 	callForBlockHash := func() (*chainhash.Hash, error) {
 		return c.client.GetBlockHash(int64(height))
 	}
@@ -64,7 +72,7 @@ func (c *BTCClient) GetBlockHashByHeight(height uint64) (*chainhash.Hash, error)
 	return blockHash, nil
 }
 
-func (c *BTCClient) GetBlockHeaderByHash(blockHash *chainhash.Hash) (*wire.BlockHeader, error) {
+func (c *BitcoinClient) GetBlockHeaderByHash(blockHash *chainhash.Hash) (*wire.BlockHeader, error) {
 	callForBlockHeader := func() (*wire.BlockHeader, error) {
 		return c.client.GetBlockHeader(blockHash)
 	}
@@ -77,7 +85,7 @@ func (c *BTCClient) GetBlockHeaderByHash(blockHash *chainhash.Hash) (*wire.Block
 	return header, nil
 }
 
-func (c *BTCClient) GetBlockHeightByTimestamp(targetTimestamp uint64) (uint64, error) {
+func (c *BitcoinClient) GetBlockHeightByTimestamp(targetTimestamp uint64) (uint64, error) {
 	// get the height of the most-work fully-validated chain
 	blockHeight, err := c.GetBlockCount()
 	if err != nil {
@@ -113,7 +121,7 @@ func (c *BTCClient) GetBlockHeightByTimestamp(targetTimestamp uint64) (uint64, e
 	return lowerBound - 1, nil
 }
 
-func (c *BTCClient) GetBlockTimestampByHeight(height uint64) (uint64, error) {
+func (c *BitcoinClient) GetBlockTimestampByHeight(height uint64) (uint64, error) {
 	// get block hash by height
 	blockHash, err := c.GetBlockHashByHeight(height)
 	if err != nil {
@@ -129,6 +137,10 @@ func (c *BTCClient) GetBlockTimestampByHeight(height uint64) (uint64, error) {
 	return uint64(blockHeader.Timestamp.Unix()), nil
 }
 
+//////////////////////////////
+// INTERNAL
+//////////////////////////////
+
 func clientCallWithRetry[T any](
 	call retry.RetryableFuncWithData[*T], logger *zap.Logger, cfg *BTCConfig,
 ) (*T, error) {

sdk/btcclient/client_test.go → btcclient/btcclient_test.go

@@ -3,8 +3,8 @@ package btcclient
 import (
 	"testing"
 
+	"github.com/babylonlabs-io/finality-gadget/log"
 	"github.com/stretchr/testify/require"
-	"go.uber.org/zap"
 )
 
 // TODO: 1) not rely on mainnet RPC; 2) add more tests for some other edge cases
@@ -13,12 +13,12 @@ func TestBtcClient(t *testing.T) {
 	var err error
 
 	// Create logger.
-	logger, err := zap.NewDevelopment()
+	logger, err := log.NewRootLogger("console", true)
 	require.Nil(t, err)
 
 	// Create BTC client
 	btcConfig := DefaultBTCConfig()
-	btc, err := NewBTCClient(btcConfig, logger)
+	btc, err := NewBitcoinClient(btcConfig, logger)
 	require.Nil(t, err)
 
 	// timestmap between block 848682 and 848683

sdk/btcclient/config.go → btcclient/config.go

@@ -47,7 +47,6 @@ func DefaultBTCConfig() *BTCConfig {
 	}
 }
 
-
 func (cfg *BTCConfig) ToConnConfig() *rpcclient.ConnConfig {
 	return &rpcclient.ConnConfig{
 		Host:                 cfg.RPCHost,
@@ -60,4 +59,4 @@ func (cfg *BTCConfig) ToConnConfig() *rpcclient.ConnConfig {
 		// we may need to re-consider it later if we need any notifications
 		HTTPPostMode: true,
 	}
-}
+}