Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/oracle reactor #1

Merged
merged 151 commits into from
Jun 11, 2024
Merged
Show file tree
Hide file tree
Changes from 135 commits
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
ff92c2c
feat: add application folder to compact
holyxiaoxin Aug 16, 2023
21fbb31
feat: ability to specify db-names for experimental-compact-goleveldb
holyxiaoxin Aug 22, 2023
ea9fac2
port over oracle service
yan-soon Feb 1, 2024
c507a87
finishing porting over oracle service and start storing votes
yan-soon Feb 6, 2024
9bc89f8
implement pruning, gossip and partial signing svc
yan-soon Feb 16, 2024
4d5a70b
rehaul gossiping and pruning and add new unsigned vote buffer
yan-soon Feb 20, 2024
7d5e05f
add notes
yan-soon Feb 20, 2024
8fd47e5
implement oracle signing svc
yan-soon Feb 21, 2024
945b2fa
remove unused code from oracle reactor
yan-soon Feb 21, 2024
37aa8e1
update and test gossip votes among cluster of vals
yan-soon Feb 23, 2024
be0f2c5
add new proto for list of gossipVotes
yan-soon Feb 23, 2024
d4339f0
Revert "add new proto for list of gossipVotes"
yan-soon Feb 23, 2024
c3501d7
test update buf build name, for exporting
yan-soon Feb 23, 2024
a6aa022
add SignGossipVote hook to proxyApp interface
yan-soon Feb 26, 2024
820ce18
use http client to query oracle and results, instead of grpc
yan-soon Feb 28, 2024
2e2f0ed
shift proto file and update package name to prevent clash with carbon
yan-soon Feb 28, 2024
8407ba7
remove unused grpc client from oracle service
yan-soon Mar 5, 2024
9cd2e53
update oracle proto, for proposer verification
yan-soon Mar 7, 2024
0262f41
add logs for validator set
yan-soon Mar 7, 2024
c14385f
remove val set logs
yan-soon Mar 7, 2024
d4f658a
fix bug where tx wasnt included in block
yan-soon Mar 7, 2024
ab6d575
account size of injected MsgCreateResult tx in block prop
yan-soon Mar 7, 2024
c96342c
account for maxTxDataBytes when inserting oracle tx in to block prop
yan-soon Mar 8, 2024
c7e19e4
inject oracle results tx into response of prepare proposal instead
yan-soon Mar 8, 2024
e6f036d
Revert "inject oracle results tx into response of prepare proposal in…
yan-soon Mar 8, 2024
6a76447
adding logs for debugging
yan-soon Mar 11, 2024
25375f6
add string compression
Mar 12, 2024
9157efb
test with larger max tx size for oracle gossip
Mar 12, 2024
ded309f
add config oracle reactor
Mar 12, 2024
9b9955c
cut down on oracle msg size
Mar 13, 2024
4315a35
add back validator field to oracle vote
Mar 13, 2024
2e8c0fe
add back validator field to gossipVote
Mar 13, 2024
3994d10
add rest endpoint check before starting up oracle service
Mar 13, 2024
37fa715
shift rest endpoint check to within go routine
Mar 13, 2024
6f52e10
add grpc health check before starting oracle service
Mar 13, 2024
af0c3f9
update default grpc addr
Mar 13, 2024
a75f1b9
add validator check during gossiping, add gossipVoteBuffer pruner
Mar 14, 2024
4ede0a5
add logs for validator checking on gossip
Mar 14, 2024
f875f13
add more logs for debugging
Mar 14, 2024
41b3fda
add gossip interval to config
Mar 15, 2024
4e037d4
merge v0.38.5
Mar 22, 2024
bbbc29a
remove oracle service
Apr 16, 2024
5ddda97
add new prepareOracleVotes hook
Apr 16, 2024
617f655
add prepareOracleVotes impl
Apr 16, 2024
5d9a887
test new flow
Apr 17, 2024
66e1120
impl inject oracleTx in prepareProposal and validate oracleTx in proc…
Apr 18, 2024
637ceef
fix index txs index out of range bug
Apr 18, 2024
d99d272
update proto
Apr 18, 2024
1dcdefd
fix bug with proto indexing
Apr 18, 2024
8149b71
add validateOracleVotes hook
Apr 18, 2024
7ba2168
add back signGossipVote hook
Apr 18, 2024
c2f9b72
add logs
Apr 19, 2024
671ffca
remove ununsed code and fix bug w process proposal
Apr 19, 2024
3392e0a
add logs
Apr 21, 2024
8618a17
add more logs
Apr 21, 2024
7e742a3
clean unused proto and fix uint64 overflow bug
Apr 21, 2024
7d2ae5c
update config, remove unsignedBuffer, change getting of oracleVotes t…
Apr 23, 2024
5ede911
fix prepareOracleVotes bug
Apr 23, 2024
ec44936
add logs
Apr 23, 2024
8651c11
include signedTimeStamp in signedVote
Apr 23, 2024
38b8196
test w pub key
Apr 26, 2024
80687bd
Revert "test w pub key"
Apr 26, 2024
a5026bc
add back unsigned buffer
Apr 29, 2024
3d218ff
remove signType from gossipVote
Apr 29, 2024
19d31dd
convert oracleId type to int64
Apr 29, 2024
a067ade
test logs
Apr 29, 2024
f7083a3
test sort
Apr 29, 2024
a161e3e
fix bug where gossipBuffer was not pruning properly
Apr 29, 2024
cd285d3
revert oracleId type to int64
Apr 29, 2024
7091d6c
remove some logs
Apr 30, 2024
1fb4a5c
remove more logs
Apr 30, 2024
7144c7f
fix bug for verifying vals in gossip
Apr 30, 2024
afc8c9b
update configs for maxGossipAge
Apr 30, 2024
6d3c735
remove unused code
Apr 30, 2024
b691924
optimise gossip slightly
Apr 30, 2024
7cbec64
fix bug where invalid gossipVotes are not removed during validation
Apr 30, 2024
f3f5ee5
improve naming for abci oracle hook methods
May 2, 2024
42fab7f
update val address type to string
May 2, 2024
7ef4c20
fix bug with prepareProposal when nil oracleCreateTx is found
May 2, 2024
5ac3eba
update validateOracleVotes response
May 2, 2024
62aef7d
test without validator index for gossip votes
May 2, 2024
664182f
test with both val index and val addr in gossipedVotes
May 3, 2024
cb5abc0
add val index as part of signature
May 3, 2024
8de22c9
add logs
May 3, 2024
efb590d
improve logs and add check to not gossip stale votes
May 3, 2024
b4640f2
improve check for stale gossip votes on gossip protocol
May 3, 2024
bd45f6c
fix nil bug on gossip
May 3, 2024
4410992
simplify pruning for vote buffers
May 3, 2024
ee4638f
edit prune interval
May 3, 2024
bcb8cd1
test without pruning gossip buffer
May 3, 2024
782e945
regen proto
May 4, 2024
a554af9
add test logs
May 4, 2024
c19c793
test without pruning
May 4, 2024
f22af4c
test gossiping without validator addr
May 4, 2024
d2b1ecf
test with gossip backoff
May 4, 2024
30ee748
more tests
May 4, 2024
4484f3d
test bug fix with votes in unsignedBuffer
May 5, 2024
8abbfbd
test with validator field as bytes
May 5, 2024
c16062f
test without sort
May 5, 2024
3b11cbf
revert back to no validator field test
May 6, 2024
455b252
test with sort again
May 6, 2024
7894d8c
test sort fix
May 6, 2024
ee9dd54
add back validator field for gossipedVotes struct
May 6, 2024
884ee67
test no sort, with validator field
May 6, 2024
9d8eb96
shift oracle validation logic
May 6, 2024
2d973d7
test with updated sort
May 6, 2024
117174e
update sort func to be stricter
May 6, 2024
a3f55e8
test with stable sort
May 7, 2024
241668b
add logs for gossip
May 7, 2024
00a4e74
fix sorting bug
May 7, 2024
e692837
add pruning logs
May 7, 2024
c21f101
sort vote by timestamp first
May 7, 2024
4c67e19
remove validateOracleVotes hook
May 7, 2024
127a875
remove testing logs
May 7, 2024
197c06a
remove validator field from gossipedVotes
May 7, 2024
361b4d4
Revert "remove validator field from gossipedVotes"
May 7, 2024
0448cb5
add config for prune interval, revert go.mod go.sum
May 8, 2024
d7bbd24
fix logs in oracle reactor and remove unused vars
May 8, 2024
4b81008
remove unused fields from createOracleResultTx hook
May 9, 2024
8c4eef2
add back validateOracleVotes hook
May 9, 2024
3144a6b
resolve comments and minor fixes
May 9, 2024
2ee5578
update fetchOracleVotes hook to take in multiple votes at once
May 10, 2024
3f70d30
fix bug with sign votes channel
May 10, 2024
1e4948b
re-implement blocking on fetchOracleVotes
May 10, 2024
14a668d
resolve more comments
May 11, 2024
2d78b6f
fix bug with appending last block timestamps
May 11, 2024
ce8eb4a
add proposerAddr in createOracleResultTx hook
May 11, 2024
7e1eb94
add hard cap for maxVoteAge in the event the blocks dont move
May 11, 2024
0ea5232
remove val index from gossipedVotes
May 12, 2024
49d88a5
update context for validateOracleVotes
May 16, 2024
9c49a87
fix logging and add exit logic for p2p on oracle reactor
May 21, 2024
fb1d6a8
comment out exit logic for peer
May 21, 2024
11ee8cf
test ctx, remove later
May 21, 2024
3af0b69
Revert "test ctx, remove later"
May 23, 2024
3d26f4f
update stop peer logic for oracle vote gossip
May 26, 2024
3051fd6
fix comments and remove unused proto imports
May 28, 2024
1b1ea81
fix compile errors
May 28, 2024
8f73c09
add maxOracleGossipAge as config
May 28, 2024
58ab1be
shift order of insertion of oracle result tx
May 31, 2024
407f495
add logs for mutex locking and unlocking
Jun 3, 2024
f10d307
remove logs for unsigned vote buffer
Jun 3, 2024
a2c8967
add clearer logs for tracking mutex lock times
Jun 3, 2024
a1e4b70
improve logs for benchmarking lock times
Jun 3, 2024
4c71a0a
optimise locking for sending gossip routine
Jun 3, 2024
f178e00
improve lock logs
Jun 3, 2024
9a55922
increase threshold for lock timings
Jun 4, 2024
baced90
update configs for oracle reactor
Jun 4, 2024
ea429b7
remove unnecessary check for pruning of oracle votes
Jun 4, 2024
3babce3
check for block timestamp buffer size before pruning
Jun 4, 2024
c3438e4
rename oracle config to oraclesvc to prevent clashing with carbon ora…
Jun 5, 2024
d947aab
add less aggresive configs and prune dup votes
Jun 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ GOOS ?= linux
GOARCH ?= amd64
GOARM ?=

DOCKER := $(shell which docker)

ifeq (linux/arm,$(findstring linux/arm,$(TARGETPLATFORM)))
GOOS=linux
GOARCH=arm
Expand Down
12 changes: 12 additions & 0 deletions abci/client/grpc_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -245,3 +245,15 @@ func (cli *grpcClient) VerifyVoteExtension(ctx context.Context, req *types.Reque
func (cli *grpcClient) FinalizeBlock(ctx context.Context, req *types.RequestFinalizeBlock) (*types.ResponseFinalizeBlock, error) {
return cli.client.FinalizeBlock(ctx, types.ToRequestFinalizeBlock(req).GetFinalizeBlock(), grpc.WaitForReady(true))
}

func (cli *grpcClient) CreateOracleResultTx(ctx context.Context, req *types.RequestCreateOracleResultTx) (*types.ResponseCreateOracleResultTx, error) {
return cli.client.CreateOracleResultTx(ctx, types.ToRequestCreateOracleResultTx(req).GetCreateOracleResultTx(), grpc.WaitForReady(true))
}

func (cli *grpcClient) FetchOracleVotes(ctx context.Context, req *types.RequestFetchOracleVotes) (*types.ResponseFetchOracleVotes, error) {
return cli.client.FetchOracleVotes(ctx, types.ToRequestFetchOracleVotes(req).GetFetchOracleVotes(), grpc.WaitForReady(true))
}

func (cli *grpcClient) ValidateOracleVotes(ctx context.Context, req *types.RequestValidateOracleVotes) (*types.ResponseValidateOracleVotes, error) {
return cli.client.ValidateOracleVotes(ctx, types.ToRequestValidateOracleVotes(req).GetValidateOracleVotes(), grpc.WaitForReady(true))
}
78 changes: 78 additions & 0 deletions abci/client/mocks/client.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

39 changes: 39 additions & 0 deletions abci/client/socket_client.go
Original file line number Diff line number Diff line change
Expand Up @@ -412,6 +412,39 @@ func (cli *socketClient) FinalizeBlock(ctx context.Context, req *types.RequestFi
return reqRes.Response.GetFinalizeBlock(), cli.Error()
}

func (cli *socketClient) CreateOracleResultTx(ctx context.Context, req *types.RequestCreateOracleResultTx) (*types.ResponseCreateOracleResultTx, error) {
reqRes, err := cli.queueRequest(ctx, types.ToRequestCreateOracleResultTx(req))
if err != nil {
return nil, err
}
if err := cli.Flush(ctx); err != nil {
return nil, err
}
return reqRes.Response.GetCreateOracleResultTx(), cli.Error()
}

func (cli *socketClient) FetchOracleVotes(ctx context.Context, req *types.RequestFetchOracleVotes) (*types.ResponseFetchOracleVotes, error) {
reqRes, err := cli.queueRequest(ctx, types.ToRequestFetchOracleVotes(req))
if err != nil {
return nil, err
}
if err := cli.Flush(ctx); err != nil {
return nil, err
}
return reqRes.Response.GetFetchOracleVotes(), cli.Error()
}

func (cli *socketClient) ValidateOracleVotes(ctx context.Context, req *types.RequestValidateOracleVotes) (*types.ResponseValidateOracleVotes, error) {
reqRes, err := cli.queueRequest(ctx, types.ToRequestValidateOracleVotes(req))
if err != nil {
return nil, err
}
if err := cli.Flush(ctx); err != nil {
return nil, err
}
return reqRes.Response.GetValidateOracleVotes(), cli.Error()
}

func (cli *socketClient) queueRequest(ctx context.Context, req *types.Request) (*ReqRes, error) {
reqres := NewReqRes(req)

Expand Down Expand Up @@ -493,6 +526,12 @@ func resMatchesReq(req *types.Request, res *types.Response) (ok bool) {
_, ok = res.Value.(*types.Response_ProcessProposal)
case *types.Request_FinalizeBlock:
_, ok = res.Value.(*types.Response_FinalizeBlock)
case *types.Request_CreateOracleResultTx:
_, ok = res.Value.(*types.Response_CreateOracleResultTx)
yan-soon marked this conversation as resolved.
Show resolved Hide resolved
case *types.Request_ValidateOracleVotes:
_, ok = res.Value.(*types.Response_ValidateOracleVotes)
case *types.Request_FetchOracleVotes:
_, ok = res.Value.(*types.Response_FetchOracleVotes)
}
return ok
}
Expand Down
21 changes: 20 additions & 1 deletion abci/types/application.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package types

import "context"
import (
"context"
)

//go:generate ../../scripts/mockery_generate.sh Application

Expand Down Expand Up @@ -32,6 +34,11 @@ type Application interface {
OfferSnapshot(context.Context, *RequestOfferSnapshot) (*ResponseOfferSnapshot, error) // Offer a snapshot to the application
LoadSnapshotChunk(context.Context, *RequestLoadSnapshotChunk) (*ResponseLoadSnapshotChunk, error) // Load a snapshot chunk
ApplySnapshotChunk(context.Context, *RequestApplySnapshotChunk) (*ResponseApplySnapshotChunk, error) // Apply a shapshot chunk

// Hooks
CreateOracleResultTx(context.Context, *RequestCreateOracleResultTx) (*ResponseCreateOracleResultTx, error)
FetchOracleVotes(context.Context, *RequestFetchOracleVotes) (*ResponseFetchOracleVotes, error)
ValidateOracleVotes(context.Context, *RequestValidateOracleVotes) (*ResponseValidateOracleVotes, error)
}

//-------------------------------------------------------
Expand Down Expand Up @@ -117,3 +124,15 @@ func (BaseApplication) FinalizeBlock(_ context.Context, req *RequestFinalizeBloc
TxResults: txs,
}, nil
}

func (BaseApplication) CreateOracleResultTx(_ context.Context, req *RequestCreateOracleResultTx) (*ResponseCreateOracleResultTx, error) {
return &ResponseCreateOracleResultTx{}, nil
}

func (BaseApplication) FetchOracleVotes(_ context.Context, req *RequestFetchOracleVotes) (*ResponseFetchOracleVotes, error) {
return &ResponseFetchOracleVotes{}, nil
}

func (BaseApplication) ValidateOracleVotes(_ context.Context, req *RequestValidateOracleVotes) (*ResponseValidateOracleVotes, error) {
return &ResponseValidateOracleVotes{}, nil
}
24 changes: 24 additions & 0 deletions abci/types/messages.go
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,24 @@ func ToRequestFinalizeBlock(req *RequestFinalizeBlock) *Request {
}
}

func ToRequestCreateOracleResultTx(req *RequestCreateOracleResultTx) *Request {
return &Request{
Value: &Request_CreateOracleResultTx{req},
}
}

func ToRequestFetchOracleVotes(req *RequestFetchOracleVotes) *Request {
return &Request{
Value: &Request_FetchOracleVotes{req},
}
}

func ToRequestValidateOracleVotes(req *RequestValidateOracleVotes) *Request {
return &Request{
Value: &Request_ValidateOracleVotes{req},
}
}

//----------------------------------------

func ToResponseException(errStr string) *Response {
Expand Down Expand Up @@ -227,3 +245,9 @@ func ToResponseFinalizeBlock(res *ResponseFinalizeBlock) *Response {
Value: &Response_FinalizeBlock{res},
}
}

func ToResponseCreateOracleResultTx(res *ResponseCreateOracleResultTx) *Response {
return &Response{
Value: &Response_CreateOracleResultTx{res},
}
}
78 changes: 78 additions & 0 deletions abci/types/mocks/application.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading