Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[P2P] [Tooling] Peer discovery peer list subcommand #892

Open
wants to merge 43 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
4badf3a
chore: simplify debug message broadcasting
bryanchriswhite Jun 13, 2023
2b83d32
refactor: CLI config parsing
bryanchriswhite Jul 7, 2023
eac7695
refactor: common CLI helpers
bryanchriswhite Jul 7, 2023
bca000b
chore: add `GetBusFromCmd()` CLI helper
bryanchriswhite Jul 11, 2023
5e963be
chore: consistent debug CLI identity
bryanchriswhite Jul 11, 2023
a883f08
chore: add `enable_peer_discovery_debug_rpc` to P2P config
bryanchriswhite Jul 6, 2023
83c3604
chore: add P2P debug message handling support
bryanchriswhite Jul 6, 2023
6ecca53
feat: add `peer` subcommand
bryanchriswhite Jun 6, 2023
d570b35
chore: add peer `--local` persistent flag
bryanchriswhite Jul 6, 2023
e952365
feat: add `peer list` subcommand
bryanchriswhite Jun 6, 2023
e80843c
chore: implement `PeerstoreProvider#GetUnstakedPeerstore()`
bryanchriswhite Jun 5, 2023
8f90e22
chore: add `PeerstoreProvider#GetStakedPeerstoreAtCurrentHeight()`
bryanchriswhite Jul 13, 2023
6e691cd
chore: interim bootstrapping changes (pre-#859)
bryanchriswhite Jul 11, 2023
430db08
fix: gofmt
bryanchriswhite Jul 12, 2023
440b59a
chore: ensure flag and config parsing
bryanchriswhite Jul 7, 2023
fcfa837
chore: add `GetBusFromCmd()` CLI helper
bryanchriswhite Jul 11, 2023
04dc0aa
chore: consistent debug CLI identity
bryanchriswhite Jul 11, 2023
3925c71
Merge branch 'refactor/cli' into feat/peer-discovery-list
bryanchriswhite Jul 13, 2023
1bbad38
fixup: add `peer list` subcommand
bryanchriswhite Jul 13, 2023
64abbc0
add generated helm docs
invalid-email-address Jul 13, 2023
d8b6296
squash: merge refactor/cli with main
h5law Jul 25, 2023
9ecc9e5
chore: address review comments
h5law Jul 25, 2023
1cbc249
Merge branch 'main' into refactor/cli
h5law Jul 25, 2023
ffbc539
fix merge error
h5law Jul 25, 2023
0cff1d2
Merge branch 'refactor/cli' into feat/peer-discovery-list
h5law Jul 25, 2023
39af37c
revert change
h5law Jul 25, 2023
c22011c
address comments
h5law Jul 25, 2023
1fc2bb4
chore: address comments
h5law Jul 25, 2023
764e171
clarify unstaked description
h5law Jul 25, 2023
9eb5a7e
Merge branch 'main' into feat/peer-discovery-list
h5law Jul 26, 2023
7380260
chore: up retry attempts and wait time, clarify error messages
h5law Jul 26, 2023
c03aa27
Merge branch 'main' into feat/peer-discovery-list
Olshansk Jul 26, 2023
da62de1
Fixed case messaging.DebugMessageEventType
Olshansk Jul 26, 2023
12e22e1
clairfy error log messages
h5law Jul 27, 2023
f8f5da5
fix waitgroup recovery in tests
h5law Jul 27, 2023
66a1347
chore: move comment line
h5law Jul 27, 2023
870805f
add NewListCommand function to fit CLI pattern
h5law Jul 27, 2023
64ec990
clarify errors
h5law Jul 27, 2023
ccec195
remove unneeded error casting as sync panics a string
h5law Jul 27, 2023
90385f0
tidy cli
h5law Jul 28, 2023
6d0d300
remove #810 from comment as merged
h5law Jul 28, 2023
c6488a5
shortern retries
h5law Jul 28, 2023
2317d42
bug: comment out buggy lines
h5law Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 7 additions & 14 deletions app/client/cli/cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"github.com/spf13/viper"

"github.com/pokt-network/pocket/app/client/cli/flags"
"github.com/pokt-network/pocket/runtime/configs"
"github.com/pokt-network/pocket/app/client/cli/peer"
"github.com/pokt-network/pocket/runtime/defaults"
)

Expand All @@ -17,8 +17,6 @@ const (
flagBindErrFormat = "could not bind flag %q: %v"
)

var cfg *configs.Config

func init() {
rootCmd.PersistentFlags().StringVar(&flags.RemoteCLIURL, "remote_cli_url", defaults.DefaultRemoteCLIURL, "takes a remote endpoint in the form of <protocol>://<host>:<port> (uses RPC Port)")
// ensure that this flag can be overridden by the respective viper-conventional environment variable (i.e. `POCKET_REMOTE_CLI_URL`)
Expand All @@ -39,20 +37,15 @@ func init() {
if err := viper.BindPFlag("verbose", rootCmd.PersistentFlags().Lookup("verbose")); err != nil {
log.Fatalf(flagBindErrFormat, "verbose", err)
}

rootCmd.AddCommand(peer.PeerCmd)
}

var rootCmd = &cobra.Command{
Use: cliExecutableName,
Short: "Pocket Network Command Line Interface (CLI)",
Long: "The CLI is meant to be an user but also a machine friendly way for interacting with Pocket Network.",
PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
// by this time, the config path should be set
cfg = configs.ParseConfig(flags.ConfigPath)

// set final `remote_cli_url` value; order of precedence: flag > env var > config > default
flags.RemoteCLIURL = viper.GetString("remote_cli_url")
return nil
},
Use: cliExecutableName,
Short: "Pocket Network Command Line Interface (CLI)",
Long: "The CLI is meant to be an user but also a machine friendly way for interacting with Pocket Network.",
PersistentPreRunE: flags.ParseConfigAndFlags,
}

func ExecuteContext(ctx context.Context) error {
Expand Down
77 changes: 11 additions & 66 deletions app/client/cli/debug.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
package cli

import (
"errors"
"fmt"
"os"

"github.com/manifoldco/promptui"
Expand All @@ -11,10 +9,7 @@ import (

"github.com/pokt-network/pocket/app/client/cli/helpers"
"github.com/pokt-network/pocket/logger"
"github.com/pokt-network/pocket/p2p/providers/peerstore_provider"
typesP2P "github.com/pokt-network/pocket/p2p/types"
"github.com/pokt-network/pocket/shared/messaging"
"github.com/pokt-network/pocket/shared/modules"
)

// TECHDEBT: Lowercase variables / constants that do not need to be exported.
Expand Down Expand Up @@ -76,7 +71,7 @@ func NewDebugCommand() *cobra.Command {
}
}

func runDebug(cmd *cobra.Command, args []string) (err error) {
func runDebug(cmd *cobra.Command, _ []string) (err error) {
for {
if selection, err := promptGetInput(); err == nil {
handleSelect(cmd, selection)
Expand Down Expand Up @@ -164,32 +159,17 @@ func handleSelect(cmd *cobra.Command, selection string) {
}
}

// Broadcast to the entire validator set
func broadcastDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage) {
// Broadcast to the entire network.
func broadcastDebugMessage(_ *cobra.Command, debugMsg *messaging.DebugMessage) {
anyProto, err := anypb.New(debugMsg)
if err != nil {
logger.Global.Fatal().Err(err).Msg("Failed to create Any proto")
}

// TODO(olshansky): Once we implement the cleanup layer in RainTree, we'll be able to use
// broadcast. The reason it cannot be done right now is because this client is not in the
// address book of the actual validator nodes, so `validator1` never receives the message.
// p2pMod.Broadcast(anyProto)

pstore, err := fetchPeerstore(cmd)
if err != nil {
logger.Global.Fatal().Err(err).Msg("Unable to retrieve the pstore")
}
for _, val := range pstore.GetPeerList() {
addr := val.GetAddress()
if err != nil {
logger.Global.Fatal().Err(err).Msg("Failed to convert validator address into pocketCrypto.Address")
}
if err := helpers.P2PMod.Send(addr, anyProto); err != nil {
logger.Global.Error().Err(err).Msg("Failed to send debug message")
}
// TECHDEBT: prefer to retrieve P2P module from the bus instead.
if err := helpers.P2PMod.Broadcast(anyProto); err != nil {
logger.Global.Error().Err(err).Msg("Failed to broadcast debug message")
}

}

// Send to just a single (i.e. first) validator in the set
Expand All @@ -199,7 +179,7 @@ func sendDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage) {
logger.Global.Error().Err(err).Msg("Failed to create Any proto")
}

pstore, err := fetchPeerstore(cmd)
pstore, err := helpers.FetchPeerstore(cmd)
if err != nil {
logger.Global.Fatal().Err(err).Msg("Unable to retrieve the pstore")
}
Expand All @@ -210,51 +190,16 @@ func sendDebugMessage(cmd *cobra.Command, debugMsg *messaging.DebugMessage) {
}

// if the message needs to be broadcast, it'll be handled by the business logic of the message handler
//
// DISCUSS_THIS_COMMIT: The statement above is false. Using `#Send()` will only
// be unicast with no opportunity for further propagation.
validatorAddress = pstore.GetPeerList()[0].GetAddress()
if err != nil {
logger.Global.Fatal().Err(err).Msg("Failed to convert validator address into pocketCrypto.Address")
}

// TECHDEBT: prefer to retrieve P2P module from the bus instead.
if err := helpers.P2PMod.Send(validatorAddress, anyProto); err != nil {
logger.Global.Error().Err(err).Msg("Failed to send debug message")
}
}

// fetchPeerstore retrieves the providers from the CLI context and uses them to retrieve the address book for the current height
func fetchPeerstore(cmd *cobra.Command) (typesP2P.Peerstore, error) {
bus, ok := helpers.GetValueFromCLIContext[modules.Bus](cmd, helpers.BusCLICtxKey)
if !ok || bus == nil {
return nil, errors.New("retrieving bus from CLI context")
}
// TECHDEBT(#810, #811): use `bus.GetPeerstoreProvider()` after peerstore provider
// is retrievable as a proper submodule
pstoreProvider, err := bus.GetModulesRegistry().GetModule(peerstore_provider.PeerstoreProviderSubmoduleName)
if err != nil {
return nil, errors.New("retrieving peerstore provider")
}
currentHeightProvider := bus.GetCurrentHeightProvider()

height := currentHeightProvider.CurrentHeight()
pstore, err := pstoreProvider.(peerstore_provider.PeerstoreProvider).GetStakedPeerstoreAtHeight(height)
if err != nil {
return nil, fmt.Errorf("retrieving peerstore at height %d", height)
}
// Inform the client's main P2P that a the blockchain is at a new height so it can, if needed, update its view of the validator set
err = sendConsensusNewHeightEventToP2PModule(height, bus)
if err != nil {
return nil, errors.New("sending consensus new height event")
}
return pstore, nil
}

// sendConsensusNewHeightEventToP2PModule mimicks the consensus module sending a ConsensusNewHeightEvent to the p2p module
// This is necessary because the debug client is not a validator and has no consensus module but it has to update the peerstore
// depending on the changes in the validator set.
// TODO(#613): Make the debug client mimic a full node.
func sendConsensusNewHeightEventToP2PModule(height uint64, bus modules.Bus) error {
newHeightEvent, err := messaging.PackMessage(&messaging.ConsensusNewHeightEvent{Height: height})
if err != nil {
logger.Global.Fatal().Err(err).Msg("Failed to pack consensus new height event")
}
return bus.GetP2PModule().HandleEvent(newHeightEvent.Content)
}
19 changes: 19 additions & 0 deletions app/client/cli/flags/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package flags

import (
"github.com/spf13/cobra"
"github.com/spf13/viper"

"github.com/pokt-network/pocket/runtime/configs"
)

var Cfg *configs.Config

func ParseConfigAndFlags(_ *cobra.Command, _ []string) error {
// by this time, the config path should be set
Cfg = configs.ParseConfig(ConfigPath)

// set final `remote_cli_url` value; order of precedence: flag > env var > config > default
RemoteCLIURL = viper.GetString("remote_cli_url")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we do this for the other flags.go?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I intended to revert this change. This is the direction I started when investigating #891 (comment), I thought I had reverted it but apparently failed.

To answer your question, the reason for this is because viper's only integration with flags is to support setting a viper key based on the flag, but not the other way around. I.e: Viper won't update the flag value. This only applies to bound flags

This means that we have to do one of the following consistently for bound flags:

  • call viper.GetString("<flag key>") (or a helper containing it) anywhere we need the value
  • update the flag consistently and reference the flag value anywhere we need it

I opted for the latter option as I felt it was more conventional and easier to read and maintain.

return nil
}
48 changes: 48 additions & 0 deletions app/client/cli/helpers/common.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
package helpers

import (
"errors"
"fmt"

"github.com/spf13/cobra"

"github.com/pokt-network/pocket/logger"
"github.com/pokt-network/pocket/p2p/providers/peerstore_provider"
"github.com/pokt-network/pocket/p2p/types"
"github.com/pokt-network/pocket/runtime"
"github.com/pokt-network/pocket/shared/messaging"
"github.com/pokt-network/pocket/shared/modules"
)

Expand All @@ -10,5 +19,44 @@ var (
genesisPath = runtime.GetEnv("GENESIS_PATH", "build/config/genesis.json")

// P2PMod is initialized in order to broadcast a message to the local network
// TECHDEBT: prefer to retrieve P2P module from the bus instead.
P2PMod modules.P2PModule
)

// fetchPeerstore retrieves the providers from the CLI context and uses them to retrieve the address book for the current height
func FetchPeerstore(cmd *cobra.Command) (types.Peerstore, error) {
bus, err := GetBusFromCmd(cmd)
if err != nil {
return nil, err
}
// TECHDEBT(#810, #811): use `bus.GetPeerstoreProvider()` after peerstore provider
// is retrievable as a proper submodule
pstoreProvider, err := bus.GetModulesRegistry().GetModule(peerstore_provider.PeerstoreProviderSubmoduleName)
if err != nil {
return nil, errors.New("retrieving peerstore provider")
}
currentHeightProvider := bus.GetCurrentHeightProvider()
height := currentHeightProvider.CurrentHeight()
pstore, err := pstoreProvider.(peerstore_provider.PeerstoreProvider).GetStakedPeerstoreAtHeight(height)
if err != nil {
return nil, fmt.Errorf("retrieving peerstore at height %d", height)
}
// Inform the client's main P2P that a the blockchain is at a new height so it can, if needed, update its view of the validator set
err = sendConsensusNewHeightEventToP2PModule(height, bus)
if err != nil {
return nil, errors.New("sending consensus new height event")
}
return pstore, nil
}

// sendConsensusNewHeightEventToP2PModule mimicks the consensus module sending a ConsensusNewHeightEvent to the p2p module
// This is necessary because the debug client is not a validator and has no consensus module but it has to update the peerstore
// depending on the changes in the validator set.
// TODO(#613): Make the debug client mimic a full node.
func sendConsensusNewHeightEventToP2PModule(height uint64, bus modules.Bus) error {
newHeightEvent, err := messaging.PackMessage(&messaging.ConsensusNewHeightEvent{Height: height})
if err != nil {
logger.Global.Fatal().Err(err).Msg("Failed to pack consensus new height event")
}
return bus.GetP2PModule().HandleEvent(newHeightEvent.Content)
}
14 changes: 14 additions & 0 deletions app/client/cli/helpers/context.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,17 @@ package helpers

import (
"context"
"fmt"

"github.com/spf13/cobra"

"github.com/pokt-network/pocket/shared/modules"
)

const BusCLICtxKey cliContextKey = "bus"

var ErrCxtFromBus = fmt.Errorf("could not get context from bus")

// NOTE: this is required by the linter, otherwise a simple string constant would have been enough
type cliContextKey string

Expand All @@ -19,3 +24,12 @@ func GetValueFromCLIContext[T any](cmd *cobra.Command, key cliContextKey) (T, bo
value, ok := cmd.Context().Value(key).(T)
return value, ok
}

func GetBusFromCmd(cmd *cobra.Command) (modules.Bus, error) {
bus, ok := GetValueFromCLIContext[modules.Bus](cmd, BusCLICtxKey)
if !ok {
return nil, ErrCxtFromBus
}

return bus, nil
}
13 changes: 12 additions & 1 deletion app/client/cli/helpers/setup.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,37 @@ import (
"fmt"

"github.com/spf13/cobra"
"github.com/spf13/viper"

"github.com/pokt-network/pocket/app/client/cli/flags"
"github.com/pokt-network/pocket/logger"
"github.com/pokt-network/pocket/p2p"
rpcCHP "github.com/pokt-network/pocket/p2p/providers/current_height_provider/rpc"
rpcPSP "github.com/pokt-network/pocket/p2p/providers/peerstore_provider/rpc"
"github.com/pokt-network/pocket/runtime"
"github.com/pokt-network/pocket/runtime/configs"
"github.com/pokt-network/pocket/shared/modules"
)

// TODO_THIS_COMMIT: add godoc comment explaining what this **is** and **is not**
// intended to be used for.
const debugPrivKey = "09fc8ee114e678e665d09179acb9a30060f680df44ba06b51434ee47940a8613be19b2b886e743eb1ff7880968d6ce1a46350315e569243e747a227ee8faec3d"

// P2PDependenciesPreRunE initializes peerstore & current height providers, and a
// p2p module which consumes them. Everything is registered to the bus.
func P2PDependenciesPreRunE(cmd *cobra.Command, _ []string) error {

// TECHDEBT: this is to keep backwards compatibility with localnet
flags.ConfigPath = runtime.GetEnv("CONFIG_PATH", "build/config/config.validator1.json")
configs.ParseConfig(flags.ConfigPath)

// set final `remote_cli_url` value; order of precedence: flag > env var > config > default
flags.RemoteCLIURL = viper.GetString("remote_cli_url")

runtimeMgr := runtime.NewManagerFromFiles(
flags.ConfigPath, genesisPath,
runtime.WithClientDebugMode(),
runtime.WithRandomPK(),
runtime.WithPK(debugPrivKey),
)

bus := runtimeMgr.GetBus()
Expand Down
Loading