Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trie storage statistics component #5401

Merged
merged 45 commits into from
Nov 6, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
b509c81
storage statistics component in storage package
ssd04 Jul 5, 2023
94db95c
addapt storage statistics component in statusCoreComponents
ssd04 Jul 5, 2023
bdcb008
Revert "addapt storage statistics component in statusCoreComponents"
ssd04 Jul 5, 2023
4e02971
use stats component in trie storage manager
ssd04 Jul 6, 2023
5ba1393
refactor trie statistics component
ssd04 Jul 6, 2023
1be0bd0
update storer with stats interface
ssd04 Jul 7, 2023
1a7db42
fix trie package tests
ssd04 Jul 7, 2023
2471b82
refactor to cast storagemanager in constructor
ssd04 Jul 14, 2023
261c492
fix epoch start unit tests
ssd04 Jul 17, 2023
088a9ac
Merge branch 'rc/v1.6.0' into trie-storage-statistics-component
ssd04 Jul 17, 2023
1b5b5de
fix test in epochstart and integration tests
ssd04 Jul 17, 2023
68cba11
add missing comments
ssd04 Jul 18, 2023
a352567
fix benchmarks integration test
ssd04 Jul 18, 2023
c990e1d
fix linter issue
ssd04 Jul 18, 2023
8aeac06
Merge branch 'rc/v1.6.0' into trie-storage-statistics-component
ssd04 Jul 26, 2023
c0afd2a
fixes after review
ssd04 Aug 7, 2023
8703e00
update trie stub
ssd04 Aug 7, 2023
239929d
Merge branch 'rc/v1.6.0' into trie-storage-statistics-component
ssd04 Sep 4, 2023
7a8fbb4
fix unit tests after merge
ssd04 Sep 4, 2023
c646309
move state stats collector in common
ssd04 Sep 6, 2023
dc3b665
Merge branch 'rc/v1.6.0' into trie-storage-statistics-component
ssd04 Oct 5, 2023
d8e86ed
cleanup trie stats handler
ssd04 Oct 6, 2023
185ea39
separate stats for sync and snapshot
ssd04 Oct 9, 2023
082a009
use atomic map from sync package
ssd04 Oct 10, 2023
f6dc8ed
renamings and comments + fix unit tests
ssd04 Oct 10, 2023
1caef49
fix unit tests - epochstart boostrap and factory
ssd04 Oct 10, 2023
3f566ec
fix integration tests
ssd04 Oct 10, 2023
aa16716
more unit tests + remove old code
ssd04 Oct 10, 2023
77f1f4a
snapshot manager stats
ssd04 Oct 10, 2023
20c0b03
fixes after review
ssd04 Oct 25, 2023
08225bd
fix test
ssd04 Oct 25, 2023
1278c3b
remove trie sync statistics
ssd04 Oct 25, 2023
55b9a70
remove unused code
ssd04 Oct 25, 2023
020f987
fix mutex usage
ssd04 Oct 25, 2023
d45437b
Merge branch 'rc/v1.6.0' into trie-storage-statistics-component
ssd04 Oct 26, 2023
9593d18
Merge branch 'rc/v1.7.0' into trie-storage-statistics-component
ssd04 Oct 26, 2023
ee9c8b6
revert changes from d45437b
ssd04 Oct 26, 2023
a444c20
fix unit test
ssd04 Oct 26, 2023
343e7c8
Merge branch 'rc/v1.7.0' into trie-storage-statistics-component
ssd04 Oct 30, 2023
2b19dd3
update stats return format
ssd04 Oct 30, 2023
97ad360
revert additional changes from 343e7c8
ssd04 Oct 30, 2023
63b2a82
Merge branch 'rc/v1.7.0' into trie-storage-statistics-component
ssd04 Nov 2, 2023
6a4be57
fix integration tests
ssd04 Nov 3, 2023
c192ba4
fix hardfork integration test
ssd04 Nov 3, 2023
d2f3f82
fix create prunning storer maps epoch
ssd04 Nov 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
move state stats collector in common
  • Loading branch information
ssd04 committed Sep 6, 2023
commit c646309b65a2ea4ca05ba3990758a29d327ea7cc
8 changes: 1 addition & 7 deletions common/interface.go
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ type StorageManager interface {
type TrieStorageInteractor interface {
BaseStorer
GetIdentifier() string
GetStatsCollector() StateStatisticsHandler
GetStateStatsHandler() StateStatisticsHandler
}

// BaseStorer define the base methods needed for a storer
Expand All @@ -133,12 +133,6 @@ type BaseStorer interface {
IsInterfaceNil() bool
}

// StorerWithStats defines the behaviour of a base storer component with operations statistics
type StorerWithStats interface {
BaseStorer
GetWithStats(key []byte) ([]byte, bool, error)
}

// SnapshotDbHandler is used to keep track of how many references a snapshot db has
type SnapshotDbHandler interface {
BaseStorer
Expand Down
1 change: 1 addition & 0 deletions common/statistics/disabled/stateStatistics_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
package disabled
BeniaminDrasovean marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 3 additions & 0 deletions common/statistics/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@ var ErrNilNetworkStatisticsProvider = errors.New("nil network statistics provide

// ErrInvalidRefreshIntervalValue signals that an invalid value for the refresh interval was provided
var ErrInvalidRefreshIntervalValue = errors.New("invalid refresh interval value")

// ErrNilStateStatsHandler signals that a nil state statistics handler was provided
var ErrNilStateStatsHandler = errors.New("nil state statistics handler")
1 change: 1 addition & 0 deletions dataRetriever/factory/storageRequestersContainer/args.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,5 @@ type FactoryArgs struct {
ManualEpochStartNotifier dataRetriever.ManualEpochStartNotifier
ChanGracefullyClose chan endProcess.ArgEndProcess
EnableEpochsHandler common.EnableEpochsHandler
StateStatsHandler common.StateStatisticsHandler
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"github.com/multiversx/mx-chain-core-go/marshal"
"github.com/multiversx/mx-chain-go/common"
"github.com/multiversx/mx-chain-go/common/disabled"
"github.com/multiversx/mx-chain-go/common/statistics"
"github.com/multiversx/mx-chain-go/config"
"github.com/multiversx/mx-chain-go/dataRetriever"
disabledRequesters "github.com/multiversx/mx-chain-go/dataRetriever/requestHandlers/requesters/disabled"
Expand All @@ -37,6 +38,7 @@ type baseRequestersContainerFactory struct {
dataPacker dataRetriever.DataPacker
manualEpochStartNotifier dataRetriever.ManualEpochStartNotifier
enableEpochsHandler common.EnableEpochsHandler
stateStatsHandler common.StateStatisticsHandler
chanGracefullyClose chan endProcess.ArgEndProcess
generalConfig config.Config
shardIDForTries uint32
Expand Down Expand Up @@ -76,6 +78,9 @@ func (brcf *baseRequestersContainerFactory) checkParams() error {
if check.IfNil(brcf.enableEpochsHandler) {
return errors.ErrNilEnableEpochsHandler
}
if check.IfNil(brcf.stateStatsHandler) {
return statistics.ErrNilStateStatsHandler
}

return nil
}
Expand Down Expand Up @@ -239,6 +244,7 @@ func (brcf *baseRequestersContainerFactory) newImportDBTrieStorage(
checkpointsStorer storage.Storer,
storageIdentifier dataRetriever.UnitType,
handler common.EnableEpochsHandler,
stateStatsHandler common.StateStatisticsHandler,
) (common.StorageManager, dataRetriever.TrieDataGetter, error) {
pathManager, err := storageFactory.CreatePathManager(
storageFactory.ArgCreatePathManager{
Expand All @@ -262,16 +268,16 @@ func (brcf *baseRequestersContainerFactory) newImportDBTrieStorage(
}

args := trieFactory.TrieCreateArgs{
MainStorer: mainStorer,
CheckpointsStorer: checkpointsStorer,
PruningEnabled: brcf.generalConfig.StateTriesConfig.AccountsStatePruningEnabled,
CheckpointsEnabled: brcf.generalConfig.StateTriesConfig.CheckpointsEnabled,
MaxTrieLevelInMem: brcf.generalConfig.StateTriesConfig.MaxStateTrieLevelInMemory,
SnapshotsEnabled: brcf.snapshotsEnabled,
IdleProvider: disabled.NewProcessStatusHandler(),
Identifier: storageIdentifier.String(),
EnableEpochsHandler: handler,
StateStatisticsEnabled: brcf.generalConfig.StateTriesConfig.StateStatisticsEnabled,
MainStorer: mainStorer,
CheckpointsStorer: checkpointsStorer,
PruningEnabled: brcf.generalConfig.StateTriesConfig.AccountsStatePruningEnabled,
CheckpointsEnabled: brcf.generalConfig.StateTriesConfig.CheckpointsEnabled,
MaxTrieLevelInMem: brcf.generalConfig.StateTriesConfig.MaxStateTrieLevelInMemory,
SnapshotsEnabled: brcf.snapshotsEnabled,
IdleProvider: disabled.NewProcessStatusHandler(),
Identifier: storageIdentifier.String(),
EnableEpochsHandler: handler,
StatsCollector: stateStatsHandler,
}
return trieFactoryInstance.Create(args)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"github.com/multiversx/mx-chain-core-go/core"
"github.com/multiversx/mx-chain-go/dataRetriever"
"github.com/multiversx/mx-chain-go/dataRetriever/factory/containers"
"github.com/multiversx/mx-chain-go/dataRetriever/storageRequesters"
storagerequesters "github.com/multiversx/mx-chain-go/dataRetriever/storageRequesters"
"github.com/multiversx/mx-chain-go/process/factory"
)

Expand Down Expand Up @@ -38,6 +38,7 @@ func NewMetaRequestersContainerFactory(
workingDir: args.WorkingDirectory,
snapshotsEnabled: args.GeneralConfig.StateTriesConfig.SnapshotsEnabled,
enableEpochsHandler: args.EnableEpochsHandler,
stateStatsHandler: args.StateStatsHandler,
}

err := base.checkParams()
Expand Down Expand Up @@ -197,6 +198,7 @@ func (mrcf *metaRequestersContainerFactory) generateTrieNodesRequesters() error
userAccountsCheckpointStorer,
dataRetriever.UserAccountsUnit,
mrcf.enableEpochsHandler,
mrcf.stateStatsHandler,
)
if err != nil {
return fmt.Errorf("%w while creating user accounts data trie storage getter", err)
Expand Down Expand Up @@ -235,6 +237,7 @@ func (mrcf *metaRequestersContainerFactory) generateTrieNodesRequesters() error
peerAccountsCheckpointStorer,
dataRetriever.PeerAccountsUnit,
mrcf.enableEpochsHandler,
mrcf.stateStatsHandler,
)
if err != nil {
return fmt.Errorf("%w while creating peer accounts data trie storage getter", err)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ func (srcf *shardRequestersContainerFactory) generateTrieNodesRequesters() error
userAccountsCheckpointStorer,
dataRetriever.UserAccountsUnit,
srcf.enableEpochsHandler,
srcf.stateStatsHandler,
BeniaminDrasovean marked this conversation as resolved.
Show resolved Hide resolved
)
if err != nil {
return fmt.Errorf("%w while creating user accounts data trie storage getter", err)
Expand Down
4 changes: 4 additions & 0 deletions epochStart/bootstrap/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"fmt"

"github.com/multiversx/mx-chain-core-go/core/check"
"github.com/multiversx/mx-chain-go/common/statistics"
"github.com/multiversx/mx-chain-go/epochStart"
)

Expand Down Expand Up @@ -115,6 +116,9 @@ func checkArguments(args ArgsEpochStartBootstrap) error {
if check.IfNil(args.CryptoComponentsHolder.ManagedPeersHolder()) {
return fmt.Errorf("%s: %w", baseErrorMessage, epochStart.ErrNilManagedPeersHolder)
}
if check.IfNil(args.StateStatsHandler) {
return fmt.Errorf("%s: %w", baseErrorMessage, statistics.ErrNilStateStatsHandler)
}

return nil
}
3 changes: 1 addition & 2 deletions epochStart/bootstrap/disabled/disabledChainStorer.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import (

"github.com/multiversx/mx-chain-go/dataRetriever"
"github.com/multiversx/mx-chain-go/storage"
"github.com/multiversx/mx-chain-go/testscommon"
logger "github.com/multiversx/mx-chain-logger-go"
)

Expand Down Expand Up @@ -60,7 +59,7 @@ func (c *chainStorer) GetStorer(unitType dataRetriever.UnitType) (storage.Storer
_, ok := c.mapStorages[unitType]
if !ok {
log.Debug("created new mem storer", "key", unitType)
c.mapStorages[unitType] = testscommon.CreateMemStorerWithStats(CreateMemUnit())
c.mapStorages[unitType] = CreateMemStorerWithStats()
}

store := c.mapStorages[unitType]
Expand Down
17 changes: 17 additions & 0 deletions epochStart/bootstrap/disabled/disabledStorer.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,20 @@ func CreateMemUnit() storage.Storer {

return unit
}

// storerWithStats -
type storerWithStats struct {
storage.Storer
}

// GetWithStats will trigger get operation with statistics
func (ss *storerWithStats) GetWithStats(key []byte) ([]byte, bool, error) {
v, err := ss.Get(key)
return v, false, err
}

// CreateMemStorerWithStats -
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add proper comments, this is productive code

func CreateMemStorerWithStats() storage.StorerWithStats {
storerUnit := CreateMemUnit()
return &storerWithStats{storerUnit}
}
1 change: 1 addition & 0 deletions epochStart/bootstrap/fromLocalStorage.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ func (e *epochStartBootstrap) prepareEpochFromStorage() (Parameters, error) {
e.generalConfig,
e.coreComponentsHolder,
e.storageService,
e.stateStatsHandler,
)
if err != nil {
return Parameters{}, err
Expand Down
5 changes: 5 additions & 0 deletions epochStart/bootstrap/process.go
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ type epochStartBootstrap struct {
trieSyncStatisticsProvider common.SizeSyncStatisticsHandler
nodeProcessingMode common.NodeProcessingMode
nodeOperationMode common.NodeOperation
stateStatsHandler common.StateStatisticsHandler
// created components
requestHandler process.RequestHandler
mainInterceptorContainer process.InterceptorsContainer
Expand Down Expand Up @@ -185,6 +186,7 @@ type ArgsEpochStartBootstrap struct {
ScheduledSCRsStorer storage.Storer
TrieSyncStatisticsProvider common.SizeSyncStatisticsHandler
NodeProcessingMode common.NodeProcessingMode
StateStatsHandler common.StateStatisticsHandler
}

type dataToSync struct {
Expand Down Expand Up @@ -511,6 +513,7 @@ func (e *epochStartBootstrap) prepareComponentsToSyncFromNetwork() error {
e.generalConfig,
e.coreComponentsHolder,
e.storageService,
e.stateStatsHandler,
)
if err != nil {
return err
Expand Down Expand Up @@ -804,6 +807,7 @@ func (e *epochStartBootstrap) requestAndProcessForMeta(peerMiniBlocks []*block.M
e.generalConfig,
e.coreComponentsHolder,
storageHandlerComponent.storageService,
e.stateStatsHandler,
)
if err != nil {
return err
Expand Down Expand Up @@ -972,6 +976,7 @@ func (e *epochStartBootstrap) requestAndProcessForShard(peerMiniBlocks []*block.
e.generalConfig,
e.coreComponentsHolder,
storageHandlerComponent.storageService,
e.stateStatsHandler,
)
if err != nil {
return err
Expand Down
4 changes: 3 additions & 1 deletion epochStart/bootstrap/storageProcess.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
"github.com/multiversx/mx-chain-go/dataRetriever"
factoryDataPool "github.com/multiversx/mx-chain-go/dataRetriever/factory"
"github.com/multiversx/mx-chain-go/dataRetriever/factory/containers"
"github.com/multiversx/mx-chain-go/dataRetriever/factory/storageRequestersContainer"
storagerequesterscontainer "github.com/multiversx/mx-chain-go/dataRetriever/factory/storageRequestersContainer"
"github.com/multiversx/mx-chain-go/dataRetriever/requestHandlers"
"github.com/multiversx/mx-chain-go/epochStart"
"github.com/multiversx/mx-chain-go/epochStart/bootstrap/disabled"
Expand Down Expand Up @@ -152,6 +152,7 @@ func (sesb *storageEpochStartBootstrap) prepareComponentsToSync() error {
sesb.generalConfig,
sesb.coreComponentsHolder,
sesb.storageService,
sesb.stateStatsHandler,
)
if err != nil {
return err
Expand Down Expand Up @@ -252,6 +253,7 @@ func (sesb *storageEpochStartBootstrap) createStorageRequesters() error {
ManualEpochStartNotifier: mesn,
ChanGracefullyClose: sesb.chanGracefullyClose,
EnableEpochsHandler: sesb.coreComponentsHolder.EnableEpochsHandler(),
StateStatsHandler: sesb.stateStatsHandler,
}

var requestersContainerFactory dataRetriever.RequestersContainerFactory
Expand Down
4 changes: 2 additions & 2 deletions epochStart/metachain/stakingDataProvider_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ func createStakingDataProviderWithMockArgs(
func createStakingDataProviderWithRealArgs(t *testing.T, owner []byte, blsKey []byte, topUpVal *big.Int) *stakingDataProvider {
args, _ := createFullArgumentsForSystemSCProcessing(config.EnableEpochs{
StakingV2EnableEpoch: 1000,
}, testscommon.CreateStorerWithStats())
}, testscommon.CreateDefaultMemStorerWithStats())
args.EpochNotifier.CheckEpoch(&testscommon.HeaderHandlerStub{
EpochField: 1000000,
})
Expand Down Expand Up @@ -468,7 +468,7 @@ func createStakingDataProviderAndUpdateCache(t *testing.T, validatorsInfo map[ui

args, _ := createFullArgumentsForSystemSCProcessing(config.EnableEpochs{
StakingV2EnableEpoch: 1,
}, testscommon.CreateStorerWithStats())
}, testscommon.CreateDefaultMemStorerWithStats())
args.EpochNotifier.CheckEpoch(&testscommon.HeaderHandlerStub{
EpochField: 1,
})
Expand Down
Loading