Disable cache #2568

stwiname · 2024-10-06T20:07:57Z

Description

Provides an option to disable the cache for the store.

Details

Define/update and interfaces to provide abstractions to the Model, Metadata and POI interfaces as well as the StoreCacheService
Implement non-cache versions of abstracted interfaces
Update cache classes to match any interface changes
Create db transactions for when cache is not used

TODOs

Fix remaining build errors and update interfaces to abstract between flushing cache/managing db transaction
Implement flag to change the option and inject the appropriate class (StoreCacheService or PlainStoreModelService)
Implement tests and update existing tests
Improve code reuse by deduplicating any functionality between cache and non-cache models

Fixes #2563

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist

I have tested locally
I have performed a self review of my changes
Updated any relevant documentation
Linked to any relevant issues
I have added tests relevant to my changes
Any dependent changes have been merged and published in downstream modules
My code is up to date with the base branch
I have updated relevant changelogs. We suggest using chan

…flushing cache/managing db transaction

packages/node-core/src/configure/ProjectUpgrade.service.ts

stwiname · 2024-10-21T03:28:35Z

packages/node-core/src/indexer/storeCache/storeModel.service.ts

@@ -19,19 +20,29 @@ export interface IStoreModelService {
  poi: IPoi | null;
  metadata: IMetadata;

+  init(historical: boolean, useCockroachDb: boolean, meta: MetadataRepo, poi?: PoiRepo): void;


This shouldn't be on the interface in the same way the constructor isn't on the interface.

packages/node-core/src/indexer/storeCache/storeModel.service.ts

stwiname · 2024-10-23T00:39:56Z

packages/node-core/src/indexer/store.service.ts

@@ -79,7 +80,7 @@ export class StoreService {
  constructor(
    private sequelize: Sequelize,
    private config: NodeConfig,
-    readonly storeModel: IStoreModelService,
+    readonly storeModel: BaseStoreModelService<ICachedModelControl | IModel<any>>,


Is there a reason this cant be IStoreModelService?

Some methods are only available in cacheModel, and not necessary for plainModel, such as resetData and flushData. However, I don’t think it’s appropriate to define them in IStoreModelService.

github-actions · 2024-10-30T10:20:08Z

Coverage report for `.`

Caution

Test run failed

St.^❔	Category	Percentage	Covered / Total
🟡	Statements	64.92% (-5.19% 🔻)	19692/30333
🟡	Branches	77.45% (-0.85% 🔻)	2541/3281
🟡	Functions	68.13% (+5.63% 🔼)	1114/1635
🟡	Lines	64.92% (-5.19% 🔻)	19692/30333

Show new covered files 🐣

St.^❔	File	Statements	Branches	Functions	Lines
🟢	`...` / migrate.ts	85.34%	16.67%	100%	85.34%
🟢	`...` / migrate-abis.controller.ts	95%	87.5%	100%	95%
🟢	`...` / constants.ts	100%	100%	100%	100%
🟢	`...` / index.ts	100%	100%	100%	100%
🟢	`...` / ethereum.ts	82.28%	83.33%	100%	82.28%
🟢	`...` / migrate-manifest.controller.ts	88%	72.41%	88.89%	88%
🟢	`...` / migrate-mapping.controller.ts	100%	100%	100%	100%
🟢	`...` / migrate-controller.ts	92.41%	85.19%	100%	92.41%
🟢	`...` / migrate.fixtures.ts	100%	100%	100%	100%
🟢	`...` / migrate-schema.controller.ts	97.23%	79.49%	100%	97.23%
🔴	`...` / types.ts	0%	0%	0%	0%
🟢	`...` / createProject.fixtures.ts	84.48%	62.5%	100%	84.48%
🟢	`...` / config.ts	100%	100%	100%	100%
🟢	`...` / index.ts	100%	100%	100%	100%
🔴	`...` / moduleLoader.ts	56%	40%	100%	56%
🟢	`...` / types.ts	100%	100%	100%	100%
🔴	`...` / utils.ts	0%	0%	0%	0%
🟢	`...` / networkFamily.ts	92.59%	88.89%	100%	92.59%
🟡	`...` / IPFSHTTPClientLite.ts	75.74%	79.17%	88.89%	75.74%
🟢	`...` / index.ts	100%	100%	100%	100%
🟢	`...` / utils.ts	92%	66.67%	100%	92%
🟡	`...` / admin.controller.ts	77.24%	96.3%	78.57%	77.24%
🔴	`...` / blockRange.ts	46.67%	100%	50%	46.67%
🟢	`...` / index.ts	100%	100%	100%	100%
🟢	`...` / SubqueryProject.ts	85.52%	83.33%	71.43%	85.52%
🟢	`...` / core.module.ts	100%	50%	100%	100%
🟢	`...` / dictionary.fixtures.ts	100%	100%	100%	100%
🟡	`...` / monitor.service.ts	70.04%	87.21%	94.12%	70.04%
🔴	`...` / sandbox.service.ts	33.33%	50%	25%	33.33%
🟡	`...` / baseCache.service.ts	62.12%	87.5%	66.67%	62.12%
🔴	`...` / baseStoreModel.service.ts	56.36%	66.67%	80%	56.36%
🟢	`...` / cacheable.ts	89.66%	75%	100%	89.66%
🟡	`...` / csvStore.service.ts	77.78%	80%	80%	77.78%
🟢	`...` / index.ts	100%	100%	100%	100%
🟡	`...` / cacheMetadata.ts	66.51%	73.17%	86.67%	66.51%
🟢	`...` / index.ts	100%	100%	100%	100%
🔴	`...` / metadata.ts	53.01%	71.43%	55.56%	53.01%
🟢	`...` / utils.ts	100%	100%	100%	100%
🟢	`...` / cacheModel.ts	86.33%	83.12%	85%	86.33%
🟢	`...` / index.ts	100%	100%	100%	100%
🟡	`...` / model.ts	76.1%	66.67%	81.82%	76.1%
🟢	`...` / utils.ts	100%	100%	100%	100%
🟡	`...` / cachePoi.ts	76.84%	89.47%	85.71%	76.84%
🟢	`...` / index.ts	100%	100%	100%	100%
🔴	`...` / poi.ts	41.07%	71.43%	44.44%	41.07%
🟢	`...` / setValueModel.ts	80.77%	72.73%	85.71%	80.77%
🔴	`...` / storeCache.service.ts	54.98%	75%	68.75%	54.98%
🔴	`...` / storeModel.service.ts	56.79%	50%	57.14%	56.79%
🟢	`...` / types.ts	100%	100%	100%	100%
🟢	`...` / utils.ts	87.5%	75%	100%	87.5%
🟢	`...` / worker.core.module.ts	100%	100%	0%	100%
🔴	`...` / worker.monitor.service.ts	41.46%	100%	0%	41.46%
🔴	`...` / meta.controller.ts	53.33%	100%	0%	53.33%
🟡	`...` / meta.module.ts	60%	100%	0%	60%
🟡	`...` / process.ts	61.29%	66.67%	40%	61.29%
🔴	`...` / foreceClean.init.ts	47.62%	100%	0%	47.62%
🔴	`...` / reindex.init.ts	34.48%	100%	0%	34.48%
🟡	`...` / string.ts	61.54%	50%	100%	61.54%
🔴	`...` / datasourceProcessors.ts	0%	0%	0%	0%
🔴	`...` / index.ts	0%	0%	0%	0%
🔴	`...` / types.ts	0%	0%	0%	0%
🔴	`...` / networking.ts	31.58%	100%	0%	31.58%

Show files with reduced coverage 🔻

St.^❔	File	Statements	Branches	Functions	Lines
🟢	`...` / index.ts	89.09% (+6.14% 🔼)	42.86% (-23.81% 🔻)	100% (+50% 🔼)	89.09% (+6.14% 🔼)
🔴	`...` / delete.ts	48.72% (-5.57% 🔻)	100%	50% (+50% 🔼)	48.72% (-5.57% 🔻)
🔴	`...` / promote.ts	44.19% (-4.53% 🔻)	100%	50% (+50% 🔼)	44.19% (-4.53% 🔻)
🟡	`...` / add.ts	61.29% (-0.78% 🔻)	100%	50% (+50% 🔼)	61.29% (-0.78% 🔻)
🔴	`...` / publish.ts	25.58% (-63.75% 🔻)	100% (+42.86% 🔼)	50% (-50% 🔻)	25.58% (-63.75% 🔻)
🔴	`...` / add-chain-controller.ts	39.62% (-0.45% 🔻)	66.67%	30%	39.62% (-0.45% 🔻)
🟢	`...` / codegen-controller.ts	88.52% (-0.86% 🔻)	84.72% (+9.14% 🔼)	100%	88.52% (-0.86% 🔻)
🟢	`...` / generate-controller.ts	97.75% (+0.1% 🔼)	93.24% (-0.79% 🔻)	100%	97.75% (+0.1% 🔼)
🟡	`...` / init-controller.ts	76.34% (+5.85% 🔼)	63.64% (-4.36% 🔻)	61.11% (+7.78% 🔼)	76.34% (+5.85% 🔼)
🟢	`...` / project-controller.ts	80.72% (-16.54% 🔻)	50% (-10% 🔻)	75% (-25% 🔻)	80.72% (-16.54% 🔻)
🔴	`...` / publish-controller.ts	13.56% (-66.69% 🔻)	100% (+28.89% 🔼)	0% (-100% 🔻)	13.56% (-66.69% 🔻)
🟡	`...` / load.ts	79.41% (-8.82% 🔻)	66.67% (-19.05% 🔻)	100%	79.41% (-8.82% 🔻)
🔴	`...` / models.ts	50% (-49.44% 🔻)	94.12% (+1.81% 🔼)	30.77% (-69.23% 🔻)	50% (-49.44% 🔻)
🟢	`...` / ProjectManifestVersioned.ts	82.19% (-2.74% 🔻)	88.89% (-11.11% 🔻)	61.54% (+3.21% 🔼)	82.19% (-2.74% 🔻)
🟢	`...` / model.ts	93.14% (-4.57% 🔻)	100%	73.33% (+1.9% 🔼)	93.14% (-4.57% 🔻)
🔴	`...` / models.ts	47.06% (-52.94% 🔻)	100%	0% (-100% 🔻)	47.06% (-52.94% 🔻)
🔴	`...` / load.ts	59.18%	37.5% (-18.06% 🔻)	66.67%	59.18%
🟡	`...` / github-reader.ts	65.52% (-23.77% 🔻)	85.71% (+10.71% 🔼)	71.43% (-11.9% 🔻)	65.52% (-23.77% 🔻)
🟡	`...` / ipfs-reader.ts	60.66% (-25.06% 🔻)	70% (+6.36% 🔼)	71.43% (-11.9% 🔻)	60.66% (-25.06% 🔻)
🟡	`...` / local-reader.ts	67.5% (-18.55% 🔻)	100% (+37.5% 🔼)	83.33% (-16.67% 🔻)	67.5% (-18.55% 🔻)
🟢	`...` / reader.ts	81.58% (-7.89% 🔻)	57.14% (-17.86% 🔻)	100%	81.58% (-7.89% 🔻)
🔴	`...` / base.ts	42.99% (-55.01% 🔻)	87.5% (-12.5% 🔻)	40% (-40% 🔻)	42.99% (-55.01% 🔻)
🔴	`...` / models.ts	26.35% (-73.65% 🔻)	100%	0% (-100% 🔻)	26.35% (-73.65% 🔻)
🟢	`...` / NodeConfig.ts	88.82% (+0.22% 🔼)	80.7% (+1.07% 🔼)	85.11% (-0.94% 🔻)	88.82% (+0.22% 🔼)
🟡	`...` / ProjectUpgrade.service.ts	75.74% (-11.33% 🔻)	68% (-23.36% 🔻)	92.31% (+1% 🔼)	75.74% (-11.33% 🔻)
🟢	`...` / configure.module.ts	83.57% (-1% 🔻)	72.73% (+8.73% 🔼)	100%	83.57% (-1% 🔻)
🟢	`...` / db.module.ts	89.06% (+0.78% 🔼)	62.5% (-20.83% 🔻)	80%	89.06% (+0.78% 🔼)
🟢	`...` / SchemaMigration.service.ts	84.69% (-2.87% 🔻)	81.82% (+0.57% 🔼)	83.33% (-16.67% 🔻)	84.69% (-2.87% 🔻)
🟡	`...` / migration-helpers.ts	74.54% (-1.48% 🔻)	86.76% (+6.17% 🔼)	90.91%	74.54% (-1.48% 🔻)
🟡	`...` / migration.ts	69.57% (-12.71% 🔻)	79.73%	86.36% (-4.11% 🔻)	69.57% (-12.71% 🔻)
🟡	`...` / StoreOperations.ts	65.88% (-26.43% 🔻)	60% (-25% 🔻)	87.5% (-12.5% 🔻)	65.88% (-26.43% 🔻)
🔴	`...` / benchmark.service.ts	43.36% (-11.89% 🔻)	66.67%	36.36% (-3.64% 🔻)	43.36% (-11.89% 🔻)
🔴	`...` / base-block-dispatcher.ts	19.4% (-27.95% 🔻)	100%	4.35% (-5.65% 🔻)	19.4% (-27.95% 🔻)
🔴	`...` / block-dispatcher.ts	26.98% (-14.01% 🔻)	66.67%	25% (-3.57% 🔻)	26.98% (-14.01% 🔻)
🔴	`...` / worker-block-dispatcher.ts	36.17% (-14.04% 🔻)	100%	25% (-2.27% 🔻)	36.17% (-14.04% 🔻)
🔴	`...` / coreDictionary.ts	45.33% (-54.67% 🔻)	87.5% (-6.94% 🔻)	77.78% (-22.22% 🔻)	45.33% (-54.67% 🔻)
🟡	`...` / dictionary.service.ts	67% (-18.15% 🔻)	80% (-5.71% 🔻)	83.33% (-7.58% 🔻)	67% (-18.15% 🔻)
🟢	`...` / utils.ts	92.59% (-3.7% 🔻)	69.23% (-24.89% 🔻)	100%	92.59% (-3.7% 🔻)
🟡	`...` / dictionaryV1.ts	75.1% (-18.13% 🔻)	76.09% (-2.17% 🔻)	91.67% (-8.33% 🔻)	75.1% (-18.13% 🔻)
🟡	`...` / dictionaryV2.ts	78.42% (-3.25% 🔻)	71.88% (+6.66% 🔼)	88.89% (-11.11% 🔻)	78.42% (-3.25% 🔻)
🔴	`...` / ds-processor.service.ts	38.12% (-23.97% 🔻)	75%	58.33% (-29.17% 🔻)	38.12% (-23.97% 🔻)
🟡	`...` / dynamic-ds.service.ts	60.16% (-30.04% 🔻)	65% (-1.67% 🔻)	80% (-5.71% 🔻)	60.16% (-30.04% 🔻)
🟢	`...` / fetch.service.ts	92.17% (-0.11% 🔻)	88.89% (+0.16% 🔼)	88% (+1.04% 🔼)	92.17% (-0.11% 🔻)
🟢	`...` / inMemoryCache.service.ts	86.21% (-13.79% 🔻)	100%	75% (-25% 🔻)	86.21% (-13.79% 🔻)
🔴	`...` / indexer.manager.ts	16.73% (-16.48% 🔻)	100%	10% (-2.5% 🔻)	16.73% (-16.48% 🔻)
🟡	`...` / PoiBlock.ts	73.39% (-25% 🔻)	57.89% (-38.66% 🔻)	75% (-25% 🔻)	73.39% (-25% 🔻)
🔴	`...` / poi.service.ts	55.7% (+31.78% 🔼)	57.69% (-42.31% 🔻)	80% (+51.43% 🔼)	55.7% (+31.78% 🔼)
🔴	`...` / poiSync.service.ts	55.88% (-31% 🔻)	77.61% (+1.05% 🔼)	88.89% (-3.42% 🔻)	55.88% (-31% 🔻)
🟡	`...` / project.service.ts	68.71% (-11.29% 🔻)	67.14% (-3.03% 🔻)	85.19% (+1.19% 🔼)	68.71% (-11.29% 🔻)
🔴	`...` / sandbox.ts	42.27% (-25.71% 🔻)	62.5% (+5.36% 🔼)	28.57% (-32.97% 🔻)	42.27% (-25.71% 🔻)
🔴	`...` / smartBatch.service.ts	10.43% (-13.91% 🔻)	100%	12.5% (-1.79% 🔻)	10.43% (-13.91% 🔻)
🔴	`...` / store.service.ts	59.89% (-6.16% 🔻)	74.6% (+6.18% 🔼)	80.65% (+23.5% 🔼)	59.89% (-6.16% 🔻)
🔴	`...` / entity.ts	50% (-11.54% 🔻)	100%	50% (+50% 🔼)	50% (-11.54% 🔻)
🔴	`...` / store.ts	28.73% (-3.47% 🔻)	62.5% (-37.5% 🔻)	33.33% (+33.33% 🔼)	28.73% (-3.47% 🔻)
🟢	`...` / test.runner.ts	95.06% (-4.27% 🔻)	77.78% (+4.44% 🔼)	75% (-25% 🔻)	95.06% (-4.27% 🔻)
🔴	`...` / testing.service.ts	12% (-16.57% 🔻)	100%	0%	12% (-16.57% 🔻)
🟡	`...` / unfinalizedBlocks.service.ts	64.76% (-19.23% 🔻)	96.72%	92% (+0.33% 🔼)	64.76% (-19.23% 🔻)
🔴	`...` / worker.builder.ts	59.83% (-19.36% 🔻)	65.38%	80%	59.83% (-19.36% 🔻)
🔴	`...` / worker.cache.service.ts	48.78% (-9.76% 🔻)	100%	0%	48.78% (-9.76% 🔻)
🔴	`...` / worker.connectionPoolState.manager.ts	38.21% (-22.76% 🔻)	100%	0%	38.21% (-22.76% 🔻)
🔴	`...` / worker.dynamic-ds.service.ts	44.44% (-17.78% 🔻)	100%	0%	44.44% (-17.78% 🔻)
🔴	`...` / worker.service.ts	20.35% (-24.29% 🔻)	100%	0%	20.35% (-24.29% 🔻)
🔴	`...` / worker.ts	41.82% (-0.63% 🔻)	100%	0%	41.82% (-0.63% 🔻)
🔴	`...` / worker.unfinalizedBlocks.service.ts	29.41% (-70.59% 🔻)	100%	0% (-100% 🔻)	29.41% (-70.59% 🔻)
🔴	`...` / logger.ts	47.89% (-16.59% 🔻)	33.33%	40% (+20% 🔼)	47.89% (-16.59% 🔻)
🔴	`...` / event.listener.ts	15.15% (-65.66% 🔻)	100%	0% (-25% 🔻)	15.15% (-65.66% 🔻)
🔴	`...` / health.controller.ts	37.93% (-17.24% 🔻)	100%	0% (-50% 🔻)	37.93% (-17.24% 🔻)
🔴	`...` / health.service.ts	21.05% (-36.48% 🔻)	100%	0% (-20% 🔻)	21.05% (-36.48% 🔻)
🔴	`...` / meta.service.ts	34.26% (-35.26% 🔻)	100% (+20% 🔼)	0% (-40% 🔻)	34.26% (-35.26% 🔻)
🔴	`...` / ready.controller.ts	34.78% (-21.74% 🔻)	100%	0% (-50% 🔻)	34.78% (-21.74% 🔻)
🔴	`...` / ready.service.ts	37.5% (-54.17% 🔻)	100%	0% (-66.67% 🔻)	37.5% (-54.17% 🔻)
🔴	`...` / forceClean.service.ts	22.08% (-4.24% 🔻)	100%	0%	22.08% (-4.24% 🔻)
🔴	`...` / reindex.service.ts	19.55% (-13.78% 🔻)	100%	0%	19.55% (-13.78% 🔻)
🟡	`...` / autoQueue.ts	60.3% (-28.09% 🔻)	85.11%	73.08% (-6.09% 🔻)	60.3% (-28.09% 🔻)
🟡	`...` / blocks.ts	74.55% (-25.45% 🔻)	100% (+11.11% 🔼)	66.67% (-33.33% 🔻)	74.55% (-25.45% 🔻)
🟡	`...` / project.ts	71.88% (-1.49% 🔻)	75% (+0.42% 🔼)	77.78%	71.88% (-1.49% 🔻)
🟢	`...` / promise.ts	86.44% (-5.32% 🔻)	100% (+13.64% 🔼)	71.43% (-14.29% 🔻)	86.44% (-5.32% 🔻)
🟢	`...` / reindex.ts	80.37% (+46.33% 🔼)	16.67% (-83.33% 🔻)	100% (+100% 🔼)	80.37% (+46.33% 🔼)
🔴	`...` / yargs.ts	41.89% (-3.6% 🔻)	100%	14.29%	41.89% (-3.6% 🔻)
🟡	`...` / apiPromise.connection.ts	78.13% (+9.05% 🔼)	54.55% (-25.45% 🔻)	66.67% (+23.81% 🔼)	78.13% (+9.05% 🔼)
🟡	`...` / block-dispatcher.service.ts	66.35% (-2.52% 🔻)	100%	33.33% (+8.33% 🔼)	66.35% (-2.52% 🔻)
🔴	`...` / worker-block-dispatcher.service.ts	23.65% (-13.31% 🔻)	0% (-100% 🔻)	0%	23.65% (-13.31% 🔻)
🟢	`...` / substrateDictionary.service.ts	98.11% (+7.55% 🔼)	66.67% (-6.06% 🔻)	100% (+20% 🔼)	98.11% (+7.55% 🔼)
🟢	`...` / substrateDictionaryV1.ts	85.19% (-0.3% 🔻)	78.72% (+8.96% 🔼)	84.62% (+1.28% 🔼)	85.19% (-0.3% 🔻)
🟡	`...` / substrateDictionaryV2.ts	73.03% (-2.48% 🔻)	75%	33.33%	73.03% (-2.48% 🔻)
🟢	`...` / fetch.module.ts	88.79% (-3.08% 🔻)	50% (-25% 🔻)	100%	88.79% (-3.08% 🔻)
🔴	`...` / indexer.manager.ts	54.97% (-2.57% 🔻)	100%	20% (+10.91% 🔼)	54.97% (-2.57% 🔻)
🟡	`...` / base-runtime.service.ts	72.17% (-0.15% 🔻)	76.92% (+1.92% 🔼)	77.78% (+2.78% 🔼)	72.17% (-0.15% 🔻)
🟢	`...` / runtimeService.ts	94.52% (+2.31% 🔼)	81.82% (-7.07% 🔻)	100% (+20% 🔼)	94.52% (+2.31% 🔼)
🟢	`...` / unfinalizedBlocks.service.ts	86% (-0.27% 🔻)	100%	60% (+10% 🔼)	86% (-0.27% 🔻)
🔴	`...` / http.ts	50.58% (-29.34% 🔻)	50% (+10% 🔼)	42.86% (+12.09% 🔼)	50.58% (-29.34% 🔻)
🟢	`...` / project.ts	88.73% (-0.16% 🔻)	68.18%	87.5%	88.73% (-0.16% 🔻)
🟢	`...` / buffer.ts	90.24% (-9.76% 🔻)	91.3% (-8.7% 🔻)	90.48% (+6.27% 🔼)	90.24% (-9.76% 🔻)
🟢	`...` / builder.ts	92.77%	82.14% (-0.62% 🔻)	100%	92.77%
🟢	`...` / entities.ts	85.4% (-2.92% 🔻)	75% (-11.09% 🔻)	100%	85.4% (-2.92% 🔻)
🟡	`...` / logger.ts	70.35% (+0.75% 🔼)	51.85% (-3.15% 🔻)	78.57% (+3.57% 🔼)	70.35% (+0.75% 🔼)

Test suite run failed

Failed tests: 10/592. Failed suites: 5/107.

  ● Store cache upper threshold › waits for flushing when threshold is met

    expect(received).toBeGreaterThanOrEqual(expected)

    Expected: >= 1000
    Received:    999

      292 |
      293 |     // Should be more than 1s, we set the db tx.commit to take 1s
    > 294 |     expect(end - start).toBeGreaterThanOrEqual(1000);
          |                         ^
      295 |   });
      296 | });
      297 |

      at Object.<anonymous> (packages/node-core/src/indexer/storeModelProvider/storeCache.service.spec.ts:294:25)

  ● CLI deploy, delete, promote › Deploy to Hosted Service and Delete

    Error deploying to hosted service: The selected cluster is not supported

      74 | export function errorHandle(e: any, msg: string): Error {
      75 |   if (axios.isAxiosError(e) && e?.response?.data) {
    > 76 |     return new Error(`${msg} ${e.response.data.message ?? e.response.data}`);
         |            ^
      77 |   }
      78 |
      79 |   return new Error(`${msg} ${e.message}`);

      at errorHandle (packages/cli/src/utils/utils.ts:76:12)
      at createDeployment (packages/cli/src/controller/deploy-controller.ts:62:22)
      at Object.<anonymous> (packages/cli/src/controller/deploy-controller.test.ts:115:27)

  ● CLI deploy, delete, promote › reDeploy to Hosted Service

    Error deploying to hosted service: The selected cluster is not supported

      74 | export function errorHandle(e: any, msg: string): Error {
      75 |   if (axios.isAxiosError(e) && e?.response?.data) {
    > 76 |     return new Error(`${msg} ${e.response.data.message ?? e.response.data}`);
         |            ^
      77 |   }
      78 |
      79 |   return new Error(`${msg} ${e.message}`);

      at errorHandle (packages/cli/src/utils/utils.ts:76:12)
      at createDeployment (packages/cli/src/controller/deploy-controller.ts:62:22)
      at Object.<anonymous> (packages/cli/src/controller/deploy-controller.test.ts:159:26)

  ● Test suite failed to run

    TypeError: Class extends value undefined is not a constructor or null

      84 | }
      85 |
    > 86 | class DsPluginSandbox<P> extends Sandbox {
         |                                  ^
      87 |   constructor(option: Omit<SandboxOption, 'store'>, nodeConfig: NodeConfig) {
      88 |     super(
      89 |       option,

      at Object.<anonymous> (packages/node-core/src/indexer/ds-processor.service.ts:86:34)
      at Object.<anonymous> (packages/node-core/src/indexer/indexer.manager.ts:13:1)
      at Object.<anonymous> (packages/node-core/src/indexer/index.ts:23:1)
      at Object.<anonymous> (packages/node-core/src/db/migration-service/SchemaMigration.service.ts:9:1)
      at Object.<anonymous> (packages/node-core/src/db/migration-service/index.ts:5:1)
      at Object.<anonymous> (packages/node-core/src/db/index.ts:5:1)
      at Object.<anonymous> (packages/node-core/src/utils/graphql.ts:19:1)
      at Object.<anonymous> (packages/node-core/src/utils/index.ts:7:1)
      at Object.<anonymous> (packages/node-core/src/indexer/sandbox.ts:14:1)
      at Object.<anonymous> (packages/node-core/src/indexer/sandbox.spec.ts:6:1)

  ● Intergration test - Publish › overwrites any exisiting CID files

    [tsl] ERROR in /tmp/UHoAXN/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/UHoAXN/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Intergration test - Publish › create ipfsCID file stored in local with dictiory path

    [tsl] ERROR in /tmp/UHoAXN/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/UHoAXN/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Intergration test - Publish › file name consistent with manfiest file name, if -f <manifest path> is used

    [tsl] ERROR in /tmp/UHoAXN/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/UHoAXN/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Cli publish › should upload appropriate project to IPFS

    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › convert to deployment and removed descriptive field

    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › convert js object to JSON object

    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › Get directory CID from multi-chain project

    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/zZG9oZ/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

Report generated by 🧪jest coverage report action from c42541a

… to MetadataEntity

stwiname

Does it make sense for the plain models to be dependencies of the cache equivalent to reduce code?

packages/node-core/src/configure/NodeConfig.ts

packages/node-core/src/indexer/storeModelProvider/storeCache.service.ts

packages/node-core/src/indexer/blockDispatcher/base-block-dispatcher.ts

packages/node-core/src/indexer/store.service.ts

packages/node-core/src/indexer/store/store.ts

packages/node-core/src/indexer/storeModelProvider/types.ts

packages/node-core/src/indexer/storeModelProvider/model/model.ts

packages/node-core/src/db/migration-service/SchemaMigration.service.ts

packages/node-core/src/indexer/store.service.spec.ts

packages/node-core/src/indexer/store.service.ts

packages/node-core/src/indexer/storeModelProvider/baseStoreModel.service.ts

packages/node-core/src/indexer/storeModelProvider/model/model.spec.ts

packages/node-core/src/indexer/storeModelProvider/model/model.ts

packages/node-core/src/indexer/storeModelProvider/storeModel.service.ts

packages/node-core/src/meta/health.service.ts

packages/node-core/src/indexer/store.service.spec.ts

stwiname · 2024-11-19T03:49:04Z

packages/node-core/src/indexer/store.service.ts

    this._blockHeight = blockHeight;
+
+    if (this.modelProvider instanceof PlainStoreModelService && !this.#transaction) {


Is there any issue with setting transaction if the cache is enabled?

Also i think it should throw if transaction is already set. There's probably some undesired behaviour if that is happening and it could lead to a hard to find bug.

The transaction for the cache store is started during the flushData step, and there is no need to start a transaction at this point.

packages/node-core/src/indexer/storeModelProvider/model/cacheModel.ts

packages/node-core/src/indexer/storeModelProvider/model/model.ts

stwiname · 2024-11-19T03:55:39Z

packages/node-core/src/indexer/storeModelProvider/model/model.ts

+    const uniqueDatas = Object.values(uniqueMap);
+
+    // Batch insert every 10000 data
+    const batchSize = 10000;


Why are we doing this?

To prevent the batch processing from causing a performance impact on the database due to the large amount of data.

I think we can remove this. Users can implement it themselves with the current API and it is very unlikely to happen.

We don't have this function for the cache so i don't think we need it for the plain version

stwiname · 2024-11-19T03:56:20Z

packages/node-core/src/indexer/storeModelProvider/model/model.ts

+        continue;
+      }
+
+      await this.model.destroy({


This will delete all historical data

I set up a beforeBulkDestroy hook, which will only delete data at the current height.

I don't think a hook is necessary for that, as there is no where else this could happen.

It also lacks any explanation, is the goal of this to remove all previous entries of the same entity in the same block?

Yes, To ensure that only one record is retained at the current height. Case: Repeat the process of create - delete - create.

packages/node-core/src/indexer/storeModelProvider/model/model.ts

…nd recreated in cache mode.

packages/node-core/src/indexer/store.service.ts

packages/node-core/src/indexer/storeModelProvider/model/model.ts

stwiname and others added 3 commits September 30, 2024 09:00

WIP implment non cache metadata

a58f4e1

WIP extracting out interfaces and implementing non-cache versions

58672be

Fix remaining build errors and update interfaces to abstract between …

838ae83

…flushing cache/managing db transaction

stwiname commented Oct 21, 2024

View reviewed changes

yoozo added 3 commits October 21, 2024 10:37

flag cache-disable and Modifying inheritance relationships

358baff

rename storeStore to modelProvider

10a4a57

rename #storeModel to #modelProvider

64a90cf

stwiname commented Oct 23, 2024

View reviewed changes

yoozo added 2 commits October 25, 2024 08:28

update cacheProvider flushData

33ad32c

Merge branch 'main' into disable-cache

3a39e78

yoozo added 4 commits October 31, 2024 09:38

remane storeModelProvider: IStoreModelProvider

d2af4bb

Rename the directory file to storeModelProvider; rename MetadataModel…

0d51561

… to MetadataEntity

node package update provider StoreCacheService to IStoreModelProvider

b428e6e

fix storeService dependency loop and repeat db transaction creation

200361b

stwiname commented Nov 4, 2024

View reviewed changes

yoozo added 8 commits November 4, 2024 02:42

remove isCachePolicy and update enable-cache flag

c53eaf3

fix poi Inject issues

fe23d05

fix plainModel _block_height

95b56c7

model unit test

9306d35

unit test

c975ee5

Merge branch 'main' into disable-cache

b5774f0

fix build issues

708cb04

fix model block range

162ea87

stwiname commented Nov 13, 2024

View reviewed changes

yoozo added 5 commits November 14, 2024 10:00

Fix the issue of duplicate data writing in the db store.

4058698

Simplify the logic

71affcb

There may be an issue with the store field in the poi table.

f7023ea

set disable cache hook

4d09b2d

db store unit test

4c9a450

yoozo added 2 commits November 15, 2024 09:43

some change

0ce7161

fix unit test

b2b5d19

stwiname commented Nov 17, 2024

View reviewed changes

yoozo added 6 commits November 18, 2024 03:24

rename test file

311e2fb

fix health check api

5ad72ee

remove currentTxCache

8943a29

rename model test

2a7d87e

function cacheProviderFlushData

337a427

db store csv export

6356909

stwiname commented Nov 19, 2024

View reviewed changes

yoozo added 3 commits November 19, 2024 06:19

some improve

a77b6ec

Fix the issue where data is written incorrectly after being deleted a…

b84f8c6

…nd recreated in cache mode.

remove batch size and beforeBulkDestroy hook

afaa536

stwiname commented Nov 20, 2024

View reviewed changes

yoozo added 2 commits November 20, 2024 07:54

cacheModel delete test

719c49f

change log

24340e5

yoozo marked this pull request as ready for review November 21, 2024 02:42

yoozo added 3 commits November 21, 2024 07:02

fix test

cd2375b

fix unit test

023c10a

unit test fix

c42541a

stwiname merged commit 91cbc63 into main Nov 21, 2024
2 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable cache #2568

Disable cache #2568

stwiname commented Oct 6, 2024 •

edited by yoozo

Loading

stwiname Oct 21, 2024

stwiname Oct 23, 2024

yoozo Oct 23, 2024

github-actions bot commented Oct 30, 2024 •

edited

Loading

stwiname left a comment

stwiname Nov 19, 2024

yoozo Nov 19, 2024

stwiname Nov 19, 2024

yoozo Nov 19, 2024

stwiname Nov 19, 2024

stwiname Nov 19, 2024

yoozo Nov 19, 2024

stwiname Nov 20, 2024

yoozo Nov 20, 2024

		this._blockHeight = blockHeight;

		if (this.modelProvider instanceof PlainStoreModelService && !this.#transaction) {

Disable cache #2568

Disable cache #2568

Conversation

stwiname commented Oct 6, 2024 • edited by yoozo Loading

Description

Details

TODOs

Type of change

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 30, 2024 • edited Loading

Coverage report for .

Test suite run failed

stwiname left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stwiname commented Oct 6, 2024 •

edited by yoozo

Loading

github-actions bot commented Oct 30, 2024 •

edited

Loading

Coverage report for `.`