Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix/multi endpint delay #2582

Merged
merged 9 commits into from
Nov 12, 2024
Merged

Fix/multi endpint delay #2582

merged 9 commits into from
Nov 12, 2024

Conversation

yoozo
Copy link
Collaborator

@yoozo yoozo commented Oct 30, 2024 •
edited by stwiname
Loading

Description

Multi-endpoint indexing logic delay and other related issues

Fixes #2572

  • Fix the issue where the backoff delay increases excessively after multiple failures.
  • Correctly handle timeout and rate limit errors.
  • Resolve the issue where default errors are treated as rate limit errors on the endpoint.
  • Add control logic for rateLimitDelay, default to 20 seconds.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist

  • I have tested locally
  • I have performed a self review of my changes
  • Updated any relevant documentation
  • Linked to any relevant issues
  • I have added tests relevant to my changes
  • Any dependent changes have been merged and published in downstream modules
  • My code is up to date with the base branch
  • I have updated relevant changelogs. We suggest using chan

@yoozo yoozo requested a review from stwiname October 30, 2024 09:50
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 30, 2024 •
edited
Loading

Coverage report for .

Caution

Test run failed

St.❔
 Category Percentage Covered / Total
🟡 Statements
65.53% (-4.57% 🔻)
 19551/29834
🟡 Branches
79.81% (+1.52% 🔼)
 2621/3284
🟡 Functions
68.47% (+5.97% 🔼)
 1088/1589
🟡 Lines
65.53% (-4.57% 🔻)
 19551/29834
Show new covered files 🐣
St.❔
 File Statements Branches Functions Lines
🟢
... / migrate.ts
 85.34% 16.67% 100% 85.34%
🟢
... / migrate-abis.controller.ts
 95% 87.5% 100% 95%
🟢
... / constants.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / ethereum.ts
 82.28% 83.33% 100% 82.28%
🟢
... / migrate-manifest.controller.ts
 88% 73.33% 88.89% 88%
🟢
... / migrate-mapping.controller.ts
 100% 100% 100% 100%
🟢
... / migrate-controller.ts
 92.41% 85.19% 100% 92.41%
🟢
... / migrate.fixtures.ts
 100% 100% 100% 100%
🟢
... / migrate-schema.controller.ts
 97.23% 79.49% 100% 97.23%
🔴
... / types.ts
 0% 0% 0% 0%
🟢
... / createProject.fixtures.ts
 84.48% 62.5% 100% 84.48%
🟢
... / config.ts
 100% 100% 100% 100%
🟢
... / index.ts
 100% 100% 100% 100%
🔴
... / moduleLoader.ts
 56% 40% 100% 56%
🟢
... / types.ts
 100% 100% 100% 100%
🔴
... / utils.ts
 0% 0% 0% 0%
🟢
... / networkFamily.ts
 92.59% 88.89% 100% 92.59%
🟡
... / IPFSHTTPClientLite.ts
 75.74% 79.17% 88.89% 75.74%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / utils.ts
 92% 66.67% 100% 92%
🟡
... / admin.controller.ts
 77.24% 96.3% 78.57% 77.24%
🔴
... / blockRange.ts
 46.67% 100% 50% 46.67%
🟢
... / index.ts
 100% 100% 100% 100%
🟢
... / SubqueryProject.ts
 85.52% 83.33% 71.43% 85.52%
🟢
... / core.module.ts
 100% 100% 100% 100%
🟢
... / dictionary.fixtures.ts
 100% 100% 100% 100%
🟡
... / monitor.service.ts
 69.61% 86.9% 94.12% 69.61%
🔴
... / sandbox.service.ts
 33.33% 50% 25% 33.33%
🟢
... / worker.core.module.ts
 100% 100% 0% 100%
🔴
... / worker.monitor.service.ts
 41.46% 100% 0% 41.46%
🔴
... / meta.controller.ts
 53.33% 100% 0% 53.33%
🟡
... / meta.module.ts
 60% 100% 0% 60%
🔴
... / process.ts
 50% 66.67% 40% 50%
🔴
... / foreceClean.init.ts
 47.62% 100% 0% 47.62%
🔴
... / reindex.init.ts
 34.48% 100% 0% 34.48%
🟢
... / string.ts
 100% 100% 100% 100%
🔴
... / datasourceProcessors.ts
 0% 0% 0% 0%
🔴
... / index.ts
 0% 0% 0% 0%
🔴
... / types.ts
 0% 0% 0% 0%
🔴
... / networking.ts
 31.58% 100% 0% 31.58%
Show files with reduced coverage 🔻
St.❔
 File Statements Branches Functions Lines
🟢
... / index.ts
89.09% (+6.14% 🔼)
42.86% (-23.81% 🔻)
100% (+50% 🔼)
89.09% (+6.14% 🔼)
🔴
... / delete.ts
48.72% (-5.57% 🔻)
 100%
50% (+50% 🔼)
48.72% (-5.57% 🔻)
🔴
... / promote.ts
44.19% (-4.53% 🔻)
 100%
50% (+50% 🔼)
44.19% (-4.53% 🔻)
🟡
... / add.ts
61.29% (-0.78% 🔻)
 100%
50% (+50% 🔼)
61.29% (-0.78% 🔻)
🔴
... / publish.ts
25.58% (-63.75% 🔻)
100% (+42.86% 🔼)
50% (-50% 🔻)
25.58% (-63.75% 🔻)
🔴
... / add-chain-controller.ts
39.62% (-0.45% 🔻)
 66.67% 30%
39.62% (-0.45% 🔻)
🟢
... / codegen-controller.ts
88.52% (-0.86% 🔻)
84.72% (+9.14% 🔼)
 100%
88.52% (-0.86% 🔻)
🟢
... / generate-controller.ts
97.75% (+0.1% 🔼)
93.24% (-0.79% 🔻)
 100%
97.75% (+0.1% 🔼)
🟡
... / init-controller.ts
76.84% (+6.36% 🔼)
67.65% (-0.35% 🔻)
61.11% (+7.78% 🔼)
76.84% (+6.36% 🔼)
🟢
... / project-controller.ts
80.72% (-16.54% 🔻)
50% (-10% 🔻)
75% (-25% 🔻)
80.72% (-16.54% 🔻)
🔴
... / publish-controller.ts
13.56% (-66.69% 🔻)
100% (+28.89% 🔼)
0% (-100% 🔻)
13.56% (-66.69% 🔻)
🟡
... / load.ts
79.41% (-8.82% 🔻)
66.67% (-19.05% 🔻)
 100%
79.41% (-8.82% 🔻)
🔴
... / models.ts
50% (-49.44% 🔻)
94.12% (+1.81% 🔼)
30.77% (-69.23% 🔻)
50% (-49.44% 🔻)
🟢
... / ProjectManifestVersioned.ts
82.19% (-2.74% 🔻)
88.89% (-11.11% 🔻)
61.54% (+3.21% 🔼)
82.19% (-2.74% 🔻)
🟢
... / model.ts
93.14% (-4.57% 🔻)
 100%
73.33% (+1.9% 🔼)
93.14% (-4.57% 🔻)
🔴
... / models.ts
47.06% (-52.94% 🔻)
 100%
0% (-100% 🔻)
47.06% (-52.94% 🔻)
🔴
... / load.ts
 59.18%
37.5% (-18.06% 🔻)
 66.67% 59.18%
🟡
... / github-reader.ts
65.52% (-23.77% 🔻)
85.71% (+10.71% 🔼)
71.43% (-11.9% 🔻)
65.52% (-23.77% 🔻)
🟡
... / ipfs-reader.ts
63.93% (-21.78% 🔻)
90.91% (+27.27% 🔼)
71.43% (-11.9% 🔻)
63.93% (-21.78% 🔻)
🟡
... / local-reader.ts
67.5% (-18.55% 🔻)
100% (+37.5% 🔼)
83.33% (-16.67% 🔻)
67.5% (-18.55% 🔻)
🔴
... / base.ts
42.99% (-55.01% 🔻)
87.5% (-12.5% 🔻)
40% (-40% 🔻)
42.99% (-55.01% 🔻)
🔴
... / models.ts
26.35% (-73.65% 🔻)
 100%
0% (-100% 🔻)
26.35% (-73.65% 🔻)
🟢
... / NodeConfig.ts
88.66% (+0.05% 🔼)
80.36% (+0.73% 🔼)
84.78% (-1.26% 🔻)
88.66% (+0.05% 🔼)
🟢
... / ProjectUpgrade.service.ts
82.17% (-4.91% 🔻)
92.05% (+0.69% 🔼)
92.31% (+1% 🔼)
82.17% (-4.91% 🔻)
🟢
... / configure.module.ts
83.57% (-1% 🔻)
75% (+11% 🔼)
 100%
83.57% (-1% 🔻)
🟢
... / db.module.ts
89.06% (+0.78% 🔼)
62.5% (-20.83% 🔻)
 80%
89.06% (+0.78% 🔼)
🟢
... / SchemaMigration.service.ts
84.85% (-2.72% 🔻)
81.82% (+0.57% 🔼)
83.33% (-16.67% 🔻)
84.85% (-2.72% 🔻)
🟡
... / migration-helpers.ts
75.28% (-0.74% 🔻)
88.24% (+7.64% 🔼)
 90.91%
75.28% (-0.74% 🔻)
🟡
... / migration.ts
69.57% (-12.71% 🔻)
 79.73%
86.36% (-4.11% 🔻)
69.57% (-12.71% 🔻)
🟡
... / StoreOperations.ts
65.88% (-26.43% 🔻)
64.71% (-20.29% 🔻)
87.5% (-12.5% 🔻)
65.88% (-26.43% 🔻)
🔴
... / benchmark.service.ts
43.36% (-11.89% 🔻)
 66.67%
36.36% (-3.64% 🔻)
43.36% (-11.89% 🔻)
🔴
... / base-block-dispatcher.ts
19% (-28.35% 🔻)
 100%
4.35% (-5.65% 🔻)
19% (-28.35% 🔻)
🔴
... / block-dispatcher.ts
26.98% (-14.01% 🔻)
 66.67%
25% (-3.57% 🔻)
26.98% (-14.01% 🔻)
🔴
... / worker-block-dispatcher.ts
36.17% (-14.04% 🔻)
 100%
25% (-2.27% 🔻)
36.17% (-14.04% 🔻)
🔴
... / coreDictionary.ts
45.33% (-54.67% 🔻)
87.5% (-6.94% 🔻)
77.78% (-22.22% 🔻)
45.33% (-54.67% 🔻)
🟡
... / dictionary.service.ts
68.47% (-16.67% 🔻)
83.78% (-1.93% 🔻)
83.33% (-7.58% 🔻)
68.47% (-16.67% 🔻)
🟡
... / dictionaryV1.ts
73.06% (-20.17% 🔻)
75% (-3.26% 🔻)
91.67% (-8.33% 🔻)
73.06% (-20.17% 🔻)
🟡
... / dictionaryV2.ts
67.37% (-14.31% 🔻)
68% (+2.78% 🔼)
88.89% (-11.11% 🔻)
67.37% (-14.31% 🔻)
🔴
... / ds-processor.service.ts
38.12% (-23.97% 🔻)
 75%
58.33% (-29.17% 🔻)
38.12% (-23.97% 🔻)
🔴
... / dynamic-ds.service.ts
59.84% (-30.35% 🔻)
65% (-1.67% 🔻)
80% (-5.71% 🔻)
59.84% (-30.35% 🔻)
🟢
... / fetch.service.ts
92.17% (-0.11% 🔻)
88.89% (+0.16% 🔼)
88% (+1.04% 🔼)
92.17% (-0.11% 🔻)
🟢
... / inMemoryCache.service.ts
86.21% (-13.79% 🔻)
 100%
75% (-25% 🔻)
86.21% (-13.79% 🔻)
🔴
... / indexer.manager.ts
16.73% (-16.48% 🔻)
 100%
10% (-2.5% 🔻)
16.73% (-16.48% 🔻)
🟢
... / PoiBlock.ts
83.87% (-14.52% 🔻)
 96.55%
75% (-25% 🔻)
83.87% (-14.52% 🔻)
🔴
... / poi.service.ts
57.21% (+33.28% 🔼)
55.56% (-44.44% 🔻)
80% (+51.43% 🔼)
57.21% (+33.28% 🔼)
🔴
... / poiModel.ts
53.64% (-10% 🔻)
75% (+3.57% 🔼)
55.56% (-1.59% 🔻)
53.64% (-10% 🔻)
🔴
... / poiSync.service.ts
55.88% (-31% 🔻)
77.61% (+1.05% 🔼)
88.89% (-3.42% 🔻)
55.88% (-31% 🔻)
🟡
... / project.service.ts
72.67% (-7.33% 🔻)
77.46% (+7.29% 🔼)
85.19% (+1.19% 🔼)
72.67% (-7.33% 🔻)
🔴
... / sandbox.ts
59.09% (-8.89% 🔻)
53.33% (-3.81% 🔻)
57.14% (-4.4% 🔻)
59.09% (-8.89% 🔻)
🔴
... / smartBatch.service.ts
10.43% (-13.91% 🔻)
 100%
12.5% (-1.79% 🔻)
10.43% (-13.91% 🔻)
🔴
... / store.service.ts
55.19% (-10.85% 🔻)
72.92% (+4.5% 🔼)
60% (+2.86% 🔼)
55.19% (-10.85% 🔻)
🔴
... / entity.ts
50% (-11.54% 🔻)
66.67% (-33.33% 🔻)
50% (+50% 🔼)
50% (-11.54% 🔻)
🔴
... / store.ts
12.21% (-19.99% 🔻)
 100% 0%
12.21% (-19.99% 🔻)
🟡
... / baseCache.service.ts
62.9% (-30.43% 🔻)
 87.5%
66.67% (-13.33% 🔻)
62.9% (-30.43% 🔻)
🟢
... / cacheModel.ts
82.29% (-2.78% 🔻)
76.12% (-2.19% 🔻)
77.27% (+1.08% 🔼)
82.29% (-2.78% 🔻)
🟢
... / cacheable.ts
89.66% (-0.97% 🔻)
 75% 100%
89.66% (-0.97% 🔻)
🟡
... / csvStore.service.ts
76.92% (-14.1% 🔻)
 80%
80% (-20% 🔻)
76.92% (-14.1% 🔻)
🔴
... / storeCache.service.ts
58.15% (-17.09% 🔻)
71.43% (+2.68% 🔼)
77.78% (-3.47% 🔻)
58.15% (-17.09% 🔻)
🟢
... / test.runner.ts
95.03% (-4.3% 🔻)
77.78% (+4.44% 🔼)
75% (-25% 🔻)
95.03% (-4.3% 🔻)
🔴
... / testing.service.ts
12% (-16.57% 🔻)
 100% 0%
12% (-16.57% 🔻)
🟡
... / unfinalizedBlocks.service.ts
64.31% (-19.68% 🔻)
 96.72%
92% (+0.33% 🔼)
64.31% (-19.68% 🔻)
🔴
... / worker.builder.ts
59.83% (-19.36% 🔻)
 65.38% 80%
59.83% (-19.36% 🔻)
🔴
... / worker.cache.service.ts
48.78% (-9.76% 🔻)
 100% 0%
48.78% (-9.76% 🔻)
🔴
... / worker.connectionPoolState.manager.ts
38.21% (-22.76% 🔻)
 100% 0%
38.21% (-22.76% 🔻)
🔴
... / worker.dynamic-ds.service.ts
44.44% (-17.78% 🔻)
 100% 0%
44.44% (-17.78% 🔻)
🔴
... / worker.service.ts
20.35% (-24.29% 🔻)
 100% 0%
20.35% (-24.29% 🔻)
🔴
... / worker.ts
41.82% (-0.63% 🔻)
 100% 0%
41.82% (-0.63% 🔻)
🔴
... / worker.unfinalizedBlocks.service.ts
30.61% (-69.39% 🔻)
 100%
0% (-100% 🔻)
30.61% (-69.39% 🔻)
🔴
... / logger.ts
47.89% (-16.59% 🔻)
 33.33%
40% (+20% 🔼)
47.89% (-16.59% 🔻)
🔴
... / event.listener.ts
15.15% (-65.66% 🔻)
 100%
0% (-25% 🔻)
15.15% (-65.66% 🔻)
🔴
... / health.controller.ts
37.93% (-17.24% 🔻)
 100%
0% (-50% 🔻)
37.93% (-17.24% 🔻)
🔴
... / health.service.ts
21.92% (-35.62% 🔻)
 100%
0% (-20% 🔻)
21.92% (-35.62% 🔻)
🔴
... / meta.service.ts
34.26% (-35.26% 🔻)
100% (+20% 🔼)
0% (-40% 🔻)
34.26% (-35.26% 🔻)
🔴
... / ready.controller.ts
34.78% (-21.74% 🔻)
 100%
0% (-50% 🔻)
34.78% (-21.74% 🔻)
🔴
... / ready.service.ts
37.5% (-54.17% 🔻)
 100%
0% (-66.67% 🔻)
37.5% (-54.17% 🔻)
🔴
... / forceClean.service.ts
22.08% (-4.24% 🔻)
 100% 0%
22.08% (-4.24% 🔻)
🔴
... / reindex.service.ts
15.2% (-18.13% 🔻)
 100% 0%
15.2% (-18.13% 🔻)
🟡
... / autoQueue.ts
60.3% (-28.09% 🔻)
 85.11%
73.08% (-6.09% 🔻)
60.3% (-28.09% 🔻)
🟡
... / blocks.ts
74.55% (-25.45% 🔻)
100% (+11.11% 🔼)
66.67% (-33.33% 🔻)
74.55% (-25.45% 🔻)
🟡
... / project.ts
71.88% (-1.49% 🔻)
75% (+0.42% 🔼)
 77.78%
71.88% (-1.49% 🔻)
🟢
... / promise.ts
86.44% (-5.32% 🔻)
100% (+13.64% 🔼)
71.43% (-14.29% 🔻)
86.44% (-5.32% 🔻)
🟢
... / reindex.ts
81.72% (+47.68% 🔼)
20% (-80% 🔻)
100% (+100% 🔼)
81.72% (+47.68% 🔼)
🔴
... / yargs.ts
42.51% (-2.97% 🔻)
 100% 14.29%
42.51% (-2.97% 🔻)
🟡
... / apiPromise.connection.ts
78.13% (+9.05% 🔼)
54.55% (-25.45% 🔻)
66.67% (+23.81% 🔼)
78.13% (+9.05% 🔼)
🟡
... / block-dispatcher.service.ts
66.35% (-2.52% 🔻)
 100%
33.33% (+8.33% 🔼)
66.35% (-2.52% 🔻)
🔴
... / worker-block-dispatcher.service.ts
23.65% (-13.31% 🔻)
 100% 0%
23.65% (-13.31% 🔻)
🟢
... / substrateDictionary.service.ts
98.11% (+7.55% 🔼)
66.67% (-6.06% 🔻)
100% (+20% 🔼)
98.11% (+7.55% 🔼)
🟢
... / substrateDictionaryV1.ts
85.19% (-0.3% 🔻)
78.72% (+8.96% 🔼)
84.62% (+1.28% 🔼)
85.19% (-0.3% 🔻)
🟡
... / substrateDictionaryV2.ts
73.03% (-2.48% 🔻)
 75% 33.33%
73.03% (-2.48% 🔻)
🟢
... / fetch.module.ts
88.7% (-3.18% 🔻)
50% (-25% 🔻)
 100%
88.7% (-3.18% 🔻)
🔴
... / indexer.manager.ts
54.97% (-2.57% 🔻)
 100%
20% (+10.91% 🔼)
54.97% (-2.57% 🔻)
🟡
... / base-runtime.service.ts
72.17% (-0.15% 🔻)
76.92% (+1.92% 🔼)
77.78% (+2.78% 🔼)
72.17% (-0.15% 🔻)
🟢
... / runtimeService.ts
94.52% (+2.31% 🔼)
81.82% (-7.07% 🔻)
100% (+20% 🔼)
94.52% (+2.31% 🔼)
🟢
... / unfinalizedBlocks.service.ts
86% (-0.27% 🔻)
 100%
60% (+10% 🔼)
86% (-0.27% 🔻)
🔴
... / http.ts
50.58% (-29.34% 🔻)
50% (+10% 🔼)
42.86% (+12.09% 🔼)
50.58% (-29.34% 🔻)
🟢
... / project.ts
88.73% (-0.16% 🔻)
 68.18% 87.5%
88.73% (-0.16% 🔻)
🟢
... / buffer.ts
90.24% (-9.76% 🔻)
91.3% (-8.7% 🔻)
90.48% (+6.27% 🔼)
90.24% (-9.76% 🔻)
🟢
... / entities.ts
85.89% (-2.42% 🔻)
82.93% (-3.16% 🔻)
 100%
85.89% (-2.42% 🔻)
🟡
... / logger.ts
69.19% (-0.41% 🔻)
48.15% (-6.85% 🔻)
78.57% (+3.57% 🔼)
69.19% (-0.41% 🔻)

Test suite run failed

Failed tests: 10/591. Failed suites: 4/105. 
  ● CLI deploy, delete, promote › Deploy to Hosted Service and Delete

    Error deploying to hosted service: The selected cluster is not supported

      74 | export function errorHandle(e: any, msg: string): Error {
      75 |   if (axios.isAxiosError(e) && e?.response?.data) {
    > 76 |     return new Error(`${msg} ${e.response.data.message ?? e.response.data}`);
         |            ^
      77 |   }
      78 |
      79 |   return new Error(`${msg} ${e.message}`);

      at errorHandle (packages/cli/src/utils/utils.ts:76:12)
      at createDeployment (packages/cli/src/controller/deploy-controller.ts:62:22)
      at Object.<anonymous> (packages/cli/src/controller/deploy-controller.test.ts:115:27)

  ● CLI deploy, delete, promote › reDeploy to Hosted Service

    Error deploying to hosted service: The selected cluster is not supported

      74 | export function errorHandle(e: any, msg: string): Error {
      75 |   if (axios.isAxiosError(e) && e?.response?.data) {
    > 76 |     return new Error(`${msg} ${e.response.data.message ?? e.response.data}`);
         |            ^
      77 |   }
      78 |
      79 |   return new Error(`${msg} ${e.message}`);

      at errorHandle (packages/cli/src/utils/utils.ts:76:12)
      at createDeployment (packages/cli/src/controller/deploy-controller.ts:62:22)
      at Object.<anonymous> (packages/cli/src/controller/deploy-controller.test.ts:159:26)


  ● Intergration test - Publish › overwrites any exisiting CID files

    [tsl] ERROR in /tmp/qehBtG/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/qehBtG/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Intergration test - Publish › create ipfsCID file stored in local with dictiory path

    [tsl] ERROR in /tmp/qehBtG/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/qehBtG/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)

  ● Intergration test - Publish › file name consistent with manfiest file name, if -f <manifest path> is used

    [tsl] ERROR in /tmp/qehBtG/mocked_starter/src/types/models/Transfer.ts(61,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/qehBtG/mocked_starter/src/types/models/Transfer.ts(67,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createTestProject (packages/cli/src/createProject.fixtures.ts:76:5)
      at async Object.<anonymous> (packages/cli/src/commands/publish.test.ts:17:22)


  ● Cli publish › should upload appropriate project to IPFS

    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › convert to deployment and removed descriptive field

    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › convert js object to JSON object

    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)

  ● Cli publish › Get directory CID from multi-chain project

    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(75,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ClaimRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(81,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/ForgeRecord.ts(87,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(63,48)
          TS2558: Expected 0 type arguments, but got 1.
    [tsl] ERROR in /tmp/Rx8Wd1/multi_mocked_starter/src/types/models/NFT.ts(69,48)
          TS2558: Expected 0 type arguments, but got 1.

      44 |       const outputDir = path.resolve(directory, flags.output ?? 'dist');
      45 |
    > 46 |       await runWebpack(buildEntries, directory, outputDir, isDev, true);
         |                  ^
      47 |       if (!flags.silent) {
      48 |         this.log('Building and packing code ...');
      49 |         this.log('Done!');

      at Object.error (node_modules/@oclif/core/lib/errors/index.js:27:15)
      at Build.error (node_modules/@oclif/core/lib/command.js:139:23)
      at Build.run (packages/cli/src/commands/build/index.ts:46:18)
      at async Build._run (node_modules/@oclif/core/lib/command.js:117:22)
      at async createMultiChainTestProject (packages/cli/src/createProject.fixtures.ts:89:5)
          at async Promise.all (index 1)
      at async Object.<anonymous> (packages/cli/src/controller/publish-controller.spec.ts:18:21)


  ● Promise Utils › utils.promise delay()

    expect(received).toBeLessThan(expected)

    Expected: < 1050
    Received:   1717

      10 |     const millsecDiff = new Date().getTime() - start.getTime();
      11 |     expect(millsecDiff).toBeGreaterThanOrEqual(1000);
    > 12 |     expect(millsecDiff).toBeLessThan(1050);
         |                         ^
      13 |   });
      14 |
      15 |   it('utils.promise timeout()', async () => {

      at Object.<anonymous> (packages/node-core/src/utils/promise.spec.ts:12:25)

Report generated by 🧪jest coverage report action from 13767b1

stwiname
stwiname reviewed Nov 3, 2024
View reviewed changes
Copy link
Collaborator

@stwiname stwiname left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see how this solves any issues. From what I can tell all that has changed is that if the error is a rate limit error then it uses 20s rather than exponential backoff.

The rate limit delay is clearly way too large, (120s, 240s, 480s, 960s, 1920s). I think we would be better changing RETRY_DELAY to 10s (20, 40, 80, 160, 320 ).

packages/node-core/src/indexer/connectionPool.service.ts Outdated
Comment on lines 107 to 109
const rateLimitDelay = await this.poolStateManager.getFieldValue(endpoint, 'rateLimitDelay');
logger.info(`throtling on ratelimited endpoint ${rateLimitDelay / 1000}s`);
await delay(rateLimitDelay / 1000);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backoffDelay is now no longer used anywhere

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The backoffDelay parameter is mainly used when setting setRecoverTimeout, and there will be an interval function that displays the current endpoint status (function logEndpointStatus).

packages/node-core/src/indexer/connectionPoolState.manager.ts Show resolved Hide resolved
@yoozo
Copy link
Collaborator Author

yoozo commented Nov 4, 2024

I agree with your point about the rate limit delay being too large. I believe the purposes of backoffDelay and rateLimitDelay are different. backoffDelay is used to mark an endpoint as unhealthy for a period of time, during which we will not select this node for RPC requests. rateLimitDelay is only used in extreme cases when all endpoints are unavailable (all have been set with backoffDelay). At this point, we should set a fixed time (20 seconds) for the request, as this duration might be long enough.

@stwiname
Copy link
Collaborator

stwiname commented Nov 5, 2024

I agree with your point about the rate limit delay being too large. I believe the purposes of backoffDelay and rateLimitDelay are different. backoffDelay is used to mark an endpoint as unhealthy for a period of time, during which we will not select this node for RPC requests. rateLimitDelay is only used in extreme cases when all endpoints are unavailable (all have been set with backoffDelay). At this point, we should set a fixed time (20 seconds) for the request, as this duration might be long enough.

I don't think we need to have different delays for that, we can see why endpoints are being throttled and can use selection based on that. i.e. If we see endpoints are being rate limited and we still need an endpoint to use we can ignore the delay.

@yoozo yoozo requested a review from stwiname November 5, 2024 02:26
stwiname
stwiname reviewed Nov 6, 2024
View reviewed changes
packages/node-core/src/indexer/connectionPoolState.manager.ts Outdated
Comment on lines 283 to 292
const delayRule = [10, 20, 40, 80, 160, 320];
let delayTime = 0;
if (poolItem.failureCount < 1) {
delayTime = 0;
} else if (poolItem.failureCount >= 1 && poolItem.failureCount <= delayRule.length) {
delayTime = delayRule[poolItem.failureCount - 1];
} else {
delayTime = delayRule[delayRule.length - 1];
}
return delayTime * 1000;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can use the old logic, Just change RETRY_DELAY to 10*1000

packages/node-core/src/indexer/connectionPool.service.ts
@@ -102,13 +102,6 @@ export class ConnectionPoolService<T extends IApiConnectionSpecific<any, any, an
if (prop === 'fetchBlocks') {
return async (heights: number[], ...args: any): Promise<any> => {
try {
// Check if the endpoint is rate-limited
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still necessary or there will be no throttling

@yoozo yoozo requested a review from stwiname November 6, 2024 07:20
stwiname
stwiname reviewed Nov 7, 2024
View reviewed changes
packages/node-core/src/indexer/connectionPoolState.manager.ts
// Make sure there is no existing timeout
await this.clearTimeout(endpoint);

this.pool[endpoint].timeoutId = setTimeout(() => {
this.pool[endpoint].backoffDelay = 0; // Reset backoff delay only if there are no consecutive errors
this.pool[endpoint].rateLimited = false;
// this.pool[endpoint].rateLimited = false; // Do not reset rateLimited status
Copy link
Collaborator

@stwiname stwiname Nov 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think here we need to set a new timeout of the same value so it stayed rate limited? So every time a request is made there's a delay until the next request for that endpoint. If theres an error then the failure count should increase and therefore increasing the timeout.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding this here doesn’t avoid the problem; it just delays its appearance.

@stwiname stwiname merged commit 26dca74 into main Nov 12, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stwiname stwiname stwiname approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[cosmos indexer] Question about delay after indexer sync to the latest block.
2 participants
@yoozo @stwiname