Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(iota-json-rpc-api): Restore explorer metric endpoints #1845

Merged
merged 16 commits into from
Aug 21, 2024

Conversation

samuel-rufi
Copy link
Member

@samuel-rufi samuel-rufi commented Aug 14, 2024

Description of change

Restores the explorer metric endpoints.

Links to any relevant issues

Fixes #1619

Dependencies

How the change has been tested

  • I have run the docker-compose setup and tested the RPC endpoints with curl.

Try it yourself:

cd docker/pg-services-local
docker compose build
docker compose up -d
curl -s localhost:9005 --header 'Content-Type: application/json' --data '{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "iotax_getEpochMetrics",
  "params": []
}' | json_pp

Type of change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

@samuel-rufi samuel-rufi self-assigned this Aug 14, 2024
@samuel-rufi samuel-rufi force-pushed the l1sc/infra/explorer-endpoints branch 2 times, most recently from c9793a2 to 37f5a1b Compare August 19, 2024 13:55
@samuel-rufi samuel-rufi marked this pull request as ready for review August 19, 2024 14:13
@samuel-rufi samuel-rufi requested review from a team as code owners August 19, 2024 14:13
Base automatically changed from sc-platform/pg-services-docker to develop August 19, 2024 14:21
@samuel-rufi samuel-rufi marked this pull request as draft August 20, 2024 08:22
@samuel-rufi samuel-rufi force-pushed the l1sc/infra/explorer-endpoints branch from 37f5a1b to 72c78ca Compare August 20, 2024 08:51
@samuel-rufi samuel-rufi marked this pull request as ready for review August 20, 2024 08:51
Copy link
Contributor

@kodemartin kodemartin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job @samuel-rufi! Left a few comments and suggestions of non-critical nature nevertheless.

docker/pg-services-local/docker-compose.yaml Outdated Show resolved Hide resolved
docker/pg-services-local/docker-compose.yaml Outdated Show resolved Hide resolved
docker/pg-services-local/docker-compose.yaml Outdated Show resolved Hide resolved
crates/iota-json-rpc-types/src/iota_extended.rs Outdated Show resolved Hide resolved
crates/iota-json-rpc-types/src/iota_extended.rs Outdated Show resolved Hide resolved
crates/iota-json-rpc-api/src/extended.rs Outdated Show resolved Hide resolved
crates/iota-indexer/src/store/mod.rs Outdated Show resolved Hide resolved
crates/iota-indexer/src/handlers/mod.rs Outdated Show resolved Hide resolved
@samuel-rufi samuel-rufi requested review from thibault-martinez and a team August 20, 2024 10:25
@kodemartin kodemartin added sc-platform Issues related to the Smart Contract Platform group. infrastructure Issues related to the Infrastructure Team labels Aug 20, 2024
@samuel-rufi samuel-rufi requested review from sergiupopescu199, tomxey and Thoralf-M and removed request for a team August 20, 2024 10:34
@kodemartin
Copy link
Contributor

One more thing @samuel-rufi: I suggest changing the target branch into a new update-rpc-indexer-metrics (branched off from `develop) so that we stage all the related changes together, and coordinate with the tooling team (cc @begonaalvarezd, @marc2332, @msarcev )

In this context please also regenerate the openrpc.json in iota-open-rpc.

@samuel-rufi samuel-rufi requested a review from a team as a code owner August 20, 2024 13:37
@samuel-rufi samuel-rufi changed the base branch from develop to update-rpc-indexer-metrics August 20, 2024 13:39
@samuel-rufi samuel-rufi requested a review from kodemartin August 20, 2024 13:40
@samuel-rufi samuel-rufi requested a review from Thoralf-M August 20, 2024 15:29
Copy link
Contributor

@kodemartin kodemartin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 🌸

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the scopes of processors and handlers overlap after all, creating confusion on what should be where. Handlers process data, as well as processors in a way.

We should care of making this less ambiguous. Therefore I think it would be plausible to consolidate everything into a single module (handlers or processors), remove any suffixes/prefixes handlers or processors and classify between metrics in a nested module (what is now in the processors module except for the ObjectSnapshotProcessor) and the rest.

Nevertheless, this is a refactoring task that should not be part of the testnet milestone. Will create an issue accordingly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@sergiupopescu199 sergiupopescu199 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm 🚀

Copy link
Contributor

@tomxey tomxey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@samuel-rufi samuel-rufi merged commit 7dc520a into update-rpc-indexer-metrics Aug 21, 2024
49 of 51 checks passed
@samuel-rufi samuel-rufi deleted the l1sc/infra/explorer-endpoints branch August 21, 2024 11:36
@marc2332
Copy link
Contributor

One more thing @samuel-rufi: I suggest changing the target branch into a new update-rpc-indexer-metrics (branched off from `develop) so that we stage all the related changes together, and coordinate with the tooling team (cc @begonaalvarezd, @marc2332, @msarcev )

In this context please also regenerate the openrpc.json in iota-open-rpc.

Any idea why this never returns anything? What am I issing?
image
image

@samuel-rufi
Copy link
Member Author

samuel-rufi commented Aug 26, 2024

@marc2332 the endpoint works in batches - it waits for enough checkpoints to calculate the metrics. The threshold default value is currently set as ADDRESS_PROCESSOR_BATCH_SIZE: usize = 80000;. However you can override this value if you like for example by adding - ADDRESS_PROCESSOR_BATCH_SIZE=100 to the indexer-analytics environment in the docker-compose. Make sure you re-create the container (e.g. docker-compose up -d --force-recreate) so that it includes your env variable.

curl -s localhost:9005 --header 'Content-Type: application/json' --data '{ "jsonrpc": "2.0", "id": 1, "method": "iotax_getAllEpochAddressMetrics", "params": [] }' | json_pp
{
   "id" : 1,
   "jsonrpc" : "2.0",
   "result" : [
      {
         "checkpoint" : 126,
         "cumulativeActiveAddresses" : 1,
         "cumulativeAddresses" : 412,
         "dailyActiveAddresses" : 1,
         "epoch" : 0,
         "timestampMs" : 1724673353380
      }
   ]
}

samuel-rufi added a commit that referenced this pull request Sep 5, 2024
* feat(iota-json-rpc-api): Add explorer endpoints back

* feat(iota-json-rpc-api): Add metric processors back

* refactor(iota-indexer): fix schema usages

* refactor(iota-indexer): redo `ExtendedApi` methods

* refactor(iota-indexer): Introduce analytical-worker in docker-compose

* refactor(iota-benchmark, iota-core): Fix format

* refactor: Adjust docker-compose params and ports

* refactor: Add missing documentation

* refactor: Explicit export

Co-authored-by: Konstantinos Demartinos <[email protected]>

* refactor: Move processors

* refactor: Regenerate OpenRPC spec

* refactor: Add license

* refactor: Fix import

* refactor: Remove outdated comment

* refactor: Add comments to clarify `address` vs `active_address` meaning

---------

Co-authored-by: Konstantinos Demartinos <[email protected]>
samuel-rufi added a commit that referenced this pull request Sep 9, 2024
* feat(iota-json-rpc-api): Restore explorer metric endpoints (#1845)

* feat(iota-json-rpc-api): Add explorer endpoints back

* feat(iota-json-rpc-api): Add metric processors back

* refactor(iota-indexer): fix schema usages

* refactor(iota-indexer): redo `ExtendedApi` methods

* refactor(iota-indexer): Introduce analytical-worker in docker-compose

* refactor(iota-benchmark, iota-core): Fix format

* refactor: Adjust docker-compose params and ports

* refactor: Add missing documentation

* refactor: Explicit export

Co-authored-by: Konstantinos Demartinos <[email protected]>

* refactor: Move processors

* refactor: Regenerate OpenRPC spec

* refactor: Add license

* refactor: Fix import

* refactor: Remove outdated comment

* refactor: Add comments to clarify `address` vs `active_address` meaning

---------

Co-authored-by: Konstantinos Demartinos <[email protected]>

* fix: Remove unimplemented ExtendedApi::query_objects

* fix: fmt

* feat(tooling-sdk): Sync TS-SDK APIs (#2250)

* fix(tooling-sdk): Run generate scripts

* fix(tooling-sdk): Update manual types

* fix(tooling-sdk): Regenerate types again (after rebase)

* fix(tooling-sdk): Regenerate types again (after base branch fixes)

* fix(tooling-explorer): Remove usage of removed type (name service) and mock the client methods for now

* fix(tooling-sdk): Ignore unused arguments in client without ts-ignore

* fix(tooling-sdk): Fix build issues triggered by type changes

* fix(tooling-sdk): Support newly added methods in client.ts

* fix(tooling-sdk): Run codegen in graphql-transport

* fix(tooling-sdk): Add graphql compatibility tests for new methods (skip them as they don't pass atm)

* fix(tooling-sdk): Remove nameservice related methods from graphql-transport

* fix(tooling-sdk): Add changeset for sdk & graphql-transport packages

* fix(tooling-sdk): Make dprint like the changeset format

* fix(tooling-sdk): Make dprint like the changeset format

* feat: fmt

---------

Co-authored-by: Konstantinos Demartinos <[email protected]>
Co-authored-by: Mario <[email protected]>
marc2332 pushed a commit that referenced this pull request Sep 10, 2024
* feat(iota-json-rpc-api): Restore explorer metric endpoints (#1845)

* feat(iota-json-rpc-api): Add explorer endpoints back

* feat(iota-json-rpc-api): Add metric processors back

* refactor(iota-indexer): fix schema usages

* refactor(iota-indexer): redo `ExtendedApi` methods

* refactor(iota-indexer): Introduce analytical-worker in docker-compose

* refactor(iota-benchmark, iota-core): Fix format

* refactor: Adjust docker-compose params and ports

* refactor: Add missing documentation

* refactor: Explicit export

Co-authored-by: Konstantinos Demartinos <[email protected]>

* refactor: Move processors

* refactor: Regenerate OpenRPC spec

* refactor: Add license

* refactor: Fix import

* refactor: Remove outdated comment

* refactor: Add comments to clarify `address` vs `active_address` meaning

---------

Co-authored-by: Konstantinos Demartinos <[email protected]>

* fix: Remove unimplemented ExtendedApi::query_objects

* fix: fmt

* feat(tooling-sdk): Sync TS-SDK APIs (#2250)

* fix(tooling-sdk): Run generate scripts

* fix(tooling-sdk): Update manual types

* fix(tooling-sdk): Regenerate types again (after rebase)

* fix(tooling-sdk): Regenerate types again (after base branch fixes)

* fix(tooling-explorer): Remove usage of removed type (name service) and mock the client methods for now

* fix(tooling-sdk): Ignore unused arguments in client without ts-ignore

* fix(tooling-sdk): Fix build issues triggered by type changes

* fix(tooling-sdk): Support newly added methods in client.ts

* fix(tooling-sdk): Run codegen in graphql-transport

* fix(tooling-sdk): Add graphql compatibility tests for new methods (skip them as they don't pass atm)

* fix(tooling-sdk): Remove nameservice related methods from graphql-transport

* fix(tooling-sdk): Add changeset for sdk & graphql-transport packages

* fix(tooling-sdk): Make dprint like the changeset format

* fix(tooling-sdk): Make dprint like the changeset format

* feat: fmt

---------

Co-authored-by: Konstantinos Demartinos <[email protected]>
Co-authored-by: Mario <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Issues related to the Infrastructure Team sc-platform Issues related to the Smart Contract Platform group.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Task (Infrastructure)]: Restore extended api metrics endpoints
6 participants