diff --git a/.gitbook.yaml b/.gitbook.yaml index 3b370ff76..b634c7bff 100644 --- a/.gitbook.yaml +++ b/.gitbook.yaml @@ -2,40 +2,3 @@ root: ./docs/ structure: summary: SUMMARY.md - -redirects: - concepts-and-fundamentals/modules: ./new/common/manifest-modules.md - developers-guide/creating-protobuf-schema: ./new/develop/creating-protobuf-schemas.md - developers-guide/creating-your-manifest: ./new/common/manifest-modules.md - developers-guide/gui: ./new/common/manifest-modules.md - developers-guide/installation-requirements: ./new/common/installing-the-cli.md - developers-guide/overview: ./new/common/introduction.md - developers-guide/parallel-execution: ./new/develop/architecture.md - developers-guide/running-substreams: ./new/common/running-substreams.md - developers-guide/modules/inputs: ./new/develop/modules/inputs.md - developers-guide/modules/outputs: ./new/develop/modules/outputs.md - developers-guide/modules/setting-up-handlers: ./new/develop/modules/setting-up-handlers.md - developers-guide/modules/types: ./new/develop/modules/types.md - developers-guide/modules/writing-module-handlers: ./new/develop/modules/writing-module-handlers.md - developers-guide/cookbook/advanced-params: ./new/develop/parameterized-modules.md - developers-guide/cookbook/aggregation-windows: ./new/develop/modules/aggregation-windows.md - developers-guide/cookbook/dynamic-data-sources: ./new/develop/modules/dynamic-data-sources.md - developers-guide/cookbook/factory-params: ./new/develop/parameterized-modules.md - developers-guide/cookbook/keys-in-stores: ./new/develop/modules/keys-in-stores.md - developers-guide/sink-targets/substreams-powered-subgraph: ./new/consume/subgraph/quickstart.md - developers-guide/sink-targets/substreams-sink-sql: ./new/consume/sql/introduction.md - getting-started/installing-the-cli: ./new/common/installing-the-cli.md - getting-started/quickstart: ./new/common/introduction.md - references-and-specs/chains-and-endpoints: ./new/references/chains-and-endpoints.md - references-and-specs/command-line-interface: ./new/references/command-line-interface.md - references-and-specs/examples: ./new/tutorials/overview.md - references-and-specs/gui: ./new/references/gui.md - references-and-specs/faq: ./new/references/faq.md - references-and-specs/manifests: ./new/references/manifests.md - references-and-specs/packages: ./new/common/packages.md - references-and-specs/rust-crates: ./new/develop/rust-crates.md - glossary/glossary: ./new/references/glossary.md - quick-access/change-log: ./new/references/change-log.md - - - diff --git a/docs/.gitbook/assets/chains-endpoints.png b/docs/.gitbook/assets/chains-endpoints.png index dddeb0332..77532cfae 100644 Binary files a/docs/.gitbook/assets/chains-endpoints.png and b/docs/.gitbook/assets/chains-endpoints.png differ diff --git a/docs/.gitbook/assets/consume/consume-services-only.png b/docs/.gitbook/assets/consume/consume-services-only.png deleted file mode 100644 index a82dc9948..000000000 Binary files a/docs/.gitbook/assets/consume/consume-services-only.png and /dev/null differ diff --git a/docs/.gitbook/assets/consume/consume-services.png b/docs/.gitbook/assets/consume/consume-services.png deleted file mode 100644 index 651bd464c..000000000 Binary files a/docs/.gitbook/assets/consume/consume-services.png and /dev/null differ diff --git a/docs/.gitbook/assets/consume/featured-packages.png b/docs/.gitbook/assets/consume/featured-packages.png deleted file mode 100644 index 0abd20640..000000000 Binary files a/docs/.gitbook/assets/consume/featured-packages.png and /dev/null differ diff --git a/docs/.gitbook/assets/consume/service-sql.png b/docs/.gitbook/assets/consume/service-sql.png deleted file mode 100644 index 1f03f8b4e..000000000 Binary files a/docs/.gitbook/assets/consume/service-sql.png and /dev/null differ diff --git a/docs/.gitbook/assets/consume/service-stream.png b/docs/.gitbook/assets/consume/service-stream.png deleted file mode 100644 index 0e017621c..000000000 Binary files a/docs/.gitbook/assets/consume/service-stream.png and /dev/null differ diff --git a/docs/.gitbook/assets/consume/service-subgraph.png b/docs/.gitbook/assets/consume/service-subgraph.png deleted file mode 100644 index eb3ced932..000000000 Binary files a/docs/.gitbook/assets/consume/service-subgraph.png and /dev/null differ diff --git a/docs/.gitbook/assets/develop/provider-gif.gif b/docs/.gitbook/assets/develop/provider-gif.gif deleted file mode 100644 index 10b8980f4..000000000 Binary files a/docs/.gitbook/assets/develop/provider-gif.gif and /dev/null differ diff --git a/docs/.gitbook/assets/develop/rust-package-gif.gif b/docs/.gitbook/assets/develop/rust-package-gif.gif deleted file mode 100644 index 4648aec29..000000000 Binary files a/docs/.gitbook/assets/develop/rust-package-gif.gif and /dev/null differ diff --git a/docs/.gitbook/assets/develop/sending-gif.gif b/docs/.gitbook/assets/develop/sending-gif.gif deleted file mode 100644 index 28cd35176..000000000 Binary files a/docs/.gitbook/assets/develop/sending-gif.gif and /dev/null differ diff --git a/docs/.gitbook/assets/develop/transformations-gif.gif b/docs/.gitbook/assets/develop/transformations-gif.gif deleted file mode 100644 index e1e357d2d..000000000 Binary files a/docs/.gitbook/assets/develop/transformations-gif.gif and /dev/null differ diff --git a/docs/.gitbook/assets/intro/consume-flow.png b/docs/.gitbook/assets/intro/consume-flow.png deleted file mode 100644 index 485bafc12..000000000 Binary files a/docs/.gitbook/assets/intro/consume-flow.png and /dev/null differ diff --git a/docs/.gitbook/assets/intro/develop-flow.png b/docs/.gitbook/assets/intro/develop-flow.png deleted file mode 100644 index 1665d3add..000000000 Binary files a/docs/.gitbook/assets/intro/develop-flow.png and /dev/null differ diff --git a/docs/.gitbook/assets/intro/supported-chains.png b/docs/.gitbook/assets/intro/supported-chains.png deleted file mode 100644 index fbb2f5fb9..000000000 Binary files a/docs/.gitbook/assets/intro/supported-chains.png and /dev/null differ diff --git a/docs/.gitbook/assets/packages/arch.png b/docs/.gitbook/assets/packages/arch.png deleted file mode 100644 index 755d414ff..000000000 Binary files a/docs/.gitbook/assets/packages/arch.png and /dev/null differ diff --git a/docs/.gitbook/assets/packages/erc20-balance-changes.png b/docs/.gitbook/assets/packages/erc20-balance-changes.png deleted file mode 100644 index f24d63c8a..000000000 Binary files a/docs/.gitbook/assets/packages/erc20-balance-changes.png and /dev/null differ diff --git a/docs/.gitbook/assets/sql/explore-pgweb.png b/docs/.gitbook/assets/sql/explore-pgweb.png deleted file mode 100644 index 368bea4de..000000000 Binary files a/docs/.gitbook/assets/sql/explore-pgweb.png and /dev/null differ diff --git a/docs/README.md b/docs/README.md index 90fb55b5e..563b3104b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,42 +1,68 @@ -Substreams is a powerful indexing technology, which allows you to: -- Extract data from several blockchains (Ethereum, Polygon, BNB, Solana...). -- Apply custom transformations to the data. -- Send the data to a place of your choice (for example, a Postgres database or a file). +--- +description: StreamingFast Substreams documentation +--- -
+# Substreams -**You can use Substreams packages to define which specific data you want to extract from the blockchain**. For example, consider that you want to retrieve data from the Uniswap v3 smart contract. You can simply use the [Uniswap v3 Substreams Package](https://substreams.dev/streamingfast/uniswap-v3/v0.2.7) and send that data wherever you want! +## Welcome to Substreams documentation -## Consume Substreams +Substreams is a powerful blockchain indexing technology, developed for [The Graph Network](https://thegraph.com). -There are many ready-to-use Substreams packages, so you can simply consume them. Use the **([Substreams.dev Registry](https://substreams.dev)) to explore packages**. +> Substreams enables developers to write Rust modules, composing data streams alongside the community, and provides extremely high-performance indexing by virtue of parallelization, in a streaming-first fashion. +> +> Substreams have all the benefits of StreamingFast Firehose, like low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling. -Once you find a package that fits your needs, you only have choose **how you want to consume the data**. Send the data to a SQL database, configure a webhook or stream directly from your application! +### Where to start -
+Learn about Substreams in a short, dense 25-minute intro and understand its impact on the blockchain ecosystem. -## Develop Substreams +{% embed url="https://www.youtube.com/watch?v=K-nhC2FCB5k" %} +A walkthrough of Firehose features, Substreams modules, including a sample Rust module and StreamingFast's vision. +{% endembed %} -If you can't find a Substreams package that retrieves exactly the data you need, **you can develop your own Substreams**. +Learn about the benefits of Substreams, and how it compares to otheressential facts about Substreams through [reading the Benefits and comparison](concepts-and-fundamentals/benefits.md). -You can write your own Rust function to extract data from the blockchain: +The primary ways to use Substreams include: -```rust -fn get_usdt_transaction(block: eth::Block) -> Result, substreams:error:Error> { - let my_transactions = block.transactions(). - .filter(|transaction| transaction.to == USDT_CONTRACT_ADDRESS) - .map(|transaction| MyTransaction(transaction.hash, transaction.from, transaction.to)) - .collect(); - Ok(my_transactions) -} -``` +* [Installing the `substreams` CLI](getting-started/installing-the-cli.md) +* [Going through the Quickstart](getting-started/quickstart.md) -
+After installing Substreams and reviewing the Quickstart: -## How Does It Work? +* You can [learn more about ](developers-guide/modules/)modules, and then [study the Developer's guide](developers-guide/overview.md). -The following video covers how Substreams works in less than 2 minutes: +Find pre-built Substreams by using the following resources: -{% embed url="https://www.youtube.com/watch?v=gVqGCqKVM08" %} -Get an overview of Substreams -{% endembed %} \ No newline at end of file +* The [Substreams Template](https://github.com/streamingfast/substreams-template) helps expedite the process of getting you up and running. +* A [list of maintained Substreams examples](reference-and-specs/examples.md) + +### Network model diagram + + + +**You can view Substreams from two perspectives** as illustrated in the high-level visual diagram. It can be viewed through the perspective of the **Substreams engine** itself and also the perspective of the **end-user developer and consumer**. + +### Community + +Substreams is an open source community effort, so feel free to suggest new topics, report issues, and provide feedback. Contribute through GitHub [pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). + +* [StreamingFast Discord](https://discord.gg/mYPcRAzeVN) +* [The Graph Discord](https://discord.gg/vtvv7FP) +* [StreamingFast on Twitter](https://twitter.com/streamingfastio) +* [StreamingFast on YouTube](https://www.youtube.com/c/streamingfast) + +### Contributing + +For additional information, [refer to the general StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md). + +### License + +Substreams uses the [Apache 2.0](../LICENSE/) license. + +### Disclaimer + +The content in the Substreams documentation was created through StreamingFast's full effort. It is up to the reader to validate the accuracy of all content presented. Substreams is in active development and, at times, the associated documentation becomes outdated. [Contact StreamingFast](https://discord.gg/mYPcRAzeVN) to report problems or service interruptions. + +{% hint style="info" %} +**Note**: The Substreams documentation uses the [Google developer documentation style guide](https://developers.google.com/style) for its style and formatting. +{% endhint %} diff --git a/docs/README_old.md b/docs/README_old.md deleted file mode 100644 index 563b3104b..000000000 --- a/docs/README_old.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -description: StreamingFast Substreams documentation ---- - -# Substreams - -## Welcome to Substreams documentation - -Substreams is a powerful blockchain indexing technology, developed for [The Graph Network](https://thegraph.com). - -> Substreams enables developers to write Rust modules, composing data streams alongside the community, and provides extremely high-performance indexing by virtue of parallelization, in a streaming-first fashion. -> -> Substreams have all the benefits of StreamingFast Firehose, like low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling. - -### Where to start - -Learn about Substreams in a short, dense 25-minute intro and understand its impact on the blockchain ecosystem. - -{% embed url="https://www.youtube.com/watch?v=K-nhC2FCB5k" %} -A walkthrough of Firehose features, Substreams modules, including a sample Rust module and StreamingFast's vision. -{% endembed %} - -Learn about the benefits of Substreams, and how it compares to otheressential facts about Substreams through [reading the Benefits and comparison](concepts-and-fundamentals/benefits.md). - -The primary ways to use Substreams include: - -* [Installing the `substreams` CLI](getting-started/installing-the-cli.md) -* [Going through the Quickstart](getting-started/quickstart.md) - -After installing Substreams and reviewing the Quickstart: - -* You can [learn more about ](developers-guide/modules/)modules, and then [study the Developer's guide](developers-guide/overview.md). - -Find pre-built Substreams by using the following resources: - -* The [Substreams Template](https://github.com/streamingfast/substreams-template) helps expedite the process of getting you up and running. -* A [list of maintained Substreams examples](reference-and-specs/examples.md) - -### Network model diagram - - - -**You can view Substreams from two perspectives** as illustrated in the high-level visual diagram. It can be viewed through the perspective of the **Substreams engine** itself and also the perspective of the **end-user developer and consumer**. - -### Community - -Substreams is an open source community effort, so feel free to suggest new topics, report issues, and provide feedback. Contribute through GitHub [pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). - -* [StreamingFast Discord](https://discord.gg/mYPcRAzeVN) -* [The Graph Discord](https://discord.gg/vtvv7FP) -* [StreamingFast on Twitter](https://twitter.com/streamingfastio) -* [StreamingFast on YouTube](https://www.youtube.com/c/streamingfast) - -### Contributing - -For additional information, [refer to the general StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md). - -### License - -Substreams uses the [Apache 2.0](../LICENSE/) license. - -### Disclaimer - -The content in the Substreams documentation was created through StreamingFast's full effort. It is up to the reader to validate the accuracy of all content presented. Substreams is in active development and, at times, the associated documentation becomes outdated. [Contact StreamingFast](https://discord.gg/mYPcRAzeVN) to report problems or service interruptions. - -{% hint style="info" %} -**Note**: The Substreams documentation uses the [Google developer documentation style guide](https://developers.google.com/style) for its style and formatting. -{% endhint %} diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 7a354b34e..c9a41200a 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -2,59 +2,80 @@ * [Substreams](README.md) -## Documentation - -* [Consume Substreams](new/consume/introduction.md) - * [Install the CLI](new/common/installing-the-cli.md) - * [Authentication](new/common/authentication.md) - * [Packages](new/common/packages.md) - * [Deployable Services](new/common/deployable-services.md) - * [Substreams:SQL](new/consume/sql/introduction.md) - * [Deployable Services](new/consume/sql/deployable-services/introduction.md) - * [Remote Service](new/consume/sql/deployable-services/remote-service.md) - * [Local Service](new/consume/sql/deployable-services/local-service.md) - * [SQL Sink](new/consume/sql/sql-sink.md) - * [Substreams:Stream](new/consume/stream/quickstart.md) - * [JavaScript](new/consume/stream/javascript.md) - * [Substreams:Subgraph](new/consume/subgraph/quickstart.md) -* [Develop Substreams](new/develop/introduction.md) - * [Install the CLI](new/common/installing-the-cli.md) - * [Authentication](new/common/authentication.md) - * [Quickstart](new/develop/init-project.md) - * [Manifest & Modules](new/common/manifest-modules.md) - * [Module types](new/develop/modules/types.md) - * [Inputs](new/develop/modules/inputs.md) - * [Output](new/develop/modules/outputs.md) - * [Module handlers](new/develop/modules/setting-up-handlers.md) - * [Module handler creation](new/develop/modules/writing-module-handlers.md) - * [Keys in stores](new/develop/modules/keys-in-stores.md) - * [Dynamic data sources](new/develop/modules/dynamic-data-sources.md) - * [Aggregation Windows](new/develop/modules/aggregation-windows.md) - * [Protobuf Schemas](new/develop/creating-protobuf-schemas.md) - * [Packages](new/common/packages.md) - * [Run a Substreams](new/common/running-substreams.md) - * [Deployable Services](new/common/deployable-services.md) - * [Rust Dependencies](new/develop/rust-crates.md) - * [Parameterized Modules](new/develop/parameterized-modules.md) - * [Chain-Specific Extensions](new/develop/chain-specific/overview.md) - * [EVM Chain-Specific Extensions](new/develop/chain-specific/evm/overview.md) - * [Making eth_calls](new/develop/chain-specific/evm/eth-calls.md) - * [Architecture](new/develop/architecture.md) +## Quick Access + +* [Glossary](glossary/glossary.md) +* [Change log](release-notes/change-log.md) + +## Concepts & Fundamentals + +* [Benefits and comparisons](concepts-and-fundamentals/benefits.md) +* [Modules basics](concepts-and-fundamentals/modules.md) +* [Fundamentals](concepts-and-fundamentals/fundamentals.md) + +## Getting Started + +* [Installing the Substreams CLI](getting-started/installing-the-cli.md) +* [Quickstart](getting-started/quickstart.md) + +## Cheatsheet + +* [EVM](cheatsheet/evm/overview.md) + * [Blocks](cheatsheet/evm/blocks.md) + * [Transactions](cheatsheet/evm/transactions.md) + * [Logs](cheatsheet/evm/logs.md) ## Tutorials -* [Exploring Ethereum](new/tutorials/ethereum/exploring-ethereum/overview.md) - * [Mapping Blocks](new/tutorials/ethereum/exploring-ethereum/map_block_meta_module.md) - * [Filtering Transactions](new/tutorials/ethereum/exploring-ethereum/map_filter_transactions_module.md) - * [Retrieving Events of a Smart Contract](new/tutorials/ethereum/exploring-ethereum/map_contract_events_module.md) -* [Rust](new/tutorials/rust/overview.md) +* [From Ethereum address to SQL](tutorials/from-ethereum-address-to-SQL.md) +* [Substreams:SQL Deployable Service](tutorials/substreams-sql.md) +* [Exploring Ethereum](tutorials/exploring-ethereum/overview.md) + * [Mapping Blocks](tutorials/exploring-ethereum/map_block_meta_module.md) + * [Filtering Transactions](tutorials/exploring-ethereum/map_filter_transactions_module.md) + * [Retrieving Events of a Smart Contract](tutorials/exploring-ethereum/map_contract_events_module.md) +* [Making eth_calls](tutorials/eth-calls.md) +* [Rust](tutorials/rust/overview.md) + * [The Option struct](tutorials/rust/option.md) + * [The Result struct](tutorials/rust/result.md) + +## Developer's Guide + +* [Overview](developers-guide/overview.md) +* [Dependency installation](developers-guide/installation-requirements.md) +* [Manifest](developers-guide/creating-your-manifest.md) +* [Protobuf schemas](developers-guide/creating-protobuf-schemas.md) +* [Modules](developers-guide/modules/README.md) + * [Module types](developers-guide/modules/types.md) + * [Inputs](developers-guide/modules/inputs.md) + * [Output](developers-guide/modules/outputs.md) + * [Module handlers](developers-guide/modules/setting-up-handlers.md) + * [Module handler creation](developers-guide/modules/writing-module-handlers.md) +* [Running Substreams](developers-guide/running-substreams.md) +* [GUI](developers-guide/gui.md) +* [Parallel Execution](developers-guide/parallel-execution.md) +* [Sink targets](developers-guide/sink-targets/README.md) + * [Substreams-powered subgraph](developers-guide/sink-targets/substreams-powered-subgraph.md) + * [Files](developers-guide/sink-targets/substreams-sink-files.md) + * [Key/value store](developers-guide/sink-targets/substreams-sink-kv.md) + * [SQL (Postgres, Clickhouse, others)](developers-guide/sink-targets/substreams-sink-sql.md) + * [Prometheus](developers-guide/sink-targets/substreams-sink-prometheus.md) + * [MongoDB](developers-guide/sink-targets/substreams-sink-mongodb.md) + * [Custom Sink Using JavaScript](developers-guide/sink-targets/custom-sink-js.md) +* [Cookbook](developers-guide/cookbook.md) + * [Dynamic data sources](developers-guide/cookbook/dynamic-data-sources.md) + * [Factory parameterization](developers-guide/cookbook/factory-params.md) + * [Advanced parameters](developers-guide/cookbook/advanced-params.md) + * [Keys in stores](developers-guide/cookbook/keys-in-stores.md) + * [Aggregation windows](developers-guide/cookbook/aggregation-windows.md) + * [Subgraph entity changes](developers-guide/cookbook/entity-changes.md) ## Reference & Specs -* [Chains and endpoints](new/references/chains-and-endpoints.md) -* [Substreams CLI reference](new/references/command-line-interface.md) -* [Manifests Reference](new/references/manifests.md) -* [GUI Reference](new/references/gui.md) -* [Glossary](new/references/glossary.md) -* [Change log](new/references/change-log.md) -* [FAQ](new/references/faq.md) +* [Chains and endpoints](reference-and-specs/chains-and-endpoints.md) +* [Substreams CLI reference](reference-and-specs/command-line-interface.md) +* [Authentication](reference-and-specs/authentication.md) +* [Manifests](reference-and-specs/manifests.md) +* [Packages](reference-and-specs/packages.md) +* [Rust crates](reference-and-specs/rust-crates.md) +* [Examples](reference-and-specs/examples.md) +* [FAQ](reference-and-specs/faq.md) diff --git a/docs/new/common/authentication.md b/docs/new/common/authentication.md deleted file mode 100644 index 6f88eabc3..000000000 --- a/docs/new/common/authentication.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -description: Learn how to authenticate with a Substreams Provider ---- - -Running a Substreams involves sending your package (`.spkg`) to a a Substreams provider for execution. Usually, Substreams provider will require you to authenticate to avoid abuses on the service. - -## Authentication with StreamingFast - -### Obtain your API key - -First, obtain an API key by visiting our Portal: - -* [https://app.streamingfast.io](https://app.streamingfast.io) - -The StreamingFast team is also available on [Discord](https://discord.gg/jZwqxJAvRs) to help you get an API key. - -### Request your authentication token - -Use your API Key to obtain a shorter-lived authentication token using `curl`: - -```bash -curl -s https://auth.streamingfast.io/v1/auth/issue --data-binary '{"api_key": "your-secret-key"}' -``` - -### Set your environment variable - -Set the token as an `ENV` variable through the terminal by using: - -```bash -export SUBSTREAMS_API_TOKEN="your_token" -``` - -The `substreams` [`run`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#run) command checks the `SUBSTREAMS_API_TOKEN` environment variable for the token by default. You can change that with the `--substreams-api-token-envvar` flag. - -### All-in-one bash function - -Place this function in your terminal profile (`.bashrc` or `.zshrc`), for a quick all-in-one token fetcher: - -```bash -export STREAMINGFAST_KEY=server_YOUR_KEY_HERE -function sftoken { - export SUBSTREAMS_API_TOKEN=$(curl https://auth.streamingfast.io/v1/auth/issue -s --data-binary '{"api_key":"'$STREAMINGFAST_KEY'"}' | jq -r .token) - echo "Token set on in SUBSTREAMS_API_TOKEN" -} -``` - -Then obtain a new key and set it in your environment by running: - -```bash -$ sftoken -``` - -[^1]: A [JSON Web Token](https://jwt.io/), or JWT - -[^2]: The `api_key` specified here is one starting with `server_`, `web_` or `mobile_`, obtained through the StreamingFast Portal. - -[^3]: Install `jq` from [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/) diff --git a/docs/new/common/deployable-services.md b/docs/new/common/deployable-services.md deleted file mode 100644 index cd3fd7c7e..000000000 --- a/docs/new/common/deployable-services.md +++ /dev/null @@ -1,40 +0,0 @@ -The Substreams Deployable Services define a common interface to easily deploy your Substreams to one of the supported sinks, such as SQL or subgraphs. Essentially, it facilitates sending data to a variety of sinks by simply using the Substreams CLI. - -## Hoes Does It Work? - -1. Choose what sink you want to use (SQL or subgraphs). -2. Add the `sink` configuration to your manifest. -3. Use the `substreams alpha service` command to deploy, stop or remove your services. - -### Choose a Sink - -Depending on your needs, you must choose how you want to consume the data: using a SQL database or a subgraph. Substreams using the SQL sink must have a `db_out` module and those using the subgraph sink must have a `graph_out` module. - -### Add the Sink Configuration - -The `sink` configuration in a Substreams manifest defines what sink should be used. To get more information about the Substreams manifest, refer to the [Manifest & Modules page](manifest-modules.md) - -Every sink has different configuration fields available, so check out the Manifest Reference for more information. In the following example, a SQL sink is defined: - -``` -sink: - module: db_out - type: sf.substreams.sink.sql.v1.Service - config: - schema: "./schema.sql" - engine: clickhouse - postgraphile_frontend: - enabled: false - pgweb_frontend: - enabled: false - dbt_config: - enabled: true - files: "./path/to/folder" - run_interval_seconds: 300 -``` - -### Deploy the Service - -Use the `substreams alpha service ` command to manage your services. Once your Substreams has the corresponding manifest configuration, you can deploy it by using the `substreams alpha service deploy` command. - -You will get a service ID, which is a unique identifier for your service. This will allow you to manage your service and apply actions to it, such a stopping or removing it. \ No newline at end of file diff --git a/docs/new/common/installing-the-cli.md b/docs/new/common/installing-the-cli.md deleted file mode 100644 index afbb64934..000000000 --- a/docs/new/common/installing-the-cli.md +++ /dev/null @@ -1,94 +0,0 @@ ---- -description: StreamingFast Substreams CLI installation documentation ---- - -# Installing the Substreams CLI - -## Dependency installation - -Substreams requires a number of different applications and tools. Instructions and links are provided to assist in the installation of the required dependencies for Substreams. - -{% hint style="success" %} -**Tip**: Instructions are also provided for cloud-based Gitpod setups. -{% endhint %} - -### Rust installation - -Developing Substreams modules requires a working [Rust](https://www.rust-lang.org/) compilation environment. - -There are [several ways to install Rust](https://www.rust-lang.org/tools/install)**.** Install Rust through `curl` by using: - -```bash -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source $HOME/.cargo/env # to configure your current shell -``` - -#### `wasm32-unknown-unknown` target - -Ensure you have the `wasm32-unknown-unknown` target installed on your Rust installation, if unsure, you can install it with: - -```bash -rustup target add wasm32-unknown-unknown -``` - -### Buf installation - -Buf simplifies the generation of typed structures in any language. Buf uses a remote builder executed on the Buf server, so an internet connection is required to generate Rust bindings from Protobuf definitions. - -Visit the [Buf website](https://buf.build/) for additional information and [installation instructions](https://docs.buf.build/installation). - -{% hint style="info" %} -**Note**_:_ [Substreams packages](../reference-and-specs/packages.md) and [Buf images](https://docs.buf.build/reference/images) are compatible. -{% endhint %} - -## Install the `substreams` CLI - -Used for connecting to endpoints, streaming data in real time, and packaging custom modules. - -{% hint style="success" %} -**Tip**_:_ [Check the official Github repository](https://github.com/streamingfast/substreams/releases) to get the **latest** [**`substreams` CLI**](../reference-and-specs/command-line-interface.md) **release available**. -{% endhint %} - -### Homebrew installation - -``` -brew install streamingfast/tap/substreams -``` - -### Pre-compiled binary installation - -```bash -# Use correct binary for your platform -LINK=$(curl -s https://api.github.com/repos/streamingfast/substreams/releases/latest | awk '/download.url.*linux/ {print $2}' | sed 's/"//g') -curl -L $LINK | tar zxf - -``` - -### Installation from source - -```bash -git clone https://github.com/streamingfast/substreams -cd substreams -go install -v ./cmd/substreams -``` - -{% hint style="warning" %} -**Important**: Add $HOME/go/bin to the system path if it's not already present. -{% endhint %} - -## Validation of installation - -Run the [`substreams` CLI](../reference-and-specs/command-line-interface.md) passing the `--version` flag to check the success of the installation. - -```bash -substreams --version -``` - -A successful installation will print the version that you have installed. - -```bash -substreams version dev -``` - -{% hint style="info" %} -**Note**: You can [also use Gitpod](../developers-guide/installation-requirements.md) instead of a local installation. -{% endhint %} diff --git a/docs/new/common/introduction.md b/docs/new/common/introduction.md deleted file mode 100644 index 87a0068c4..000000000 --- a/docs/new/common/introduction.md +++ /dev/null @@ -1,6 +0,0 @@ - -Substreams is a technology that allows you to extract blockchain data in a fast and reliable way! - -## Consume Substreams - -## Develop Substreams \ No newline at end of file diff --git a/docs/new/common/manifest-modules.md b/docs/new/common/manifest-modules.md deleted file mode 100644 index 6ba39a408..000000000 --- a/docs/new/common/manifest-modules.md +++ /dev/null @@ -1,129 +0,0 @@ ---- -description: Learn the basics about modules and manifests ---- - -## Modules and Manifests - -In Substreams, manifests and modules are concepts tighly related because they are fundamental to understand how Substreams works. - -In simple terms, a Substreams module is a Rust function that receives an input and returns an output. For example, the following Rust function receives an Ethereum block and returns a custom object containing fields such as block number, hash or parent hash. - -```rust -fn get_my_block(blk: Block) -> Result { - let header = blk.header.as_ref().unwrap(); - - Ok(MyBlock { - number: blk.number, - hash: Hex::encode(&blk.hash), - parent_hash: Hex::encode(&header.parent_hash), - }) -} -``` - -And also in simple terms, a Substreams manifest (`substreams.yaml`) is a configuration file (a YAML file) for your Substreams, which defines the different modules (functions) for your Substreams, among other configurations. For example, the following manifest receives a raw Ethereum block as input (`sf.ethereum.type.v2.Block`) and outputs a custom object (`eth.example.MyBlock`). - -```yaml -modules: - - name: map_block - kind: map - initialBlock: 12287507 - inputs: - - source: sf.ethereum.type.v2.Block - output: - type: proto:eth.example.MyBlock -``` - -Among other things, the manifest allows you to define: -- How many modules your Substreams uses, along with their corresponding inputs and outputs. -- The schema(s) (i.e. the data model) your Substreams uses. -- How you will consume the data emitted by your Substreams (SQL, Webhooks...). - -## Module Chaining - -Modules were built with composability in mind, so it is possible to chain them. Given two modules, `module1` and `module2`, you can set the output of `module1` to be the input of `module2`, creating a chain of interconnected Substreams modules. Let's take a look at the following example: - -```yaml -modules: - - name: map_events - kind: map - initialBlock: 4634748 - inputs: - - source: sf.ethereum.type.v2.Block - output: - type: proto:contract.v1.Events - - - name: db_out - kind: map - initialBlock: 4634748 - inputs: - - map: map_events - output: - type: proto:sf.substreams.sink.database.v1.DatabaseChanges -``` - -There are two modules defined: `map_events` and `graph_out`. -- The `map_events` module receives a `sf.ethereum.type.v2.Block` object (a raw Ethereum block) as a parameter and outputs a custom `contract.v1.Events` object. -- The `db_out` module receives `map_events`'s output as an input, and outputs another custom object, `sf.substreams.sink.database.v1.DatabaseChanges`. - -Technically, modules have one or more inputs, which can be in the form of a `map` or `store`, or a `Block` or `Clock` object received from the blockchain's data source. Every time a new `Block` is processed, all of the modules are executed as a directed acyclic graph (DAG). - -## Module Kinds - -There are two types of modules: `map` and `store`. `map` modules are used for stateless transformations and `store` modules are used for stateful transformations. - -Substreams executes the Rust function associated with module for every block on the blockchain, but there will be times when you will have to save data between blocks. `store` modules allow you to save in-memory data. - -### `map` modules - -`map` modules are used for data extraction, filtering, and transformation. They should be used when direct extraction is needed avoiding the need to reuse them later in the DAG. - -To optimize performance, you should use a single `map` module instead of multiple `map` modules to extract single events or functions. It is more efficient to perform the maximum amount of extraction in a single top-level `map` module and then pass the data to other Substreams modules for consumption. This is the recommended, simplest approach for both backend and consumer development experiences. - -Functional `map` modules have several important use cases and facts to consider, including: - -* Extracting model data from an event or function's inputs. -* Reading data from a block and transforming it into a custom protobuf structure. -* Filtering out events or functions for any given number of contracts. - -### `store` modules - -`store` modules are used for the aggregation of values and to persist state that temporarily exists across a block. - -{% hint style="warning" %} -**Important:** Stores should not be used for temporary, free-form data persistence. -{% endhint %} - -Unbounded `store` modules are discouraged. `store` modules shouldn't be used as an infinite bucket to dump data into. - -Notable facts and use cases for working `store` modules include: - -* `store` modules should only be used when reading data from another downstream Substreams module. -* `store` modules cannot be output as a stream, except in development mode. -* `store` modules are used to implement the Dynamic Data Sources pattern from Subgraphs, keeping track of contracts created to filter the next block with that information. -* Downstream of the Substreams output, do not use `store` modules to query anything from them. Instead, use a sink to shape the data for proper querying. - -## Defining Modules - -Modules are defined as a YAML list under the `modules` section of the manifest. In the following example, a `map_events` module is defined: - -```yaml -modules: - - name: map_events - kind: map - initialBlock: 4634748 - inputs: - - source: sf.ethereum.type.v2.Block - output: - type: proto:contract.v1.Events -``` - -Then, you create the corresponding Rust function under the `src/lib.rs` file. - -```rust -#[substreams::handlers::map] -fn map_events(blk: eth::Block) -> Result { - -...output omitted... - -} -``` diff --git a/docs/new/common/packages.md b/docs/new/common/packages.md deleted file mode 100644 index 5f414f1c8..000000000 --- a/docs/new/common/packages.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -description: Learn about basics of Substreams packages ---- - -There are a lot of developers building Substreams and creating very useful transformations that can be reused by other people. -Once a Substreams is developed, you can pack it into a Substreams package and share it with other people! - -Essentially, a Substreams package is a ready-to-consume binary file, which contains all the necessary dependencies (manifest, modules, protobufs...). The standard file extension for a Substreams package is `.spkg`. - -
- -## The Substreams Registry - -In order to facilitate how developers share Substreams packages, the Substreams Registry (https://substreams.dev) was created. In the Registry, deelopers can discover and push Substreams. - -For example, the [ERC20 Balance Changes](https://github.com/streamingfast/substreams-erc20-balance-changes) package is stored at the registry (https://substreams.dev/streamingfast/erc20-balance-changes/v1.1.0). - -
- -## Using a Package - -You can easily run a Substreams package by inputting the `.spkg` file in the CLI: - -```bash -substreams gui \ - https://spkg.io/streamingfast/erc20-balance-changes-v1.1.0.spkg \ - map_balance_changes \ - -e mainnet.eth.streamingfast.io:443 \ - --start-block 1397553 -``` - -## Creating a Package - -You can create a Substreams package by executing the `substreams pack` command in the CLI. Given a Substreams project, you can create a new package from a manifest (`substreams.yaml`): - -```bash -substreams pack ./substreams.yaml -``` - -### Package Dependencies - -Developers can use modules and protobuf definitions from other Substreams packages when `imports` is defined in the manifest. - -{% hint style="warning" %} -**Important**: To avoid potential naming collisions select unique `.proto` filenames and namespaces specifying fully qualified paths. -{% endhint %} - -Local protobuf filenames take precedence over the imported package's proto files. \ No newline at end of file diff --git a/docs/new/common/running-substreams.md b/docs/new/common/running-substreams.md deleted file mode 100644 index 5ac838562..000000000 --- a/docs/new/common/running-substreams.md +++ /dev/null @@ -1,123 +0,0 @@ - -Running a Substreams means executing a Substreams package and consuming the data emitted. The data produced by a Substreams can be consumed in a variety of ways, such as using the CLI, a SQL database or a simple files. - -## Running a Local Project - -A typical Substreams local project contains Protobufs (`1`), modules (`2`) and a manifest (`substreams.yaml`) file (`3`): - -

Ethereum Explorer Project Structure

- -If you are currently working on a local project, you can run your Substreams by specifying the location of the manifest: - -```bash -substreams run -e mainnet.eth.streamingfast.io:443 \ - substreams.yaml \ - map_transfers \ - --start-block 12292922 \ - --stop-block +1 -``` - -### Substreams `run` - -First, start the [`substreams` CLI](../reference-and-specs/command-line-interface.md) passing it a [`run`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#run) command. - -### Firehose URI - -The server address is required by Substreams to connect to for data retrieval. The data provider for Substreams is located at the address, which is a running Firehose instance.\ -`-e mainnet.eth.streamingfast.io:443` - -### Substreams YAML configuration file - -Inform Substreams where to find the `substreams.yaml` configuration file. - -{% hint style="info" %} -**Note**: The `substreams.yaml` configuration file argument in the command is optional if you are within the root folder of your Substreams and your manifest file is named `substreams.yaml. -{% endhint %} - -### Module - -The `map_transfers` module is defined in the manifest and it is the module run by Substreams. - -### Block mapping - -Start mapping at the specific block `12292922` by using passing the flag and block number `--start-block 12292922`. - -Cease block processing by using `--stop-block +1.` The `+1` option requests a single block. In the example, the next block is `12292923`. - -### Successful Substreams results - -Messages are printed to the terminal for successfully installed and configured Substreams setups. - -```bash - substreams run -e mainnet.eth.streamingfast.io:443 \ - substreams.yaml \ - map_transfers \ - --start-block 12292922 \ - --stop-block +1 -``` - -The `substreams` [`run`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#run) command outputs: - -```bash -2022-05-30T10:52:27.256-0400 INFO (substreams) connecting... -2022-05-30T10:52:27.389-0400 INFO (substreams) connected - ------------ IRREVERSIBLE BLOCK #12,292,922 (12292922) --------------- -map_transfers: log: NFT Contract bc4ca0eda7647a8ab7c2061c2e118a18a936f13d invoked -[...] -map_transfers: message "eth.erc721.v1.Transfers": { - "transfers": [ - { - "from": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", - "to": "q6cWGn+2nIjhbtn0Vc5it5HuTQM=", - "trxHash": "z7GX9i7Fx/DnGhHsDEoOOUo6pB21OG6FUm+GyEs/J5Y=", - "ordinal": "85" - }, - ... - ] -} -``` - -## Running a Substreams Package (.spkg) - -If you have an `spkg` file, you can run it by providing a path to the file: - -```bash -substreams gui \ - https://spkg.io/streamingfast/erc20-balance-changes-v1.1.0.spkg \ - map_balance_changes \ - -e mainnet.eth.streamingfast.io:443 \ - --start-block 1397553 -``` - -## Development and production mode - -Substreams has two mode when executing your module(s) either development mode or production mode. Development -and production modes impact the execution of Substreams, important aspects of execution include: - -* The time required to reach the first byte. -* The speed that large ranges get executed. -* The module logs and outputs sent back to the client. - -### Differences - -Differences between production and development modes include: - -* Forward parallel execution is enabled in production mode and disabled in development mode -* The time required to reach the first byte in development mode is faster than in production mode. - -Specific attributes of development mode include: - -* The client will receive all of the executed module's logs. -* It's possible to request specific store snapshots in the execution tree. -* Multiple module's output is possible. - -### Examples - -In most cases, you will run production mode, using a Substreams sink. Development mode is enabled by default in the CLI unless the `-p` flag is specified. - -Examples: (given the dependencies: `[block] --> [map_pools] --> [store_pools] --> [map_transfers])` - -* Running the `substreams run substreams.yaml map_transfers` command executes in development mode and only prints the `map_transfers` module's outputs and logs. -* Running the `substreams run substreams.yaml map_transfers --debug-modules-output=map_pools,map_transfers,store_pools` command executes in development mode and only prints the outputs of the `map_pools`, `map_transfers`, and `store_pools` modules. -* Running the `substreams run substreams.yaml map_transfers -s 1000 -t +5 --debug-modules-initial-snapshot=store_pools` command executes in development mode and prints all the entries in the `store_pools` module at block 999, then continues with outputs and logs from the `map_transfers` module in blocks 1000 through 1004. diff --git a/docs/new/consume/introduction.md b/docs/new/consume/introduction.md deleted file mode 100644 index 38497d233..000000000 --- a/docs/new/consume/introduction.md +++ /dev/null @@ -1,25 +0,0 @@ -Substreams is a technology that allows you to extract blockchain data in a fast and reliable way! - -You can retrieve blockchain data by using packages. A Substreams package is a file that contains which data you want to retrieve from the blockchain. For example, the Uniswap v3 package allows you to extract all the information related to the Uniswap v3 smart contract. Use the [Substreams Registry](https://substreams.dev) to explore new packages! - -If you can't find a package that fits your needs, you can always develop your own Substreams to extract custom data from the blockchain! - -The _Consume_ sections focus on consuming data from Substreams packages, instead of developing your own Substreams. -If you want to learn how to develop your own Substreams package, navigate to the Develop Substreams sections. - -## How to Consume a Package - -### Choose a Substreams Package -Among the wide range of packages available, choose whichever retrieves the data that you need! - -
- -### Choose a Consuming Service -What do you want to do with the data extracted from the blockchain? You have several options: send it to a PosgreSQL database, directly stream the data from your application (JS, Python, Rust...) or configure a webhook. - -
- -### Access the Data! -Depending on the consuming service selected, the way you access the data will vary. - -For example, with PosgreSQL, you create SQL queries to perform aggregations over the data; if you are streaming data with JavaScript, you can directly embed the data into your application! \ No newline at end of file diff --git a/docs/new/consume/sql/deployable-services/remote-service.md b/docs/new/consume/sql/deployable-services/remote-service.md deleted file mode 100644 index 4dde6475a..000000000 --- a/docs/new/consume/sql/deployable-services/remote-service.md +++ /dev/null @@ -1,86 +0,0 @@ -Using the StreamingFast SQL remote (hosted) service is the easiest way to get started with Substreams:SQL. The following tutorial teaches you how to deploy a Substreams package from the [Substreams Registry](https://substreams.dev) to StreamingFast remote service. - -{% hint style="success" %} -This tutorial shows you how to deploy Substreams package to the **StreamingFast remote service**. -You can also set up a Substreams deployable service environment in your computer, which will allow you to test your deployable services easily. - -The Substreams:SQL Tutorial -{% endhint %} - -## Tutorial - -In this short tutorial, you will: - -- Deploy the Substreams package to the StreamingFast SQL remote service. -- Explore the SQL database in your browser. - -Before you get started, make sure you have: -- The Substreams CLI installed. - -The package used in this tutorial is the USDT Ethereum package, which retrieves events from the USDT smart contract on the Ethereum blockchain. - -### Deploying a Substreams Package to the Remote Service - -1. You can find the USDT Ethereum package in the Substreams Registry. - -3. Deploy the package using the `substreams alpha service deploy` command. - -```bash -substreams alpha service deploy https://spkg.io/enoldev/substreams-ethereum-usdt-v0.1.0.spkg -e https://deploy.streamingfast.io -``` - -* The `substreams alpha service deploy` command is used to deploy a package to the remote service. -* In this example, you deploy the USDT Ethereum package (`https://spkg.io/enoldev/substreams-ethereum-usdt-v0.1.0.spkg`). -* The `-e` flag specifies the location of the remote service. In this example, you are using the StreamingFast Remote Service (`https://deploy.streamingfast.io`). - -The deployment of the package might take 1-2 minutes. - -4. After the deployment is completed, some useful data will be displayed: - -```bash -Deploying... (creating services, please wait) -Deployed substreams sink "60589e45": - Status: RUNNING -Deployment *60589e45* is *RUNNING* - - Database type is *postgres* - - Owner is *0qeru2bd28b954a35c12e* - -Indexed blocks: [4634748 - 4785000] - - Sink status is available at 'https://srv.streamingfast.io/60589e45/sinkinfo' - -Environment is *Development*, which means: - - Read-only direct access to the database is available at 'postgresql://db.srv.streamingfast.io:17441/substreams?sslmode=disable&user=dev-node-ro&password=JWgg68gP33lZ' - - Read/write direct access to the database is available at 'postgresql://db.srv.streamingfast.io:17441/substreams?sslmode=disable&user=dev-node&password=iESYNNa5EihR' - - Read/write access to the database via PGWeb at 'https://srv.streamingfast.io/60589e45/pgweb' - -Postgraphile is available at these URL: - - GraphiQL (browser): 'https://srv.streamingfast.io/60589e45/graphiql' - - GraphQL (apps): 'https://srv.streamingfast.io/60589e45/graphql' - -Documentation: https://substreams.streamingfast.io -Services: - - pgweb: pod running - - postgraphile: pod running - - postgres: pod running - - sink: pod running - - sinkinfo: pod running -``` - -1. **Service ID:** the identifier of the deployed service, which you can use to manage the service. -2. **URL of the service status:** use this URL to verify the status of the service. -3. **URL of the PostgreSQL client:** use this client to run SQL queries, update the SQL schema, and manage the SQL database in general. -4. **URL of the GraphQL clent:** use this client to run GraphQL queries. - -### Inspecting the PostgreSQL client - -Once the package is deployed, your Substreams starts indexing data. You can access this data through in the form of a PostgreSQL database or a GraphQL API. To access the PostgreSQL client (PGWeb), copy the URL provided. - -
- -### Stopping a Service - -If you have not included a stop block in the package manifest, your Substreams will keep indexing the head of the chain, thus consuming bytes from your StreamingFast plan. To stop a service, get the service ID and run the following command: - -```bash -substreams alpha service stop -e https://deploy.streamingfast.io -``` \ No newline at end of file diff --git a/docs/new/consume/sql/introduction.md b/docs/new/consume/sql/introduction.md deleted file mode 100644 index 2b9df0c1a..000000000 --- a/docs/new/consume/sql/introduction.md +++ /dev/null @@ -1,32 +0,0 @@ -The **Substreams:SQL service** allows you to consume the data extracted from the blockchain through a SQL database. - -
- -## Different Ways of Setting Up the SQL Consumption - -Substreams offers two different ways of consuming data as SQL: -- Using the Deployable Services (beta) -- Using the SQL sink, which currently supports PostgresSQL and Clickhouse (recommended) - -### Substreams:SQL Deployable Service -Use the Substreams CLI to easily send the data of your Substreams to a database. It also has support for **dbt transformations**, so it's great for data analyts! - -You can deploy a new service by using the `substreams alpha service` command. - -#### Hosted Service -The StreamingFast Hosted Service deploys a remote database in the StreamingFast servers, so you don't have to take of the infrastructure. - -#### Local Service -If you want to manage your own infrastructure, you can set up a services environment locally using Docker. - -### SQL sink (recommended) - -Previous to the implementation of the Deployable Services, the Postgres Sink was used to send data to a database. This is still available and there are a lot developers using it. - - - -## Module Requirements - -In order to the send the data to a SQL database, your Substreams must have a `db_out` module that emits [`DatabaseChanges`](https://docs.rs/substreams-database-change/latest/substreams_database_change/pb/database/struct.DatabaseChanges.html) objects. - -The `DatabaseChanges` object is something that the Postgres sink can understand, thus acting as a conversion layet between the data model of your Substreams and the table structure of the database. diff --git a/docs/new/consume/sql/sql-sink.md b/docs/new/consume/sql/sql-sink.md deleted file mode 100644 index 76d6214d6..000000000 --- a/docs/new/consume/sql/sql-sink.md +++ /dev/null @@ -1,232 +0,0 @@ ---- -description: StreamingFast Substreams SQL sink ---- - -# `substreams-sink-sql` introduction - -### Purpose - -Learn how to use the StreamingFast [`substreams-sink-sql`](https://github.com/streamingfast/substreams-sink-sql) tool with this documentation. A basic Substreams module example is provided to help you get started. We are going to showcase a Substreams module to extract data from the Ethereum blockchain and route it into a Protobuf for persistence in a SQL database. - -The `substreams-sink-sql` today supports two database drivers namely _PostgresSQL_ and _Clickhouse_. The tutorial below will focus on Postgres but we will describe how to connect to the other supported drivers. - -## Installation - -### 1. Install `substreams-sink-sql` - -Install `substreams-sink-sql` by using the pre-built binary release [available in the official GitHub repository](https://github.com/streamingfast/substreams-sink-sql/releases). - -Extract `substreams-sink-sql` into a folder and ensure this folder is referenced globally via your `PATH` environment variable. - -### 2. Set up accompanying code example - -Access the accompanying code example for this tutorial in the official `substreams-sink-sql` repository. You will find the Substreams project for the tutorial in the [docs/tutorial](https://github.com/streamingfast/substreams-sink-sql/tree/develop/docs/tutorial) directory. - -To create the required Protobuf files, run the included `make protogen` command. - -```bash -make protogen -``` - -To ensure proper setup and functionality, use your installation of the [`substreams` CLI](https://substreams.streamingfast.io/reference-and-specs/command-line-interface) to run the example code. - -Use the `make build` and `make stream_db` commands to verify the setup for the example project. Use the included `make` command to build the Substreams module. - -```bash -make build -make stream_db -``` - -### Module handler for sink - -The Rust source code file [`lib.rs`](https://github.com/streamingfast/substreams-sink-sql/blob/develop/docs/tutorial/src/lib.rs) contains an example code, the `db_out` module handler, which prepares and returns the module's [`DatabaseChanges`](https://docs.rs/substreams-database-change/latest/substreams_database_change/pb/database/struct.DatabaseChanges.html) output. The `substreams-sink-sql` tool captures the data sent out of the Substreams module and routes it into the appropriate columns and tables in the SQL database. - -```rust -#[substreams::handlers::map] -fn db_out(block_meta_start: store::Deltas>) -> Result { - let mut database_changes: DatabaseChanges = Default::default(); - transform_block_meta_to_database_changes(&mut database_changes, block_meta_start); - Ok(database_changes) -} -``` - -To gain a full understanding of the procedures and steps required for a database sink Substreams module, review the code in [`lib.rs`](https://github.com/streamingfast/substreams-sink-sql/blob/develop/docs/tutorial/src/lib.rs). The complete code includes the addition of a Substreams store module and other helper functions related to the database. - -**DatabaseChanges** - -The [`DatabaseChanges`](https://github.com/streamingfast/substreams-sink-database-changes/blob/develop/proto/sf/substreams/sink/database/v1/database.proto#L7) Protobuf definition can be viewed at the following link for a peek into the crates implementation. - -When developing your Substreams, the Rust crate [substreams-database-change](https://docs.rs/substreams-database-change/latest/substreams_database_change) can be used to create the required `DatabaseChanges` output type. - -**Note**: An output type of `proto:sf.substreams.sink.database.v1.DatabaseChanges` is required by the map module in the Substreams manifest when working with this sink. - -## 3. Install PostgreSQL - -To proceed with this tutorial, you must have a working PostgreSQL installation. Obtain the software by [downloading it from the vendor](https://www.postgresql.org/download/) and [install it by following the instructions](https://www.postgresql.org/docs/current/tutorial-install.html) for your operating system and platform. - -If you encounter any issues, [refer to the Troubleshooting Installation page](https://wiki.postgresql.org/wiki/Troubleshooting_Installation) on the official PostgreSQL Wiki for assistance. - -## 4. Create example database - -To store the blockchain data output by the Substreams module, you must create a new database in your PostgreSQL installation. The tutorial provides a schema and the PostgreSQL sink tool that handle the detailed aspects of the database design. - -Use the `psql` command in your terminal to launch PostgreSQL. - -Upon successful launch, you will see a prompt similar to the following, ready to accept commands for PostgreSQL. - -```bash -psql (15.1) -Type "help" for help. - -default-database-name=# -``` - -Use the following `SQL` command to create the example database: - -```bash -CREATE DATABASE "substreams_example"; -``` - -## 5. Create configuration file - -Once the database has been created, you must now define the Substreams Sink Config in a Substreams manifest creating a deployable unit. - -Let's create a folder `sink` and in it create a file called `substreams.dev.yaml` with the following content: - -```yaml -specVersion: v0.1.0 -package: - name: "" - version: - -imports: - sql: https://github.com/streamingfast/substreams-sink-sql/releases/download/protodefs-v1.0.1/substreams-sink-sql-protodefs-v1.0.1.spkg - main: ../substreams.yaml - -network: 'mainnet' - -sink: - module: main:db_out - type: sf.substreams.sink.sql.v1.Service - config: - schema: "../schema.sql" - wire_protocol_access: true -``` - -The `package.name` and `package.version` are meant to be replaced to fit your project. - -The `imports.main` defines your Substreams manifest that you want to sink. The `sink.module` defines which import key (`main` here) and which module's name (`db_out` here). - -The `network` field defines which network this deployment should be part of, in our case `mainnet` - -The `sink.type` defines the type of the config that we are expecting, in our case it's [sf.substreams.sink.sql.v1.Service](https://buf.build/streamingfast/substreams-sink-sql/docs/main:sf.substreams.sink.sql.v1#sf.substreams.sink.sql.v1.Service) (click on the link to see the message definition). - -The `sink.config` is the instantiation of this `sink.type` with the config fully filled. Some config are special because they load from a file or from a folder. For example in our case the `sink.config.schema` is defined with a Protobuf option `load_from_file` which means the content of the `../schema.sql` will actually be inlined in the Substreams manifest. - -> The final final can be found at [`sink/substreams.dev.yaml`](https://github.com/streamingfast/substreams-sink-sql/blob/develop/docs/tutorial/sink/substreams.dev.yaml) - -## 6. Run setup command - -Use the following command to run the `substreams-sink-sql` tool and set up the database for the tutorial. - -```bash -substreams-sink-sql setup "psql://dev-node:insecure-change-me-in-prod@127.0.0.1:5432/substreams_example?sslmode=disable" ./sink/substreams.dev.yaml -``` - -The `"psql://..."` is the DSN (Database Source Name) containing the connection details to your database packed as an URL. The `scheme` (`psql` here) part of the DSN's url defines which driver to use, `psql` is what we are going to use here, see [Drivers](#drivers) section below to see what other DSN you can use here. - -The DSN's URL defines the database IP address, username, and password, which depend on your PostgreSQL installation. Adjust `dev-node` to your own username `insecure-change-me-in-prod` to your password and `127.0.0.1:5432` to where your database can be reached. - -### Drivers - -For DSN configuration for the currently supported drivers, see the list below: - -- [Clickhouse DSN](https://github.com/streamingfast/substreams-sink-sql#clickhouse) -- [PostgresSQL DSN](https://github.com/streamingfast/substreams-sink-sql#postgresql) - -## 7. Sink data to database - -The `substreams-sink-sql` tool sinks data from the Substreams module to the SQL database. Use the tool's `run` command, followed by the endpoint to reach and your Substreams config file to use: - -```bash -substreams-sink-sql run "psql://dev-node:insecure-change-me-in-prod@127.0.0.1:5432/substreams_example?sslmode=disable" ./sink/substreams.dev.yaml -``` - -The endpoint needs to match the blockchain targeted in the Substreams module. The example Substreams module uses the Ethereum blockchain. - -Successful output from the `substreams-sink-sql` tool will resemble the following: - -```log -2023-01-18T12:32:19.107-0800 INFO (sink-sql) starting prometheus metrics server {"listen_addr": "localhost:9102"} -2023-01-18T12:32:19.107-0800 INFO (sink-sql) sink from psql {"dsn": "psql://dev-node:insecure-change-me-in-prod@127.0.0.1:5432/substreams_example?sslmode=disable", "endpoint": "mainnet.eth.streamingfast.io:443", "manifest_path": "substreams.yaml", "output_module_name": "db_out", "block_range": ""} -2023-01-18T12:32:19.107-0800 INFO (sink-sql) starting pprof server {"listen_addr": "localhost:6060"} -2023-01-18T12:32:19.127-0800 INFO (sink-sql) reading substreams manifest {"manifest_path": "sink/substreams.dev.yaml"} -2023-01-18T12:32:20.283-0800 INFO (pipeline) computed start block {"module_name": "store_block_meta_start", "start_block": 0} -2023-01-18T12:32:20.283-0800 INFO (pipeline) computed start block {"module_name": "db_out", "start_block": 0} -2023-01-18T12:32:20.283-0800 INFO (sink-sql) validating output store {"output_store": "db_out"} -2023-01-18T12:32:20.285-0800 INFO (sink-sql) resolved block range {"start_block": 0, "stop_block": 0} -2023-01-18T12:32:20.287-0800 INFO (sink-sql) ready, waiting for signal to quit -2023-01-18T12:32:20.287-0800 INFO (sink-sql) starting stats service {"runs_each": "2s"} -2023-01-18T12:32:20.288-0800 INFO (sink-sql) no block data buffer provided. since undo steps are possible, using default buffer size {"size": 12} -2023-01-18T12:32:20.288-0800 INFO (sink-sql) starting stats service {"runs_each": "2s"} -2023-01-18T12:32:20.730-0800 INFO (sink-sql) session init {"trace_id": "4605d4adbab0831c7505265a0366744c"} -2023-01-18T12:32:21.041-0800 INFO (sink-sql) flushing table rows {"table_name": "block_data", "row_count": 2} -2023-01-18T12:32:21.206-0800 INFO (sink-sql) flushing table rows {"table_name": "block_data", "row_count": 2} -2023-01-18T12:32:21.319-0800 INFO (sink-sql) flushing table rows {"table_name": "block_data", "row_count": 0} -2023-01-18T12:32:21.418-0800 INFO (sink-sql) flushing table rows {"table_name": "block_data", "row_count": 0} -``` - -{% hint style="info" %} -**Note**: If you have an error looking like `load psql table: retrieving table and schema: pq: SSL is not enabled on the server`, it's because SSL is not enabled to reach you database, add `?sslmode=disable` at the end of the `sink.config.dsn` value to connect without SSL. -{% endhint %} - -You can view the database structure by using the following command, after launching PostgreSQL through the `psql` command. - -```bash -=# \c substreams_example -``` - -The table information is displayed in the terminal resembling the following: - -```bash - List of relations - Schema | Name | Type | Owner ---------+------------+-------+---------- - public | block_data | table | postgres - public | cursors | table | postgres -(2 rows) -``` - -You can view the data extracted by Substreams and routed into the database table by using the following command: - -```bash -substreams_example=# SELECT * FROM "block_data"; -``` - -Output similar to the following is displayed in the terminal: - -```bash - id | version | at | number | hash | parent_hash | timestamp ---------------------+---------+---------------------+--------+------------------------------------------------------------------+------------------------------------------------------------------+---------------------- - day:first:19700101 | | 1970-01-01 00:00:00 | 0 | d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3 | 0000000000000000000000000000000000000000000000000000000000000000 | 1970-01-01T00:00:00Z - month:first:197001 | | 1970-01-01 00:00:00 | 0 | d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3 | 0000000000000000000000000000000000000000000000000000000000000000 | 1970-01-01T00:00:00Z - day:first:20150730 | | 2015-07-30 00:00:00 | 1 | 88e96d4537bea4d9c05d12549907b32561d3bf31f45aae734cdc119f13406cb6 | d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3 | 2015-07-30T15:26:28Z - month:first:201507 | | 2015-07-01 00:00:00 | 1 | 88e96d4537bea4d9c05d12549907b32561d3bf31f45aae734cdc119f13406cb6 | d4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3 | 2015-07-30T15:26:28Z - day:first:20150731 | | 2015-07-31 00:00:00 | 6912 | ab79f822909750f88dfb9dd0350c1ebe98d5495e9c969cdeb6e0ac993b80175b | 8ffd8c04cb89ef45e0e1163639d51d9ed4fa03dd169db90123a1e047361b46fe | 2015-07-31T00:00:01Z - day:first:20150801 | | 2015-08-01 00:00:00 | 13775 | 2dcecad4cf2079d18169ca05bc21e7ba0add7132b9382984760f43f2761bd822 | abaabb8f8b7f7fa07668fb38fd5a08da9814cd8ad18a793e54eef6fa9b794ab4 | 2015-08-01T00:00:03Z - month:first:201508 | | 2015-08-01 00:00:00 | 13775 | 2dcecad4cf2079d18169ca05bc21e7ba0add7132b9382984760f43f2761bd822 | abaabb8f8b7f7fa07668fb38fd5a08da9814cd8ad18a793e54eef6fa9b794ab4 | 2015-08-01T00:00:03Z -``` - -### Cursors - -When you use Substreams, it sends back a block to a consumer using an opaque cursor. This cursor points to the exact location within the blockchain where the block is. In case your connection terminates or the process restarts, upon re-connection, Substreams sends back the cursor of the last written bundle in the request so that the stream of data can be resumed exactly where it left off and data integrity is maintained. - -You will find that the cursor is saved in the `cursors` table of the `substreams_example` database. - -### Batching - -Insertion for historical blocks is performed in batched to increase ingestion speed. The `--flush-interval` flag can be used to change the default value of 1000 blocks. Also, the flag `--live-block-time-delta ` can be used to change the delta at which we start considering blocks to be live, the logic is `isLive = (now() - block.timestamp) < valueOfFlag(live-block-time-delta)`. - -## Conclusion and review - -Routing data extracted from the blockchain using Substreams is a powerful and useful feature. With Substreams, you can route data to various types of sinks, including files and databases such as PostgreSQL. For more information on other types of sinks and sinking strategies, consult the core Substreams sinks documentation at https://substreams.streamingfast.io/developers-guide/substreams-sinks. diff --git a/docs/new/consume/stream/javascript.md b/docs/new/consume/stream/javascript.md deleted file mode 100644 index 073471e9d..000000000 --- a/docs/new/consume/stream/javascript.md +++ /dev/null @@ -1,148 +0,0 @@ -## JavaScript - -The [Substreams JavaScript library](https://github.com/substreams-js/substreams-js) enables you to run a Substreams, just like you would through the CLI, but programatically. - -The library works both on the client-side and the server-side. - -### Installing the Library - -In your JavaScript project, use your preferred JavaScript package manager to install the required dependencies: - -1. The Substreams Core library: - -```bash -npm install @substreams/core -``` - -2. The Substreams Manifest library: - -```bash -npm install @substreams/manifest -``` - -3. The Protobuf library, which will be used to decode the Substreams response: - -```bash -npm install @bufbuild/connect-web -``` - -### Using the Library - -In order to use the library, you will need: - -- A Substreams endpoint. -- An authentication token (visit https://app.streamingfast.io to get one). -- A Substreams package (`spkg`). - -Consider that you want to consume the `map_block_meta` module of the [Ethereum Explorer package](https://github.com/streamingfast/substreams-explorers/tree/main/ethereum-explorer), which is hosted on Google Cloud (`https://storage.googleapis.com/substreams-registry/spkg/ethereum-explorer-v0.1.1.spkg`). - -1. First, let's define a few helper variables: - -```javascript -const TOKEN = "YOUR_TOKEN" // Your authentication token -const SPKG = "https://storage.googleapis.com/substreams-registry/spkg/ethereum-explorer-v0.1.1.spkg" // URL of the SPKG -const MODULE = "map_block_meta" // Name of the Substreams Module to run -``` - -2. Use the `fetchSubstream` method from the library to download the Substreams. Then, the `createRegistry` function creates the Protobuf definitions from the package: - -```javascript -const fetchPackage = async () => { - return await fetchSubstream(SPKG) -} - -const main = async () => { - // Fetch Substreams - const pkg = await fetchPackage() - // Create Protobuf registry - const registry = createRegistry(pkg); -} -``` - -3. Use the `createConnectTransport` to define the networking details of the connection (Substreams endpoint and authentication token): - -```javascript -const main = async () => { - const pkg = await fetchPackage() - const registry = createRegistry(pkg); - - const transport = createConnectTransport({ - // Substreams endpoint - baseUrl: "https://api.streamingfast.io", - // Authentication token - interceptors: [createAuthInterceptor(TOKEN)], - useBinaryFormat: true, - jsonOptions: { - // Protobuf Registry - typeRegistry: registry, - }, - }); -} -``` - -4. The `createRequest` function encapsulates the information of the execution (package, module, start block and stop block): - -```javascript -const main = async () => { - const pkg = await fetchPackage() - const registry = createRegistry(pkg); - - const transport = createConnectTransport({ - baseUrl: "https://api.streamingfast.io", - interceptors: [createAuthInterceptor(TOKEN)], - useBinaryFormat: true, - jsonOptions: { - typeRegistry: registry, - }, - }); - - // Execution details - const request = createRequest({ - substreamPackage: pkg, - outputModule: MODULE, - productionMode: true, - startBlockNum: 100000, - stopBlockNum: '+10', - }); -} -``` - -5. Finally, you can use the `streamBlocks` function to iterate over the stream of blocks returned by the Substreams endpoint: - -```javascript -const main = async () => { - const pkg = await fetchPackage() - const registry = createRegistry(pkg); - - const transport = createConnectTransport({ - baseUrl: "https://api.streamingfast.io", - interceptors: [createAuthInterceptor(TOKEN)], - useBinaryFormat: true, - jsonOptions: { - typeRegistry: registry, - }, - }); - - const request = createRequest({ - substreamPackage: pkg, - outputModule: MODULE, - productionMode: true, - startBlockNum: 100000, - stopBlockNum: '+10', - }); - - // Iterate over blocks - for await (const response of streamBlocks(transport, request)) { - const output = unpackMapOutput(response.response, registry); - - if (output !== undefined && !isEmptyMessage(output)) { - const outputAsJson = output.toJson({typeRegistry: registry}); - console.log(outputAsJson) - } - } -} -``` - -Now, you can send the data anywhere and create your own custom sink! If you have created a sink and you think it can be reused by other developers, [let us know on Discord](https://discord.gg/jZwqxJAvRs)! - -The previous code is availalble [on GitHub](https://gist.github.com/enoldev/b9f32e045f47675bd5c20f92246aed84). diff --git a/docs/new/consume/stream/quickstart.md b/docs/new/consume/stream/quickstart.md deleted file mode 100644 index e2ad17643..000000000 --- a/docs/new/consume/stream/quickstart.md +++ /dev/null @@ -1,4 +0,0 @@ - -The **Substreams:Stream service** allows you to stream blockchain data from your JavaScript, Python or Rust application. - -
\ No newline at end of file diff --git a/docs/new/consume/subgraph/quickstart.md b/docs/new/consume/subgraph/quickstart.md deleted file mode 100644 index 9b6a1555a..000000000 --- a/docs/new/consume/subgraph/quickstart.md +++ /dev/null @@ -1,5 +0,0 @@ -It is possible to the send data the of a Substreams to a subgraph, thus creating a Substreams-powered Subgraph. - -You can find information about Substreams-powered Subgraphs in [The Graph documentation](https://thegraph.com/docs/en/cookbook/substreams-powered-subgraphs/). - -
\ No newline at end of file diff --git a/docs/new/develop/architecture.md b/docs/new/develop/architecture.md deleted file mode 100644 index b970e9be0..000000000 --- a/docs/new/develop/architecture.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -description: Learn about the Substreams architecture ---- - -# Parallel execution - -Parallel execution is the process of a Substreams module's code executing multiple segments of blockchain data simultaneously in a forward or backward direction. Substreams modules can be executed in parallel, rapidly producing data for consumption in end-user applications. Parallel execution enables Substreams' highly efficient blockchain data processing capabilities. - -Parallel execution occurs when a requested module's start block is further back in the blockchain's history than the requested start block. For example, if a module starts at block 12,000,000 and a user requests data at block 15,000,000, parallel execution is used. This applies to both the development and production modes of Substreams operation. - -Parallel execution addresses the problem of the slow single linear execution of a module. Instead of running a module in a linear fashion, one block after the other without leveraging full computing power, N number of workers are executed over a different segment of the chain. It means data can be pushed back to the user N times faster than cases using a single worker. - -The server will define an execution schedule and take the module's dependencies into consideration. The server's execution schedule is a list of pairs of (`module, range`), where range contains `N` blocks. This is a configurable value set to 25K blocks, on the server. - -The single map_transfer module will fulfill a request from 0 - 75,000. The server's execution plan returns the results of `[(map_transfer, 0 -> 24,999), (map_transfer, 25,000 -> 49,999), (map_transfer, 50,000 -> 74,999)]`. - -The three pairs will be simultaneously executed by the server handling caching of the output of the store. For stores, an additional step will combine the store keys across multiple segments producing a unified and linear view of the store's state. - -Assuming a chain has 16,000,000 blocks, which translates to 640 segments of 25K blocks. The server currently has a limited amount of concurrency. In theory, 640 concurrent workers could be spawned. In practice, the number of concurrent workers depends on the capabilities of the service provider. For the production endpoint, StreamingFast sets the concurrency to 15 to ensure fair usage of resources for the free service. - -## Production versus development mode for parallel execution - -The amount of parallel execution for the two modes is illustrated in the diagram. Production mode results in more parallel processing than development mode for the requested range. In contrast, development mode consists of more linear processing. Another important note is, forward parallel execution only occurs in production mode. - -

Substreams production versus development mode for parallel execution diagram

- -## Backward and forward parallel execution steps - -The two steps involved during parallel execution are **backward execution and forward execution**. - -Backward parallel execution consists of executing in parallel block ranges, from the module's initial block, up to the start block of the request. If the start block of the request matches the module's initial block no backward execution is performed. - -Forward parallel execution consists of executing in parallel block ranges from the start block of the request up to the last known final block, also called an irreversible block, or the stop block of the request depending on which is smaller. Forward parallel execution significantly improves the performance of Substreams. - -Backward parallel execution will occur in both development and production modes. - -Forward parallel execution only occurs in production mode. diff --git a/docs/new/develop/chain-specific/evm/eth-calls.md b/docs/new/develop/chain-specific/evm/eth-calls.md deleted file mode 100644 index 33cd0087f..000000000 --- a/docs/new/develop/chain-specific/evm/eth-calls.md +++ /dev/null @@ -1,388 +0,0 @@ ---- -description: Learn how to perform Contract Calls (eth_calls) in EVM-compatible Substreams ---- - -# Contract Calls - -EVM-compatible smart contracts are queryable, which means that you can get real-time data from the contract's internal database. -In this tutorial, you will learn how to perform contract calls (`eth_calls`) through Substreams. - -Specifically, you will query the USDT smart contract (`0xdac17f958d2ee523a2206206994597c13d831ec7`) to get the number of decimals used by the token. -The USDT smart contract exposes a read function called `decimals`. - -## Pre-requisites - -- You have some knowledge about Substreams ([modules](https://substreams.streamingfast.io/concepts-and-fundamentals/modules) and [fundamentals](https://substreams.streamingfast.io/concepts-and-fundamentals/fundamentals)). -- You have the latest version of the [CLI](https://substreams.streamingfast.io/getting-started/installing-the-cli) installed. - -## Querying on EthScan - -You can query the `decimals` function by [visiting EthScan](https://etherscan.io/address/0xdac17f958d2ee523a2206206994597c13d831ec7#readContract). - - -

USDT contract on EthScan

- -## Initializing the Substreams project - -1. First, let's use `substreams init` to scaffold a new Substreams project that uses the USDT smart contract: - -```bash -substreams init -``` - -Complete the information required by the previous command, such as name of the project or smart contract to track. -In the `Contract address to track` step, write `0xdac17f958d2ee523a2206206994597c13d831ec7`, the address of the USDT smart contract. - -```bash -Project name (lowercase, numbers, undescores): usdttracker -Protocol: Ethereum -Ethereum chain: Mainnet -Contract address to track (leave empty to use "Bored Ape Yacht Club"): 0xdac17f958d2ee523a2206206994597c13d831ec7 -Would you like to track another contract? (Leave empty if not): -Retrieving Ethereum Mainnet contract information (ABI & creation block) -Fetched contract ABI for dac17f958d2ee523a2206206994597c13d831ec7 -Fetched initial block 4634748 for dac17f958d2ee523a2206206994597c13d831ec7 (lowest 4634748) -Generating ABI Event models for - Generating ABI Events for AddedBlackList (_user) - Generating ABI Events for Approval (owner,spender,value) - Generating ABI Events for Deprecate (newAddress) - Generating ABI Events for DestroyedBlackFunds (_blackListedUser,_balance) - Generating ABI Events for Issue (amount) - Generating ABI Events for Params (feeBasisPoints,maxFee) - Generating ABI Events for Redeem (amount) - Generating ABI Events for RemovedBlackList (_user) - Generating ABI Events for Transfer (from,to,value) -Writing project files -Generating Protobuf Rust code -``` - -2. Move to the project folder and build the Substreams. - -```bash -make build -``` - -3. Then, verify that the Substreams runs correctly. By default, it will output all the events of the smart contract. - -```bash -substreams run -e mainnet.eth.streamingfast.io:443 \ - substreams.yaml \ - map_events \ - --start-block 12292922 \ - --stop-block +1 -``` - -The previous command will output the following: - -```bash -Progress messages received: 0 (0/sec) -Backprocessing history up to requested target block 12292922: -(hit 'm' to switch mode) - ------------ BLOCK #12,292,922 (e2d521d11856591b77506a383033cf85e1d46f1669321859154ab38643244293) --------------- -{ - "@module": "map_events", - "@block": 12292922, - "@type": "contract.v1.Events", - "@data": { - "transfers": [ - { - "evtTxHash": "90e4fd16c989cdc7ecdfd0b6f458eb4be1c538901106bb794bb608f38ac9dd9f", - "evtIndex": 1, - "evtBlockTime": "2021-04-22T23:13:40Z", - "evtBlockNumber": "12292922", - "from": "odjZclYML4FEr4cdtQjwsLEKP78=", - "to": "XmM2sGcWQDHSwcLHo85fcWEdAcw=", - "value": "372200000" - } - ] - } -} - -all done -``` - -## Adding Calls to the Substreams - -The `substreams init` command generates Rust structures based on the ABI of the smart contract provided. All the calls are available under the `abi::contract::functions` namespace of the generated code. Let's take a look. - -1. Open the project in an editor of your choice (for example, VS Code) and navigate to the `lib.rs` file, which contains the main Substreams code. - -2. Create a new function, `get_decimals`, which returns a `BigInt` struct: - -```rust -fn get_decimals() -> substreams::scalar::BigInt { - -} -``` - -3. Import the `abi::contract::functions::Decimals` struct from the generated ABI code. - -```rust -fn get_decimals() -> substreams::scalar::BigInt { - let decimals = abi::contract::functions::Decimals {}; - -} -``` - -4. Next, use the `call` method to make the actual _eth_call_ by providing the smart contract address: - -```rust -fn get_decimals() -> substreams::scalar::BigInt { - let decimals = abi::contract::functions::Decimals {}; - let decimals_option = decimals.call(TRACKED_CONTRACT.to_vec()); - - decimals_option.unwrap() -} -``` - -In this case, the `call` method returns a `substreams::scalar::BigInt` struct containing the number of decimals used in the USDT token (`6`). - -5. You can include this function in the `map_events` module just for testing purposes: - -```rust -#[substreams::handlers::map] -fn map_events(blk: eth::Block) -> Result { - let evt_block_time = - (blk.timestamp().seconds as u64 * 1000) + (blk.timestamp().nanos as u64 / 1000000); - - // Using the decimals function - let decimals = get_decimals(); - substreams::log::info!("Number of decimals for the USDT token: {}", decimals.to_string()); - -...output omitted... -} -``` - -{% hint style="warning" %} -**Important:** Remember that this tutorial shows how to call the `decimals` function, but all the available calls are under the `abi::contract::functions` namespace, so you should be able to find them just by exploring the auto-generated ABI Rust files. -{% endhint %} - -6. To see it in action, just re-build and re-run the Substreams: - -```bash -make build -``` - -```bash -substreams run -e mainnet.eth.streamingfast.io:443 \ - substreams.yaml \ - map_events \ - --start-block 12292922 \ - --stop-block +1 -``` - -The output should be similar to the following: - -```bash -Connected (trace ID 6fb1a55ed17001d850d8c6655226ef6f) -Progress messages received: 0 (0/sec) -Backprocessing history up to requested target block 12292922: -(hit 'm' to switch mode) - ------------ BLOCK #12,292,922 (e2d521d11856591b77506a383033cf85e1d46f1669321859154ab38643244293) --------------- -map_events: log: Number of decimals for the USDT token: 6 -{ - "@module": "map_events", - "@block": 12292922, - "@type": "contract.v1.Events", - "@data": { - "transfers": [ - { - "evtTxHash": "90e4fd16c989cdc7ecdfd0b6f458eb4be1c538901106bb794bb608f38ac9dd9f", - "evtIndex": 1, - "evtBlockTime": "1619133220000", - "evtBlockNumber": "12292922", - "from": "odjZclYML4FEr4cdtQjwsLEKP78=", - "to": "XmM2sGcWQDHSwcLHo85fcWEdAcw=", - "value": "372200000" - } - ] - } -} - -all done -``` - -## Batching Calls - -RPC calls add latency to your Substreams, so you should avoid them as much as possible. However, if you still have to use `eth_calls`, you should batch them. Batching RPC calls meaning making several calls within the same request. - -In the previous USDT example, consider that you want to make three RPC calls: `Decimals`, `Name` and `Symbol`. Instead of creating a request for every call, you can use the `substreams_ethereum::rpc::RpcBatch` struct to make a single request for all the calls. - -1. In the `lib.rs` file, create a new function, `get_calls()` and initialize a batch struct. - -```rust -fn get_calls() { - let batch = substreams_ethereum::rpc::RpcBatch::new(); - -} -``` - -2. Add the calls that you want to retrieve by using the ABI of the smart contract: `abi::contract::functions::Decimals`, `abi::contract::functions::Name` and `abi::contract::functions::Symbol`. - -```rust -fn get_calls() { - let batch = substreams_ethereum::rpc::RpcBatch::new(); - - let responses = batch - .add( - abi::contract::functions::Decimals {}, - TRACKED_CONTRACT.to_vec(), - ) - .add( - abi::contract::functions::Name {}, - TRACKED_CONTRACT.to_vec(), - ) - .add( - abi::contract::functions::Symbol {}, - TRACKED_CONTRACT.to_vec(), - ) - .execute() - .unwrap() - .responses; -} -``` - -The `execute()` method make the actual RPC call and returns an array of responses. In this case, the array will have 3 responses, one for each call made. - -The order used for the response is the same as the order of addition to the request. In this example, `responses[0]` contains `Decimals`, `responses[1]` contains `Name`, and `response[2]` contains `Symbol`. - -3. Decode the `Decimals` response using the ABI. - -```rust -fn get_calls() { - let batch = substreams_ethereum::rpc::RpcBatch::new(); - - let responses = batch - .add( - abi::contract::functions::Decimals {}, - TRACKED_CONTRACT.to_vec(), - ) - .add( - abi::contract::functions::Name {}, - TRACKED_CONTRACT.to_vec(), - ) - .add( - abi::contract::functions::Symbol {}, - TRACKED_CONTRACT.to_vec(), - ) - .execute() - .unwrap() - .responses; - - let decimals: u64; - match substreams_ethereum::rpc::RpcBatch::decode::<_, abi::contract::functions::Decimals>(&responses[0]) { - Some(decoded_decimals) => { - decimals = decoded_decimals.to_u64(); - substreams::log::debug!("decoded_decimals ok: {}", decimals); - } - None => { - substreams::log::debug!("failed to get decimals"); - } - }; -} -``` - -4. Then, do the same for `Name` and `Symbol`. - -```rust -fn get_calls() { - let token_address = &TRACKED_CONTRACT.to_vec(); - let batch = substreams_ethereum::rpc::RpcBatch::new(); - let responses = batch - .add( - abi::contract::functions::Decimals {}, - TRACKED_CONTRACT.to_vec(), - ) - .add( - abi::contract::functions::Name {}, - TRACKED_CONTRACT.to_vec(), - ) - .add( - abi::contract::functions::Symbol {}, - TRACKED_CONTRACT.to_vec(), - ) - .execute() - .unwrap() - .responses; - - let decimals: u64; - match substreams_ethereum::rpc::RpcBatch::decode::<_, abi::contract::functions::Decimals>(&responses[0]) { - Some(decoded_decimals) => { - decimals = decoded_decimals.to_u64(); - substreams::log::debug!("decoded_decimals ok: {}", decimals); - } - None => { - substreams::log::debug!("failed to get decimals"); - } - }; - - let name: String; - match substreams_ethereum::rpc::RpcBatch::decode::<_, abi::contract::functions::Name>(&responses[1]) { - Some(decoded_name) => { - name = decoded_name; - substreams::log::debug!("decoded_name ok: {}", name); - } - None => { - substreams::log::debug!("failed to get name"); - } - }; - - let symbol: String; - match substreams_ethereum::rpc::RpcBatch::decode::<_, abi::contract::functions::Symbol>(&responses[2]) { - Some(decoded_symbol) => { - symbol = decoded_symbol; - substreams::log::debug!("decoded_symbol ok: {}", symbol); - } - None => { - substreams::log::debug!("failed to get symbol"); - } - }; -} -``` - -5. Build and run the Substreams. - -```bash -make build -``` - -```bash -substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_events --start-block 12292922 --stop-block +1 -``` - -You should see an output similar to the following: - -```bash -Connected (trace ID 0f3e3f3868d4f8028b8fd4d6eab7d0b4) -Progress messages received: 0 (0/sec) -Backprocessing history up to requested target block 12292922: -(hit 'm' to switch mode) - - ------------ BLOCK #12,292,922 (e2d521d11856591b77506a383033cf85e1d46f1669321859154ab38643244293) --------------- -map_events: log: decoded_decimals ok: 6 -map_events: log: decoded_name ok: Tether USD -map_events: log: decoded_symbol ok: USDT -{ - "@module": "map_events", - "@block": 12292922, - "@type": "contract.v1.Events", - "@data": { - "transfers": [ - { - "evtTxHash": "90e4fd16c989cdc7ecdfd0b6f458eb4be1c538901106bb794bb608f38ac9dd9f", - "evtIndex": 1, - "evtBlockTime": "2021-04-22T23:13:40Z", - "evtBlockNumber": "12292922", - "from": "odjZclYML4FEr4cdtQjwsLEKP78=", - "to": "XmM2sGcWQDHSwcLHo85fcWEdAcw=", - "value": "372200000" - } - ] - } -} - -all done -``` \ No newline at end of file diff --git a/docs/new/develop/chain-specific/evm/overview.md b/docs/new/develop/chain-specific/evm/overview.md deleted file mode 100644 index c3d387c5d..000000000 --- a/docs/new/develop/chain-specific/evm/overview.md +++ /dev/null @@ -1 +0,0 @@ -## EVM Chain Extensions \ No newline at end of file diff --git a/docs/new/develop/chain-specific/overview.md b/docs/new/develop/chain-specific/overview.md deleted file mode 100644 index 90fab7e3f..000000000 --- a/docs/new/develop/chain-specific/overview.md +++ /dev/null @@ -1 +0,0 @@ -## Chain-Specific Extensions \ No newline at end of file diff --git a/docs/new/develop/creating-protobuf-schemas.md b/docs/new/develop/creating-protobuf-schemas.md deleted file mode 100644 index 1413d7f8c..000000000 --- a/docs/new/develop/creating-protobuf-schemas.md +++ /dev/null @@ -1,147 +0,0 @@ ---- -description: StreamingFast Substreams protobuf schemas ---- - -# Protobuf schemas - -## Protobuf overview - -Substreams uses Google Protocol Buffers extensively. Protocol Buffers, also referred to as protobufs, are used as the API for data models specific to the different blockchains. Manifests contain references to the protobufs for your Substreams module. - -{% hint style="success" %} -**Tip**: Protobufs define the input and output for modules. -{% endhint %} - -Learn more about the details of Google Protocol Buffers in the official documentation provided by Google. - -**Google Protocol Buffer Documentation** - -[Learn more about Google Protocol Buffers](https://protobuf.dev/) in the official documentation provided by Google. - -**Google Protocol Buffer Tutorial** - -[Explore examples and additional learning material](https://protobuf.dev/programming-guides/proto3/) for Google Protocol Buffers provided by Google. - -### Protobuf definition for Substreams - -Define a protobuf model as [`proto:eth.erc721.v1.Transfers`](https://github.com/streamingfast/substreams-template/blob/develop/proto/erc721.proto) representing a list of ERC721 transfers. - -{% hint style="info" %} -**Note**: The `Transfers` protobuf in the Substreams Template example is located in the proto directory. -{% endhint %} - -{% code title="eth/erc721/v1/erc721.proto" lineNumbers="true" %} - -```protobuf -syntax = "proto3"; - -package eth.erc721.v1; - -message Transfers { - repeated Transfer transfers = 1; -} - -message Transfer { - bytes from = 1; - bytes to = 2; - uint64 token_id = 3; - bytes trx_hash = 4; - uint64 ordinal = 5; -} -``` - -{% endcode %} - -[View the `erc721.proto`](https://github.com/streamingfast/substreams-template/blob/develop/proto/erc721.proto) file in the official Substreams Template example repository. - -#### Identifying data types - -The ERC721 smart contract used in the Substreams Template example contains a `Transfer` event. You can use the event data through a custom protobuf. - -The protobuf file serves as the interface between the module handlers and the data being provided by Substreams. - -{% hint style="success" %} -**Tip**: Protobufs are platform-independent and are defined and used for various blockchains. - -- The ERC721 smart contracts used in the Substreams Template example are generic contracts used across many different Ethereum applications. -- The size and scope of the Substreams module dictates the number of and complexity of protobufs. - {% endhint %} - -The Substreams Template example extracts `Transfer` events from the [Bored Ape Yacht Club smart contract](https://etherscan.io/address/0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d) which is located on the Ethereum blockchain. - -Several specific data types exist in the Ethereum smart contract ecosystem, some extending the ERC20 and ERC721 base modules. Complex protobufs are created and refined based on the various data types used across the different blockchains. - -{% hint style="success" %} -**Tip**_:_ The use of fully qualified protobuf file paths reduces the risk of naming conflicts when other community members build their [Substreams packages](../reference-and-specs/packages.md#dependencies). -{% endhint %} - -### Generating protobufs - -The [`substreams` CLI](../reference-and-specs/command-line-interface.md) is used to generate the associated Rust code for the protobuf. - -Notice the `protogen` command and Substreams manifest passed into the [`substreams` CLI](../reference-and-specs/command-line-interface.md). - -{% code overflow="wrap" %} - -```bash -substreams protogen ./substreams.yaml --exclude-paths="sf/ethereum,sf/substreams,google" -``` - -{% endcode %} - -The pairing code is generated and saved into the [`src/pb/eth.erc721.v1.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/pb/eth.erc721.v1.rs)Rust file. - -The [`mod.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/pb/mod.rs) file located in the `src/pb` directory of the Substreams Template example is responsible for exporting the freshly generated Rust code. - -{% code title="src/pb/mod.rs" overflow="wrap" lineNumbers="true" %} - -```rust -#[path = "eth.erc721.v1.rs"] -#[allow(dead_code)] -pub mod erc721; -``` - -{% endcode %} - -View the [`mod.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/pb/mod.rs) file in the repository. - -### Protobuf and Rust optional fields - -Protocol buffers define fields' type by using standard primitive data types, such as integers, booleans, and floats or a complex data type such as `message`, `enum`, `oneof` or `map`. View the [full list](https://developers.google.com/protocol-buffers/docs/proto#scalar) of types in the [Google Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/overview). - -Any primitive data types in a message generate the corresponding Rust type,[`String`](https://doc.rust-lang.org/std/string/struct.String.html) for `string`, `u64` for `uint64,` and assign the default value of the corresponding Rust type if the field is not present in a message, an empty string for [`String`](https://doc.rust-lang.org/std/string/struct.String.html), 0 for integer types, `false` for `bool`. - -Rust generates the corresponding `message` type wrapped by an [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) enum type for fields referencing other complex `messages`. The [`None`](https://doc.rust-lang.org/std/option/) variant is used if the field is not present in the message. - -The [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) [`enum`](https://doc.rust-lang.org/book/ch06-01-defining-an-enum.html) is used to represent the presence through [`Some(x)`](https://doc.rust-lang.org/std/option/) or absence through [`None`](https://doc.rust-lang.org/std/option/) of a value in Rust. [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) allows developers to distinguish between a field containing a value versus a field without an assigned a value. - -{% hint style="info" %} -**Note**: The standard approach to represent nullable data in Rust is to wrap optional values in [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html). -{% endhint %} - -The Rust [`match`](https://doc.rust-lang.org/rust-by-example/flow_control/match.html) keyword is used to compare the value of an [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) to a [`Some`](https://doc.rust-lang.org/std/option/) or [`None`](https://doc.rust-lang.org/std/option/) variant. Handle a type wrapped [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) in Rust by using: - -```rust -match person.Location { - Some(location) => { /* Value is present, do something */ } - None => { /* Value is absent, do something */ } -} -``` - -If you are only interested in finding the presence of a value, use the [`if let`](https://doc.rust-lang.org/rust-by-example/flow_control/if_let.html) statement to handle the [`Some(x)`](https://doc.rust-lang.org/std/option/) arm of the [`match`](https://doc.rust-lang.org/rust-by-example/flow_control/match.html) code. - -```rust -if let Some(location) = person.location { - // Value is present, do something -} -``` - -If a value is present, use the [`.unwrap()`](https://doc.rust-lang.org/rust-by-example/error/option_unwrap.html) call on the [`Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) to obtain the wrapped data. You'll need to account for these types of scenarios if you control the creation of the messages yourself or if the field is documented as always being present. - -{% hint style="info" %} -**Note**: You need to be **absolutely sure** **the field is always defined**, otherwise Substreams panics and never completes, getting stuck on a block indefinitely. -{% endhint %} - -_**PROST!**_ is a tool for generating Rust code from protobuf definitions. [Learn more about `prost`](https://github.com/tokio-rs/prost) in the project's official GitHub repository. - -[Learn more about `Option`](https://doc.rust-lang.org/rust-by-example/std/option.html) in the official Rust documentation. diff --git a/docs/new/develop/init-project.md b/docs/new/develop/init-project.md deleted file mode 100644 index 476504fd4..000000000 --- a/docs/new/develop/init-project.md +++ /dev/null @@ -1,49 +0,0 @@ - -Getting started with Substreams is very easy! If you have the Substreams CLI installed in your computer, you can use the `substreams init` command to initialize a Substreams project from a smart contract. - -Given a smart contract address, the `substreams init` command will download the contract's ABI, inspect it and create a Substreams that extracts all the events. - -{% embed url="https://www.youtube.com/watch?v=vWYuOczDiAA&" %} -Initialize a Substreams project -{% endembed %} - -## Initialize the Project - -1. Run `substreams init`: - -```bash -substreams init -``` - -2. Provide a name for the project: - -```bash -✗ Project name (lowercase, numbers, undescores): -``` - -3. Select what protocol you want to use (e.g. Ethereum): - -```bash -? Select protocol: - ▸ Ethereum - Other -``` - -4. Select what chain to use (Ethereum Mainnet, BNB, Polygon...): - -```bash -? Select Ethereum chain: - ▸ Mainnet - BNB - Polygon - Goerli -↓ Mumbai -``` - -5. Input the smart contrat address. If you do not provider an addressm the "Bored Ape Yacht Club" smart contract will be used: - -```bash -✔ Contract address to track (leave empty to use "Bored Ape Yacht Club"): -``` - -After providing all the previous information, a new Substreams project will be generated. You can now start coding! \ No newline at end of file diff --git a/docs/new/develop/introduction.md b/docs/new/develop/introduction.md deleted file mode 100644 index 779e24a7a..000000000 --- a/docs/new/develop/introduction.md +++ /dev/null @@ -1,34 +0,0 @@ - -Substreams is a technology that allows you to extract blockchain data in a fast and reliable way! - -## How Does It Work? - -1. **The Substreams Provider** - -
- -First of all, there is a Substreams provider. The Substreams provider takes care of storing blockchain data and making it available in a fast and reliable way - -2. **Apply Your Transformations** - -
- -In order for the Substreams provider to know which specific data you want to retrieve, you write a Rust program that defines the transformations that you want to apply to the blockchain data. - -3. **The Substreams Package** - -
- -Then, you pack your Rust program into a Substreams package. - -4. **Execution of the Package** - -
- -Lastly, you send the Substreams package to the Substreams provider for execution. The Substreams provider will start streaming data back to you ultra-fast! - -5. **Consume the Data** - -Use one of the Substreams services to consume the extracted data (SQL, Webhooks, Stream, Subgraphs...) - -
\ No newline at end of file diff --git a/docs/new/develop/modules/README.md b/docs/new/develop/modules/README.md deleted file mode 100644 index 66c6bb8e7..000000000 --- a/docs/new/develop/modules/README.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -description: StreamingFast Substreams modules overview ---- - -# Modules - -## Overview - -Modules are an important part of Substreams, offering hooks into the execution of the Substreams compute engine. You can create Substreams data manipulation and transformation strategies within modules. - -Modules are small pieces of Rust code running in a [WebAssembly (WASM)](https://webassembly.org/) virtual machine. They coexist within the stream of blocks sent by the Substreams compute engine, which arrives from a blockchain node. - -Modules have one or more inputs, which can be in the form of a `map` or `store`, or a `Block` or `Clock` object received from the blockchain's data source. - -{% embed url="https://mermaid.ink/svg/pako:eNp1kM0KwjAQhF8l7NkWvEbwIPUJ9NYUWZKtLTZJ2WwEEd_dCAr-4GFhd_h2GOYKNjoCDUfGeVD7ZmWCUqmvSQZiyr6Wy0z1eVlvpmhPbYqZLen_RKeqaq2EMaSe-OBxfhi-320Z_aF8_diYgxC3SSKT_tE7WIAn9ji6kvv6sDdQsngyoMvqqMc8iQETbgXNs0OhrRuLG-gep0QLwCxxdwkWtHCmF9SMWGrwT-p2B02rZZY" %} -Substreams modules data interaction diagram -{% endembed %} - -The diagram shows how the `transfer_map` module extracts the transfers in a `Block` and tracks the total number of transfers. - -{% hint style="info" %} -**Note:** You can use multiple inputs in blockchains because they are clocked, which allows for synchronization between multiple execution streams and improved performance compared to conventional streaming engines. -{% endhint %} - -As seen in the `counters` `store` example diagram, modules can also take in multiple inputs. In this case, two modules feed into a `store`, effectively tracking multiple `counters`. - -{% embed url="https://mermaid.ink/svg/pako:eNqdkE1qAzEMha9itE4GsnWgi5KcINmNh6LamozJeGxsuSGE3L1KW1PIptCdnnjv088NbHQEGk4Z06SOu61ZlHqfoz33JdZsSasydsQTZaqh42ui7mPTvT4cg1qvX1TA9HbxPLmMF5zLv_KOUiyev8JPvF60fm5-J22sC1MufeGYZVDTQ8M07C-jdf4AwAoC5YDeyWtuD5wBOSGQAS2loxHrzAbMchdrTQ6Z9s4LBfQo-9EKsHI8XBcLmnOlZtp5lE-HH9f9EylZic0" %} -Multiple module inputs diagram -{% endembed %} - -Every time a new `Block` is processed, all of the modules are executed as a directed acyclic graph (DAG). - -{% hint style="info" %} -**Note:** The protocol's Block protobuf model always serves as the top-level data source and executes deterministically. -{% endhint %} - -### Single output - -Modules have a single typed output, which is typed to inform consumers of the types of data to expect and how to interpret the bytes being sent. - -{% hint style="success" %} -**Tip**: In subsequent modules, input from one module's data output is used to form a chain of data flow from module to module. -{% endhint %} - -### `map` versus `store` modules - -To develop most non-trivial Substreams, you will need to use multiple `map` and `store` modules. The specific number, responsibilities, and communication methods for these modules will depend on the developer's specific goals for the Substreams development effort. - -The two module types are commonly used together to construct the directed acyclic graph (DAG) outlined in the Substreams manifest. The two module types are very different in their use and how they work. Understanding these differences is vital for harnessing the full power of Substreams. - -### `map` modules - -`map` modules are used for data extraction, filtering, and transformation. They should be used when direct extraction is needed avoiding the need to reuse them later in the DAG. - -To optimize performance, you should use a single `map` module instead of multiple `map` modules to extract single events or functions. It is more efficient to perform the maximum amount of extraction in a single top-level `map` module and then pass the data to other Substreams modules for consumption. This is the recommended, simplest approach for both backend and consumer development experiences. - -Functional `map` modules have several important use cases and facts to consider, including: - -* Extracting model data from an event or function's inputs. -* Reading data from a block and transforming it into a custom protobuf structure. -* Filtering out events or functions for any given number of contracts. - -### `store` modules - -`store` modules are used for the aggregation of values and to persist state that temporarily exists across a block. - -{% hint style="warning" %} -**Important:** Stores should not be used for temporary, free-form data persistence. -{% endhint %} - -Unbounded `store` modules are discouraged. `store` modules shouldn't be used as an infinite bucket to dump data into. - -Notable facts and use cases for working `store` modules include: - -* `store` modules should only be used when reading data from another downstream Substreams module. -* `store` modules cannot be output as a stream, except in development mode. -* `store` modules are used to implement the Dynamic Data Sources pattern from Subgraphs, keeping track of contracts created to filter the next block with that information. -* Downstream of the Substreams output, do not use `store` modules to query anything from them. Instead, use a sink to shape the data for proper querying. - - diff --git a/docs/new/develop/modules/aggregation-windows.md b/docs/new/develop/modules/aggregation-windows.md deleted file mode 100644 index e86f36a4b..000000000 --- a/docs/new/develop/modules/aggregation-windows.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -description: Building and freeing up aggregation windows ---- - -# Building and freeing up aggregation windows - -Store module key-value storage can hold at most 1 GiB. It is usually enough if used correctly, but it is still a good idea (and sometimes even necessary) to free up unused keys. It is especially true for cases where you work with aggregation windows. - -Consider this store module that aggregates hourly trade counter for each token: -```rust -#[substreams::handlers::store] -pub fn store_total_tx_counts(clock: Clock, events: Events, output: StoreAddBigInt) { - let timestamp_seconds = clock.timestamp.unwrap().seconds; - let hour_id = timestamp_seconds / 3600; - let prev_hour_id = hour_id - 1; - - output.delete_prefix(0, &format!("TokenHourData:{prev_hour_id}:")); - - for event in events.pool_events { - output.add_many( - event.log_ordinal, - &vec![ - format!("TokenHourData:{}:{}", hour_id, event.token0), - format!("TokenHourData:{}:{}", hour_id, event.token1), - ], - &BigInt::from(1 as i32), - ); - } -} -``` - -Let's break it down. - -First, we use `Clock` input source to get the current and previous hour id for the block. - -```rust -let hour_id = timestamp_seconds / 3600; -let prev_hour_id = hour_id - 1; -``` - -Then we build hourly keys for our counters and use `add_many` method to increment them. These counters will be consumed downstream by other modules. - -```rust -output.add_many( - event.log_ordinal, - &vec![ - format!("TokenHourData:{}:{}", hour_id, event.token0), - format!("TokenHourData:{}:{}", hour_id, event.token1), - ], - &BigInt::from(1 as i32), -); -``` - -Here's the trick. Since we don't need these counters outside of the hourly window, we can safely delete these key-value pairs for the previous hourly window and free up the memory. - -This is done using `delete_prefix` method: -```rust -output.delete_prefix(0, &format!("TokenHourData:{prev_hour_id}:")); -``` - -## Links -* [Uniswap-v3 Subgraph and Substreams](https://github.com/streamingfast/substreams-uniswap-v3) \ No newline at end of file diff --git a/docs/new/develop/modules/dynamic-data-sources.md b/docs/new/develop/modules/dynamic-data-sources.md deleted file mode 100644 index bd3a7fe58..000000000 --- a/docs/new/develop/modules/dynamic-data-sources.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -description: Dynamic data sources and Substreams ---- - -# Dynamic data sources and Substreams - -Using Factory contract is a quite common pattern used by dApps when the main smart contract deploys and manages multiple identical associated contracts, i.e. one smart contract for each Uniswap or Curve swap pool. - -When developing traditional subgraphs, you could use [data source templates](https://thegraph.com/docs/en/developing/creating-a-subgraph/#data-source-templates) approach to keep track of such dynamically deployed smart contracts. - -Here's how you can achieve that with Substreams. - -We'll be using Uniswap V3 example where the Factory creates and deploys its smart contract for each pool. - -You start with a simple map module that emits all pool creation events: -```yaml -- name: map_pools_created - kind: map - inputs: - - source: sf.ethereum.type.v2.Block - output: - type: proto:uniswap.types.v1.Pools -``` - -```rust -#[substreams::handlers::map] -pub fn map_pools_created(block: Block) -> Result { - Ok(Pools { - pools: block - .events::(&[&UNISWAP_V3_FACTORY]) - .filter_map(|(event, log)| { - // skipped: extracting pool information from the transaction - Some(Pool { - address, - token0, - token1, - ..Default::default() - }) - }) - .collect(), - }) -} -``` - -We can now take that map module output and direct these pool creation events into a Substreams key-value store using a store module: -```yaml - - name: store_pools_created - kind: store - updatePolicy: set - valueType: proto:uniswap.types.v1.Pool - inputs: - - map: map_pools_created -``` -```rust -#[substreams::handlers::store] -pub fn store_pools_created(pools: Pools, store: StoreSetProto) { - for pool in pools.pools { - let pool_address = &pool.address; - store.set(pool.log_ordinal, format!("pool:{pool_address}"), &pool); - } -} -``` - -Above we are using `pool:{pool_address}` as a key to store the pool information. Eventually, our store will contain all Uniswap pools. -Now, in the downstream modules, we can easily retrieve our pool from the store whenever we need it. - -```yaml -- name: map_events - kind: map - inputs: - - source: sf.ethereum.type.v2.Block - - store: store_pools_created - output: - type: proto:uniswap.types.v1.Events -``` - -```rust -#[substreams::handlers::map] -pub fn map_events(block: Block, pools_store: StoreGetProto) -> Result { - let mut events = Events::default(); - - for trx in block.transactions() { - for (log, call_view) in trx.logs_with_calls() { - let pool_address = &Hex(&log.address).to_string(); - - let pool = match pools_store.get_last(format!("pool:{pool_address}")) { - Some(pool) => pool, - None => { continue; } - }; - - // use the pool information from the store - } - } - - Ok(events) -} -``` - -Here we use `pools_store.get_last()` method to get the pool from the store by its smart contract address. Once we have it, we can use that information to analyze the swap transaction and emit the events. - -Alternatively, we could make RPC calls to get the pool details from an RPC node but that would be extremely inefficient considering that we would need to make RPC calls for millions of such events. Using a store will be much faster. - -For a real-life application of this pattern see [Uniswap V3 Substreams](https://github.com/streamingfast/substreams-uniswap-v3) - - -## Links -* [Uniswap-v3 Subgraph and Substreams](https://github.com/streamingfast/substreams-uniswap-v3) -* [Substreams Sink Entity Changes](https://github.com/streamingfast/substreams-sink-entity-changes) \ No newline at end of file diff --git a/docs/new/develop/modules/inputs.md b/docs/new/develop/modules/inputs.md deleted file mode 100644 index 3849f47a6..000000000 --- a/docs/new/develop/modules/inputs.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -description: StreamingFast Substreams module inputs ---- - -# Inputs - -## `inputs` overview - -Modules receive `inputs` of three types: - -* `source` -* `map` -* `store` -* `params` - -## Input type `source` - -An `inputs` of type `source` represents a chain-specific, Firehose-provisioned protobuf object. Learn more about the supported protocols and their corresponding message types in the [Chains and inputs documentation](../../reference-and-specs/chains-and-endpoints.md). - -{% hint style="info" %} -**Note**: The different blockchains reference different `Block` objects. For example, Solana references its `Block` object as `sf.solana.type.v1.Block`. Ethereum-based Substreams modules specify `sf.ethereum.type.v2.Block.` -{% endhint %} - -The `source` `inputs` type \_\_ is defined in the Substreams manifest. It is important to specify the correct `Block` object for the chain. - -```yaml -modules: -- name: my_mod - inputs: - - source: sf.ethereum.type.v2.Block -``` - -#### `Clock` object - -The `sf.substreams.v1.Clock` object is another source type available on any of the supported chains. - -The `sf.substreams.v1.Clock` represents: - -* `Block` `number` -* `Block` `ID` -* `Block` `timestamp` - -## Input type `params` - -An `inputs` of type `params` represents a parameterizable module input. Those parameters can be specified either: - -* in the `params` section of the manifest, -* on the command-line (using `substreams run -p` for instance), -* by tweaking the protobuf objects directly when consuming from your favorite language - -See the [Manifest's `params` manifest section of the Reference & specs](../../reference-and-specs/manifests.md#params) for more details. - -## Input type `map` - -An input of type `map` represents the output of another `map` module. It defines a parent-child relationship between modules. - -The object's type is defined in the [`output.type`](../../reference-and-specs/manifests.md#modules-.output) attribute of the `map` module. - -{% hint style="warning" %} -**Important**: The graph built by input dependencies is a Directed Acyclic Graph, which means there can be no circular dependencies. -{% endhint %} - -Define the `map` input type in the manifest and choose a name for the `map` reflecting the logic contained within it. - -{% code title="manifest excerpt" %} -```yaml - inputs: - - map: my_map -``` -{% endcode %} - -Learn more about `maps` in the [Modules](./) section. - -## Input type `store` - -A `store inputs` type represents the state of another `store` used by the Substreams module being created. - -The developer defines the `store` `inputs` type in the Substreams manifest and gives the `store` a descriptive name that reflects the logic contained within it, similar to a `map`. - -Store modules are set to `get` mode by default: - -{% code title="manifest excerpt" %} -```yaml - inputs: - - store: my_store # defaults to mode: get -``` -{% endcode %} - -Alternatively, set `stores` to `deltas` mode by using: - -{% code title="manifest excerpt" %} -```yaml - inputs: - - store: my_delta_store - mode: deltas -``` -{% endcode %} - -### Store access `mode` - -Substreams uses two types of `mode` for modules: - -* `get` -* `delta` - -### Store constraints - -* A `store` can only receive `inputs` as read-only. -* A `store` cannot depend on itself. - -### `get` mode - -`get` mode provides a key-value store readily queryable and guaranteed to be in sync with the block being processed. - -{% hint style="success" %} -**Tip**: `get` `mode` is the default mode for modules. -{% endhint %} - -### `delta` mode - -`delta` `mode` modules are [protobuf objects](../../../pb/sf/substreams/v1/substreams.proto#L124) containing all the changes occurring in the `store` module available in the same block. - -`delta` mode enables you to loop through keys and decode values mutated in the module. - -#### `store` `deltas` - -The protobuf model for `StoreDeltas` is defined by using: - -{% code overflow="wrap" %} -```protobuf -message StoreDeltas { - repeated StoreDelta deltas = 1; -} - -message StoreDelta { - enum Operation { - UNSET = 0; - CREATE = 1; - UPDATE = 2; - DELETE = 3; - } - Operation operation = 1; - uint64 ordinal = 2; - string key = 3; - bytes old_value = 4; - bytes new_value = 5; -} -``` -{% endcode %} diff --git a/docs/new/develop/modules/keys-in-stores.md b/docs/new/develop/modules/keys-in-stores.md deleted file mode 100644 index a75b98608..000000000 --- a/docs/new/develop/modules/keys-in-stores.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -description: Using keys in stores ---- - -# Keys in stores - -We use store modules to aggregate the data in the underlying key-value storage. It is important to have a system for organizing your keys to be able to efficiently retrieve, filter and free them when needed. - -In most cases, you will encode data into your keys into segmented parts, adding a prefix as namespace for example `user` and `
` joined together using a separator. Segments in a key are conventionally joined with `:` as a separator. - -Here are some examples, -- `Pool:{pool_address}:volumeUSD` - `{pool_address}` pool total traded USD volume -- `Token:{token_addr}:volume` - total `{token_addr}` token volume traded -- `UniswapDayData:{day_id}:volumeUSD` - `{day_id}` daily USD trade volume -- `PoolDayData:{day_id}:{pool_address}:{token_addr}:volumeToken1` - total `{day_id}` daily volume of `{token_addr}` token that went through a `{pool_address}` pool in token1 equivalent - -In the example of a counter store below, we increment transaction counters for different metrics that we could use in the downstream modules: -```rust -#[substreams::handlers::store] -pub fn store_total_tx_counts(clock: Clock, events: Events, output: StoreAddBigInt) { - let timestamp_seconds = clock.timestamp.unwrap().seconds; - let day_id = timestamp_seconds / 86400; - let hour_id = timestamp_seconds / 3600; - let prev_day_id = day_id - 1; - let prev_hour_id = hour_id - 1; - - for event in events.pool_events { - let pool_address = &event.pool_address; - let token0_addr = &event.token0; - let token1_addr = &event.token1; - - output.add_many( - event.log_ordinal, - &vec![ - format!("pool:{pool_address}"), - format!("token:{token0_addr}"), - format!("token:{token1_addr}"), - format!("UniswapDayData:{day_id}"), - format!("PoolDayData:{day_id}:{pool_address}"), - format!("PoolHourData:{hour_id}:{pool_address}"), - format!("TokenDayData:{day_id}:{token0_addr}"), - format!("TokenDayData:{day_id}:{token1_addr}"), - format!("TokenHourData:{hour_id}:{token0_addr}"), - format!("TokenHourData:{hour_id}:{token1_addr}"), - ], - &BigInt::from(1 as i32), - ); - } -} -``` - -In the downstream modules consuming this store, you can query the store by key in `get` mode. Or, an even more powerful approach would be to filter needed store deltas by segments. `key` module of the `substreams` crates offers several helper functions. Using these functions you can extract the first/last/nth segment from a key: - -```rust -for delta in deltas.into_iter() { - let kind = key::first_segment(delta.get_key()); - let address = key::segment_at(delta.get_key(), 1); - // Do something for this kind and address -} -``` - -`key` module also provides corresponding `try_` methods that don't panic: -- `first_segment` & `try_first_segment` -- `last_segment` & `try_last_segment` -- `segment_at` & `try_segment_at` - -For a full example see [Uniswap V3 Substreams](https://github.com/streamingfast/substreams-uniswap-v3/blob/ca90fe3908a76905b43e05f0522e1e9338d88972/src/lib.rs#L1139-L1163) - -## Links -* [Uniswap-v3 Subgraph and Substreams](https://github.com/streamingfast/substreams-uniswap-v3) -* [Key module documentation](https://docs.rs/substreams/latest/substreams/key/index.html) diff --git a/docs/new/develop/modules/outputs.md b/docs/new/develop/modules/outputs.md deleted file mode 100644 index 8bf554133..000000000 --- a/docs/new/develop/modules/outputs.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -description: StreamingFast Substreams module outputs ---- - -# Output - -## Output overview - -Substreams `map` modules support a single `output`. The `output` must be a protobuf populated by data acquired inside the `map` module. If the module intends to provide a basic `output` type of a single value, such as a `String` or `bool`, a protobuf is still required. The single value needs to be wrapped in a protobuf for use as the `output` value from a `map` module. - -{% hint style="info" %} -**Note:** `store` modules **cannot** define an `output`. -{% endhint %} - -An `output` object has a `type` attribute defining the `type` of the `output` for the `map` module. The `output` definition is located in the Substreams manifest, within the module definition. - -```yaml -output: - type: proto:eth.erc721.v1.Transfers -``` diff --git a/docs/new/develop/modules/setting-up-handlers.md b/docs/new/develop/modules/setting-up-handlers.md deleted file mode 100644 index 01c62a31d..000000000 --- a/docs/new/develop/modules/setting-up-handlers.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -description: StreamingFast Substreams module handlers ---- - -# Module handlers - -## Module handlers overview - -To begin creating the custom module handlers initialize a new Rust project by using the `cargo` `init` command. - -```bash -# Creates a empty Rust project suitable for WASM compilation -cargo init --lib -``` - -Update the generated [`Cargo.toml`](https://github.com/streamingfast/substreams-template/blob/develop/Cargo.toml) file by using: - -{% code title="Cargo.toml" overflow="wrap" lineNumbers="true" %} -```rust -[package] -name = "substreams-template" -version = "0.1.0" -description = "Substream template demo project" -edition = "2021" -repository = "https://github.com/streamingfast/substreams-template" - -[lib] -name = "substreams" -crate-type = ["cdylib"] - -[dependencies] -ethabi = "17" -hex-literal = "0.3.4" -prost = "0.11" -# Use latest from https://crates.io/crates/substreams -substreams = "0.5" -# Use latest from https://crates.io/crates/substreams-ethereum -substreams-ethereum = "0.9" - -# Required so ethabi > ethereum-types build correctly under wasm32-unknown-unknown -[target.wasm32-unknown-unknown.dependencies] -getrandom = { version = "0.2", features = ["custom"] } - -[build-dependencies] -anyhow = "1" -substreams-ethereum = "0.8" - -[profile.release] -lto = true -opt-level = 's' -strip = "debuginfo" -``` -{% endcode %} - -View the [`Cargo.toml`](https://github.com/streamingfast/substreams-template/blob/develop/Cargo.toml) file in the repository. - -You compile the Rust code into [WebAssembly (WASM)](https://webassembly.org/), a binary instruction format that runs in a virtual machine. The compilation process generates a .so file. - -### **`Cargo.toml` configuration file breakdown** - -Build the Rust dynamic system library after the `package` by using: - -{% code title="Cargo.toml excerpt" %} -```toml -... - -[lib] -crate-type = ["cdylib"] -``` -{% endcode %} - -The next definition in the [`Cargo.toml`](https://github.com/streamingfast/substreams-template/blob/develop/Cargo.toml) configuration file is for `dependencies`. - -{% hint style="info" %} -**Note**: Module handlers compile down to a WASM module. Explicitly specify the target`asm32-unknown-unknown` by using `[target.wasm32-unknown-unknown.dependencies]`. -{% endhint %} - -#### `ethabi` - -The [`ethabi` crate ](https://crates.io/crates/ethabi)is used to decode events from the application binary interface (ABI) and is required for `substreams-ethereum` ABI capabilities. - -#### `hex-literal` - -The [`hex-literal` crate ](https://crates.io/crates/hex-literal)is used to define bytes from hexadecimal string literals at compile time. - -#### `substreams` - -The [`substreams` crate](https://docs.rs/substreams/latest/substreams/) offers all the basic building blocks for the module handlers. - -#### `substreams-ethereum` - -The [`substreams-ethereum` crate](https://crates.io/crates/substreams-ethereum-core) offers all the Ethereum constructs including blocks, transactions, eth, and useful ABI decoding capabilities. - -Because code is being built by WASM output it's necessary to configure Rust to match the correct architecture. Create and add a [`rust-toolchain.toml`](https://github.com/streamingfast/substreams-template/blob/develop/rust-toolchain.toml) configuration file at the root of your Substreams directory. - -### Rust toolchain - -{% code title="rust-toolchain.toml" overflow="wrap" lineNumbers="true" %} -```toml -[toolchain] -channel = "1.65" -components = [ "rustfmt" ] -targets = [ "wasm32-unknown-unknown" ] -``` -{% endcode %} - -View the [`rust-toolchain.toml`](https://github.com/streamingfast/substreams-template/blob/develop/rust-toolchain.toml) file in the repository. - -Build the code by using: - -```bash -cargo build --target wasm32-unknown-unknown --release -``` - -### **Rust build target** - -When running `cargo build` the target is set to `wasm32-unknown-unknown`, which is important because it specifies the goal is to generate compiled WASM code. - -To avoid having to specify the target `wasm32-unknown-unknown` for every `cargo` command, create a `config.toml` configuration file in the `.cargo` directory at the root of the Substreams project. The `config.toml` configuration file allows the target to be set automatically for all `cargo` commands. - -The content for the `config.toml` configuration file is: - -{% code title=".cargo/config.toml" %} -```toml -[build] -target = "wasm32-unknown-unknown" -``` -{% endcode %} - -The `config.toml` configuration file updates the default `cargo build` command to `cargo build --target wasm32-unknown-unknown` eliminating the need to specify the target manually every time you build. - -### ABI generation - -The [`substreams-ethereum` crate](https://crates.io/crates/substreams-ethereum) offers an [`Abigen`](https://docs.rs/substreams-ethereum-abigen/latest/substreams_ethereum_abigen/build/struct.Abigen.html) API to generate Rust types from a smart contract's ABI. - -Place the contract's [ABI JSON file](../../.gitbook/assets/erc721.json) in the Substreams project in the `abi` directory. - -### **Rust build script** - -Before building a package, Cargo compiles a build script into an executable if it has not already been built. The build script runs as part of the build process responsible for performing a variety of tasks. - -To cause Cargo to compile and run a script before building a package, place a file called `build.rs` in the root of the package. - -Create a [`build.rs`](https://github.com/streamingfast/substreams-template/blob/develop/build.rs) build script file in the root of the Substreams project by using: - -{% code title="build.rs" overflow="wrap" lineNumbers="true" %} -```rust -use anyhow::{Ok, Result}; -use substreams_ethereum::Abigen; - -fn main() -> Result<(), anyhow::Error> { - Abigen::new("ERC721", "abi/erc721.json")? - .generate()? - .write_to_file("src/abi/erc721.rs")?; - - Ok(()) -} -``` -{% endcode %} - -View the [`build.rs`](https://github.com/streamingfast/substreams-template/blob/develop/build.rs) file in the repository. - -Run the build script to generate the ABI directory and files. - -```bash -cargo build --target wasm32-unknown-unknown --release -``` - -Create a [`mod.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/abi/mod.rs) export file in the ABI directory, which is created by the Rust build process. The [`mod.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/abi/mod.rs) export file is responsible for exporting the generated Rust code. - -{% code title="src/abi/mod.rs" lineNumbers="true" %} -```rust -pub mod erc721; -``` -{% endcode %} - -View the [`mod.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/abi/mod.rs) file in the repository. - -You're now ready to [write the module handlers](writing-module-handlers.md). diff --git a/docs/new/develop/modules/types.md b/docs/new/develop/modules/types.md deleted file mode 100644 index 623ed0f9f..000000000 --- a/docs/new/develop/modules/types.md +++ /dev/null @@ -1,115 +0,0 @@ ---- -description: StreamingFast Substreams module types ---- - -# Module types - -## Module types overview - -Substreams uses two types of modules, `map` and `store`. - -* `map` modules are functions receiving bytes as input and output. These bytes are encoded protobuf messages. -* `store` modules are stateful, saving and tracking data through the use of key-value stores. - -### `store` modules - -`store` modules write to key-value stores. - -{% hint style="info" %} -**Note**: To ensure successful and proper parallelization can occur, `store` modules are not permitted to read any of their own data or values. -{% endhint %} - -Stores declaring their own data types expose methods capable of mutating keys within the `store`. - -### Core principle usage of stores - -* Do not save keys in stores **unless they are going to be read by a downstream module**. Substreams stores are a way to aggregate data, but they are **not meant to be a storage layer**. -* Do not save all transfers of a chain in a `store` module, rather, output them in a `map` and have a downstream system store them for querying. - -There are limitations impose on store usage. Specifically, each key/value entry must be smaller than 10MiB while a store cannot exceed 1GiB total. Keys being string, each character in the key account for 1 byte of storage space. - -### Important store properties - -The two important store properties are `valueType,`and `updatePolicy`. - -#### `valueType` property - -The `valueType` property instructs the Substreams runtime of the data to be saved in the `stores`. - -| Value | Description | -| ------------------------------ | -------------------------------------------------------------------------------- | -| `bytes` | A basic list of bytes | -| `string` | A UTF-8 string | -| `proto:fully.qualified.Object` | Decode bytes by using the protobuf definition `fully.qualified.Object` | -| `int64` | A string-serialized integer by using int64 arithmetic operations | -| `float64` | A string-serialized floating point value, used for float64 arithmetic operations | -| `bigint` | A string-serialized integer, supporting precision of any depth | -| `bigfloat` | A string-serialized floating point value, supporting precision up to 100 digits | - -#### `updatePolicy` property - -The `updatePolicy` property determines what methods are available in the runtime. - -The `updatePolicy` also defines the merging strategy for identical keys found in two contiguous stores produced through parallel processing. - -| Method | Supported Value Types | Merge strategy\* | -| ------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `set` | `bytes`, `string`, `proto:...` | The last key wins | -| `set_if_not_exists` | `bytes`, `string`, `proto:...` | The first key wins | -| `add` | `int64`, `bigint`, `bigfloat`, `float64` | Values are summed up | -| `min` | `int64`, `bigint`, `bigfloat`, `float64` | The lowest value is kept | -| `max` | `int64`, `bigint`, `bigfloat`, `float64` | The highest value is kept | -| `append` | `string`, `bytes` | Both keys are concatenated in order. Appended values are limited to 8Kb. Aggregation pattern examples are available in the [`lib.rs`](https://github.com/streamingfast/substreams-uniswap-v3/blob/develop/src/lib.rs#L760) file | - -{% hint style="success" %} -**Tip**: All update policies provide the `delete_prefix` method. -{% endhint %} - -The merge strategy is **applied during parallel processing**. - -* A module has built two partial stores containing keys for segment A, blocks 0-1000, and segment B, blocks 1000-2000, and is prepared to merge them into a complete store. -* The complete store is represented acting as if the processing was done in a linear fashion, starting at block 0 and proceeding up to block 2000. - -{% hint style="warning" %} -**Important**_**:** _ To preserve the parallelization capabilities of the system, **Substreams is not permitted to read what it has written or read from a `store` actively being written**. - -A downstream module is created to read from a store by using one of its inputs to point to the output of the `store` module. -{% endhint %} - -### Ordinals - -Ordinals allow a key-value store to have multiple versions of a key within a single block. The `store` APIs contain different methods of `ordinal` or `ord`. - -For example, the price for a token can change after transaction B and transaction D, and a downstream module might want to know the value of a key before transaction B **and between B and D**_._ - -{% hint style="warning" %} -**Important**: Ordinals **must be set every time a key is set** and **you can only set keys in increasing ordinal order**, or by using an ordinal equal to the previous. -{% endhint %} - -In situations where a single key for a block is required and ordering in the store is not important, the ordinal uses a value of zero. - -### `store` modes - -You can consume data in one of two modes when declaring a `store` as an input to a module. - -#### `get mode` - -The `get mode` function provides the module with a key-value store that is guaranteed to be synchronized up to the block being processed. It's possible to query stores by using the `get_at`, `get_last` and `get_first` methods. - -{% hint style="success" %} -**Tip:** Lookups are local, in-memory, and **extremely high-speed**. -{% endhint %} - -The definition of `store` method behavior is: - -* The `get_last` method is the fastest because it queries the store directly. -* The `get_first` method first goes through the current block's deltas in reverse order, before querying the store, in case the key being queried was mutated in the block. -* The `get_at` method unwinds deltas up to a specific ordinal, ensuring values for keys set midway through a block are still reachable. - -#### `deltas mode` - -`deltas` mode provides the module with **all the changes** occurring in the source `store` module. Updates, creates, and deletes of the keys mutated during the block processing become available. - -{% hint style="info" %} -**Note:** When a `store` is set as an input to the module, it is read-only and you cannot modify, update or mutate them. -{% endhint %} diff --git a/docs/new/develop/modules/writing-module-handlers.md b/docs/new/develop/modules/writing-module-handlers.md deleted file mode 100644 index a1895c977..000000000 --- a/docs/new/develop/modules/writing-module-handlers.md +++ /dev/null @@ -1,287 +0,0 @@ ---- -description: StreamingFast Substreams module handler creation ---- - -# Module handler creation - -## Module handler creation overview - -After generating the ABI and protobuf Rust code, you need to write the handler code. Save the code into the `src` directory and use the filename [`lib.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/lib.rs). - -{% code title="src/lib.rs" overflow="wrap" lineNumbers="true" %} -```rust -mod abi; -mod pb; -use hex_literal::hex; -use pb::erc721; -use substreams::prelude::*; -use substreams::{log, store::StoreAddInt64, Hex}; -use substreams_ethereum::{pb::eth::v2 as eth, NULL_ADDRESS}; - -// Bored Ape Club Contract -const TRACKED_CONTRACT: [u8; 20] = hex!("bc4ca0eda7647a8ab7c2061c2e118a18a936f13d"); - -substreams_ethereum::init!(); - -/// Extracts transfers events from the contract -#[substreams::handlers::map] -fn map_transfers(blk: eth::Block) -> Result { - Ok(erc721::Transfers { - transfers: blk - .events::(&[&TRACKED_CONTRACT]) - .map(|(transfer, log)| { - substreams::log::info!("NFT Transfer seen"); - - erc721::Transfer { - trx_hash: log.receipt.transaction.hash.clone(), - from: transfer.from, - to: transfer.to, - token_id: transfer.token_id.low_u64(), - ordinal: log.block_index() as u64, - } - }) - .collect(), - }) -} - -/// Store the total balance of NFT tokens for the specific TRACKED_CONTRACT by holder -#[substreams::handlers::store] -fn store_transfers(transfers: erc721::Transfers, s: StoreAddInt64) { - log::info!("NFT holders state builder"); - for transfer in transfers.transfers { - if transfer.from != NULL_ADDRESS { - log::info!("Found a transfer out {}", Hex(&transfer.trx_hash)); - s.add(transfer.ordinal, generate_key(&transfer.from), -1); - } - - if transfer.to != NULL_ADDRESS { - log::info!("Found a transfer in {}", Hex(&transfer.trx_hash)); - s.add(transfer.ordinal, generate_key(&transfer.to), 1); - } - } -} - -fn generate_key(holder: &Vec) -> String { - return format!("total:{}:{}", Hex(holder), Hex(TRACKED_CONTRACT)); -} -``` -{% endcode %} - -View the [`lib.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/lib.rs) file in the repository. - -### **Module handler breakdown** - -The logical sections of the [`lib.rs`](https://github.com/streamingfast/substreams-template/blob/develop/src/lib.rs) file are outlined and described in greater detail. - -Import the necessary modules. - -{% code title="lib.rs excerpt" overflow="wrap" %} -```rust -mod abi; -mod pb; -use hex_literal::hex; -use pb::erc721; -use substreams::{log, store, Hex}; -use substreams_ethereum::{pb::eth::v2 as eth, NULL_ADDRESS, Event}; -``` -{% endcode %} - -Store the tracked contract in the example in a `constant`. - -{% code title="lib.rs excerpt" %} -```rust -const TRACKED_CONTRACT: [u8; 20] = hex!("bc4ca0eda7647a8ab7c2061c2e118a18a936f13d"); -``` -{% endcode %} - -Define the `map` module in the Substreams manifest. - -{% code title="manifest excerpt" %} -```yaml -- name: map_transfers - kind: map - initialBlock: 12287507 - inputs: - - source: sf.ethereum.type.v2.Block - output: - type: proto:eth.erc721.v1.Transfers -``` -{% endcode %} - -Notice the: `name: map_transfers`, the module in the manifest name matches the handler function name. Also notice, there is one [`inputs`](inputs.md) and one [`output`](outputs.md) definition. - -The [`inputs`](inputs.md) uses the standard Ethereum Block, `sf.ethereum.type.v2.Block,` provided by the [`substreams-ethereum` crate](https://crates.io/crates/substreams-ethereum-core). - -The output uses the `type` `proto:eth.erc721.v1.Transfers` which is a custom protobuf definition provided by the generated Rust code. - -The function signature produced resembles: - -{% code title="lib.rs excerpt" %} -```rust -#[substreams::handlers::map] -fn map_transfers(blk: eth::Block) -> Result { - ... -} -``` -{% endcode %} - -### **Rust macros** - -Did you notice the `#[substreams::handlers::map]` on top of the function? It is a [Rust macro](https://doc.rust-lang.org/book/ch19-06-macros.html) provided by the [`substreams` crate](https://docs.rs/substreams/latest/substreams/). - -The macro decorates the handler function as a `map.` Define `store` modules by using the syntax `#[substreams::handlers::store]`. - -### Module handler function - -The `map` extracts ERC721 transfers from a _`Block`_ object. The code finds all the `Transfer` `events` emitted by the tracked smart contract. As the events are encountered they are decoded into `Transfer` objects. - -{% code title="lib.rs excerpt" overflow="wrap" %} -```rust -/// Extracts transfers events from the contract -#[substreams::handlers::map] -fn map_transfers(blk: eth::Block) -> Result { - Ok(erc721::Transfers { - transfers: blk - .events::(&[&TRACKED_CONTRACT]) - .map(|(transfer, log)| { - substreams::log::info!("NFT Transfer seen"); - - erc721::Transfer { - trx_hash: log.receipt.transaction.hash.clone(), - from: transfer.from, - to: transfer.to, - token_id: transfer.token_id.low_u64(), - ordinal: log.block_index() as u64, - } - }) - .collect(), - }) -} -``` -{% endcode %} - -Define the `store` module in the Substreams manifest. - -{% code title="manifest excerpt" %} -```yaml -- name: store_transfers - kind: store - initialBlock: 12287507 - updatePolicy: add - valueType: int64 - inputs: - - map: map_transfers -``` -{% endcode %} - -{% hint style="info" %} -**Note:** `name: store_transfers` corresponds to the handler function name. -{% endhint %} - -The `inputs` corresponds to the `output` of the `map_transfers` `map` module typed as `proto:eth.erc721.v1.Transfers`. The custom protobuf definition is provided by the generated Rust code. - -{% code title="lib.rs excerpt" %} -```rust -#[substreams::handlers::store] -fn store_transfers(transfers: erc721::Transfers, s: store::StoreAddInt64) { - ... -} -``` -{% endcode %} - -{% hint style="info" %} -**Note**: the `store` always receives itself as its own last input. -{% endhint %} - -In the example the `store` module uses an `updatePolicy` set to `add` and a `valueType` set to `int64` yielding a writable `store` typed as `StoreAddInt64`. - -{% hint style="info" %} -**Note**: **Store types** - -* The writable `store` is always the last parameter of a `store` module function. -* The `type` of the writable `store` is determined by the `updatePolicy` and `valueType` of the `store` module. -{% endhint %} - -The goal of the `store` in the example is to track a holder's current NFT `count` for the smart contract provided. The tracking is achieved through the analysis of `Transfers`. - -**`Transfers` in detail** - -* If the "`from`" address of the `transfer` is the `null` address (`0x0000000000000000000000000000000000000000`) and the "`to`" address is not the `null` address, the "`to`" address is minting a token, which results in the `count` being incremented. -* If the "`from`" address of the `transfer` is not the `null` address and the "`to`" address is the `null` address, the "`from`" address has burned a token, which results in the `count` being decremented. -* If both the "`from`" and the "`to`" address is not the `null` address, the `count` is decremented from the "`from`" address and incremented for the "`to`" address. - -### `store` concepts - -There are three important things to consider when writing to a `store`: - -* `ordinal` -* `key` -* `value` - -#### `ordinal` - -`ordinal` represents the order in which the `store` operations are applied. - -The `store` handler is called once per `block.` - -The `add` operation may be called multiple times during execution, for various reasons such as discovering a relevant event or encountering a call responsible for triggering a method call. - -{% hint style="info" %} -**Note**: Blockchain execution models are linear. Operations to add must be added linearly and deterministically. -{% endhint %} - -If an `ordinal` is specified, the order of execution is guaranteed. In the example, when the `store` handler is executed by a given set of `inputs`, such as a list of `Transfers`, it emits the same number of `add` calls and `ordinal` values for the execution. - -#### `key` - -Stores are [key-value stores](https://en.wikipedia.org/wiki/Key%E2%80%93value\_database). Care needs to be taken when crafting a `key` to ensure it is unique **and flexible**. - -If the `generate_key` function in the example returns the `TRACKED_CONTRACT` address as the `key`, it is not unique among different token holders. - -The `generate_key` function returns a unique `key` for holders if it contains only the holder's address. - -{% hint style="warning" %} -**Important**: Issues are expected when attempting to track multiple contracts. -{% endhint %} - -#### `value` - -The value being stored. The `type` is dependent on the `store` `type` being used. - -{% code title="lib.rs excerpt" overflow="wrap" %} -```rust -#[substreams::handlers::store] -fn store_transfers(transfers: erc721::Transfers, s: StoreAddInt64) { - log::info!("NFT holders state builder"); - for transfer in transfers.transfers { - if transfer.from != NULL_ADDRESS { - log::info!("Found a transfer out {}", Hex(&transfer.trx_hash)); - s.add(transfer.ordinal, generate_key(&transfer.from), -1); - } - - if transfer.to != NULL_ADDRESS { - log::info!("Found a transfer in {}", Hex(&transfer.trx_hash)); - s.add(transfer.ordinal, generate_key(&transfer.to), 1); - } - } -} - -fn generate_key(holder: &Vec) -> String { - return format!("total:{}:{}", Hex(holder), Hex(TRACKED_CONTRACT)); -} -``` -{% endcode %} - -### Summary - -Both handler functions have been written. - -One handler function for extracting relevant _`transfers`_, and a second to store the token count per recipient. - -Build Substreams to continue the setup process. - -```bash -cargo build --target wasm32-unknown-unknown --release -``` - -The next step is to run Substreams with all of the changes made by using the generated code. diff --git a/docs/new/develop/parameterized-modules.md b/docs/new/develop/parameterized-modules.md deleted file mode 100644 index b373e3259..000000000 --- a/docs/new/develop/parameterized-modules.md +++ /dev/null @@ -1,156 +0,0 @@ - - -Substreams allows you to pass parameters to your modules by specifying them in the manifest. - -## Parameterization of a Factory contract - -It's quite common for a smart contract to be deployed on different networks or even by different dApps within the same network. Uniswap Factory smart contract is a good example of that. - -When running Substreams for a dApp, you need to know the smart contract deployment address and for obvious reasons, this address will be different for each deployment. - -Instead of hard-coding the address in the Substreams binary, you can customize it without having to rebuild or even repackage the Substreams package. The consumer can then just provide the address as a parameter. - -First, you need to add the `params` field as an input. Note that it's always a string and it's always the first input for the module: - -```yaml -modules: - - name: map_pools_created - kind: map - inputs: - - params: string - - source: sf.ethereum.type.v2.Block - output: - type: proto:uniswap.types.v1.Pools -params: - map_params: 1f98431c8ad98523631ae4a59f267346ea31f984 -``` - -You can specify the default value directly in the manifest. In this case, we use `0x1f98431c8ad98523631ae4a59f267346ea31f984` - the deployment address for UniswapV3 contract on Ethereum Mainnet. - -Handling the parameter in the module is easy. The module handler receives it as a first input parameter and you can use it to filter transactions instead of the hard-coded value: - -```rust -#[substreams::handlers::map] -pub fn map_pools_created(params: String, block: Block) -> Result { - let factory_address = Hex::decode(params).unwrap(); - Ok(Pools { - pools: block - .events::(&[&factory_address]) - .filter_map(|(event, log)| { - // skipped: extracting pool information from the transaction - Some(Pool { - address, - token0, - token1, - ..Default::default() - }) - }) - .collect(), - }) -} -``` - -To pass the parameter to the module using `substreams` CLI you can use `-p` key: - -```bash -substreams gui -e $SUBSTREAMS_ENDPOINT map_pools_created -t +1000 -p map_pools_created="1f98431c8ad98523631ae4a59f267346ea31f984"` -``` - -### Documenting parameters -It's always a good idea to document what the params represent and how they are structured, so the consumers of your modules know how to properly parameterize them. You can use `doc` field for the module definition in the manifest. - -```yaml -modules: - - name: map_pools_created - kind: map - inputs: - - source: sf.ethereum.type.v2.Block - - params: string - output: - type: proto:uniswap.types.v1.Pools - doc: | - Params contains Uniswap factory smart contract address without `0x` prefix, i.e. 1f98431c8ad98523631ae4a59f267346ea31f984 for Ethereum Mainnet -``` - -## Advanced parameters - -Sometimes you may need to use multiple parameters for a module. To pass multiple parameters, you can encode them as a URL-encoded query string, i.e. `param1=value1¶m2=value2`. - -Suppose you want to track transfers to/from a certain address exceeding a certain amount of ETH. Your module manifest could look like this: - -```yaml -modules: - - name: map_whale_transfers - kind: map - inputs: - - params: string - - source: sf.ethereum.type.v2.Block - output: - type: proto:Transfers -params: - map_params: address=aaa..aaa&amount=100 -``` - -Our module gets a params string with two parameters: `address` and `amount`. - -In your module handler, you can decode your parameters using one of the URL decoding crates such as `serde_qs`, `serde_urlencoded` or your own helper functions. Here's an example using `serde_qs`: - -```rust -#[derive(Debug, Deserialize)] -struct Params { - address: String, - amount: u64, -} - -#[substreams::handlers::map] -pub fn map_whale_transfers(params: String, block: Block) -> Result { - let query: Params = serde_qs::from_str(params.as_str()).unwrap(); - log::info!("Tracking transfers for address: {} of more than {} ETH", query.address, query.amount); - - // filter transfers by address and amount -} -``` - -Sometimes parameters can be optional, i.e. you want to track all transfers rather than a specific address. Decoding will look like this in that case: - -```rust -#[derive(Debug, Deserialize)] -struct QueryParams { - address: Option, - amount: u64, -} - -#[substreams::handlers::map] -pub fn map_whale_transfers(params: String, block: Block) -> Result { - let query: QueryParams = serde_qs::from_str(params.as_str()).unwrap(); - - if query.address.is_none() { - log::info!("Tracking all of more than {} ETH", query.amount); - } - else { - log::info!("Tracking transfers for address: {} of more than {} ETH", query.address, query.amount); - } -} -``` - -You can even pass a vector of addresses to track multiple specific whales in our example: - -```rust -#[derive(Debug, Deserialize)] -struct QueryParams { - address: Vec, - amount: u64, -} - -#[substreams::handlers::map] -pub fn map_whale_transfers(params: String, block: Block) -> Result { - let query: QueryParams = serde_qs::from_str(params.as_str()).unwrap(); - log::info!("Tracking transfers for addresses: {:?} of more than {} ETH", query.address, query.amount); -} -``` - -Depending on the crate you use to decode params string, you can pass them to Substreams CLI like this for example: - -```bash -substreams gui map_whale_transfers -p map_whale_transfers="address[]=aaa..aaa&address[]=bbb..bbb&amount=100" -``` diff --git a/docs/new/develop/rust-crates.md b/docs/new/develop/rust-crates.md deleted file mode 100644 index 606ac78df..000000000 --- a/docs/new/develop/rust-crates.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -description: Substreams Rust APIs ---- - -# Rust crates - -## Rust crates overview - -The official [`substreams` crate](https://crates.io/crates/substreams) helps developers create module handlers. - -* Use the [`substreams-ethereum` crate](https://crates.io/crates/substreams-ethereum) for Ethereum and other Ethereum-compatible chains. -* Use the [`substreams-solana` crate](https://crates.io/crates/substreams-solana) for the Solana blockchain. -* Use the [`substreams-antelope` crate](https://github.com/pinax-network/substreams-antelope) for the Antelope blockchain (by [Pinax Network](https://pinax.network/)) - -{% hint style="info" %} -**Note**: If a crate is not available for Substreams, you can use the `spkg` release for the chain, which includes the `Block` protobuf model, and generate the Rust structs yourself. -{% endhint %} - -### Third-party libraries - -Any third-party library capable of compiling `wasm32` can be used for execution in Substreams services. - -Some libraries include kernel syscalls or other operations unavailable in the Substreams execution environment and cannot be compiled to WASM. The internal functionality of third-party libraries is an essential consideration for Substreams development. - -Helpful information people found through the use of third-party libraries and Substreams together include: - -* [`tiny_keccak`](https://docs.rs/tiny-keccak): an implementation of Keccak-derived functions specified in FIPS-202, SP800-185, and KangarooTwelve. - -### Git Versions - -[Specifying dependencies from Git repositories](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories) is possible in Rust, but it is **NOT recommended** by the Substreams team, as they are not fully tested and can bring bugs to your Substreams project. - -The Substreams team recommends using the templates provided in the [Examples section](examples.md) as a starting point to develop your Substreams application. \ No newline at end of file diff --git a/docs/new/references/chains-and-endpoints.md b/docs/new/references/chains-and-endpoints.md deleted file mode 100644 index 25c597461..000000000 --- a/docs/new/references/chains-and-endpoints.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -description: StreamingFast Substreams chains and endpoints ---- - -# Chains and endpoints - -## Chains and endpoints overview - -The different blockchains have separate endpoints that Substreams uses. You will use the endpoint that matches the blockchain you've selected for your development initiative. - -### Supported blockchains and protobuf models - -Protobuf definitions and public endpoints are provided for the supported protocols and chains. - -{% hint style="success" %} -**Tip**: All of the endpoints listed in the documentation require [authentication](authentication.md) before use. -{% endhint %} - -{% hint style="warning" %} -**Important**_:_ Endpoints serve protobuf models specific to the underlying blockchain protocol and must match the `source:` field for the module. - -**Streaming a `sf.near.type.v1.Block` from an Ethereum endpoint does not work!** -{% endhint %} - -

Protobuf for the different supported chains

- -| Protocol | Proto model | Latest package | -| -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -| Ethereum | [`sf.ethereum.type.v2.Block`](https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/type/v2/type.proto) | [ethereum-v0.10.4.spkg](https://github.com/streamingfast/sf-ethereum/releases/download/v0.10.2/ethereum-v0.10.4.spkg) | -| NEAR | [`sf.near.type.v1.Block`](https://github.com/streamingfast/firehose-near/blob/develop/proto/sf/near/type/v1/type.proto) | | -| Solana | [`sf.solana.type.v1.Block`](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto) | [solana-v0.1.0.spkg](https://github.com/streamingfast/sf-solana/releases/download/v0.1.0/solana-v0.1.0.spkg) | -| Cosmos | [`sf.cosmos.type.v1.Block`](https://github.com/figment-networks/proto-cosmos/blob/main/sf/cosmos/type/v1/type.proto) | | -| Arweave | [`sf.arweave.type.v1.Block`](https://github.com/streamingfast/firehose-arweave/blob/develop/proto/sf/arweave/type/v1/type.proto) | | -| Aptos | [`aptos.extractor.v1.Block`](https://github.com/aptos-labs/aptos-core/blob/main/crates/aptos-protos/proto/aptos/extractor/v1/extractor.proto) | | -| Bitcoin | [`sf.bitcoin.type.v1.Block`](https://github.com/streamingfast/firehose-bitcoin/blob/develop/proto/sf/bitcoin/type/v1/type.proto) | | - -## Endpoints - -* **Ethereum Mainnet**: `mainnet.eth.streamingfast.io:443` -* **Ethereum Görli**: `goerli.eth.streamingfast.io:443` -* **Ethereum Sepolia**: `sepolia.eth.streamingfast.io:443` -* **Ethereum Holesky**: `holesky.eth.streamingfast.io:443` -* **Polygon** **Mainnet**: `polygon.streamingfast.io:443` -* **Mumbai Testnet**: `mumbai.streamingfast.io:443` -* **Arbitrum One:** `arb-one.streamingfast.io:443` -* **BNB**: `bnb.streamingfast.io:443` -* **Optimism:** `opt-mainnet.streamingfast.io:443` -* **Avalanche C-Chain Mainnet**: `avalanche-mainnet.streamingfast.io:443` -* **NEAR Mainnet**: `mainnet.near.streamingfast.io:443` -* **NEAR Testnet**: `testnet.near.streamingfast.io:443` -* **Solana mainnet-beta**: `mainnet.sol.streamingfast.io:443` -* **Arweave Mainnet**: `mainnet.arweave.streamingfast.io:443` -* **Aptos Testnet**: `testnet.aptos.streamingfast.io:443` -* **Bitcoin Mainnet**: `mainnet.btc.streamingfast.io:443` - -## Others - -You can support other blockchains for Substreams through Firehose instrumentation. Learn more in the [official Firehose documentation](https://firehose.streamingfast.io/). diff --git a/docs/new/references/command-line-interface.md b/docs/new/references/command-line-interface.md deleted file mode 100644 index 90802692a..000000000 --- a/docs/new/references/command-line-interface.md +++ /dev/null @@ -1,242 +0,0 @@ ---- -description: StreamingFast Substreams command line interface (CLI) ---- - -# Substreams CLI reference - -## `substreams` CLI reference overview - -The `substreams` command line interface (CLI) is the primary user interface and the main tool for sending requests and receiving data. - -The `substreams` CLI exposes many commands to developers enabling a range of features. - -{% hint style="info" %} -**Note**: When a package is specified, it is optional. If you do use it, you can use: - -* Local `substreams.yaml` configuration files -* Local `.spkg` package files -* Remote `.spkg` package URLs -* Local directory containing a `substreams.yaml` file - -If you choose to not use it, make sure that you are in a directory that contains a substreams.yaml file. Otherwise, you will get a usage error back. -{% endhint %} - -### **`run`** - -The `run` command connects to a Substreams endpoint and begins processing data. - -{% code title="run command" overflow="wrap" %} -```bash -substreams run -e mainnet.eth.streamingfast.io:443 \ - -t +1 \ - ./substreams.yaml \ - module_name -``` -{% endcode %} - -The details of the run command are: - -* `-e mainnet.eth.streamingfast.io:443` is the endpoint of the provider running your Substreams. -* `-t +1` or `--stop-block` only requests a single block; the stop block is the manifest's `initialBlock` + 1. -* `substreams.yaml` is the path where you have defined your [Substreams manifest](manifests.md). You can use a `.spkg` or `substreams.yaml` configuration file. -* `module_name` is the module we want to `run`, referring to the module name [defined in the Substreams manifest](manifests.md#modules-.name). - -{% hint style="success" %} -**Tip**: Passing a different `-s` or `--start-block` runs prior modules at a higher speed. Output is provided at the requested start block, keeping snapshots along the way if you want to process it again. -{% endhint %} - -#### Run example with output - -{% code title="substreams run " overflow="wrap" %} -```bash -$ substreams run -e mainnet.eth.streamingfast.io:443 \ - https://github.com/Jannis/gravity-substream/releases/download/v0.0.1/gravity-v0.1.0.spkg \ - gravatar_updates -o json -``` -{% endcode %} - -The output of the `gravatar_updates` module starting at block `6200807` will print a message resembling: - -{% code title="run output" %} -```bash -{ - "updates": [ - { - "id": "39", - "owner": "0xaadcc13071fdf9c73cfbb8d97639ea68aa6fd1d2", - "displayName": "alex | OpenSea", - "imageUrl": "https://ucarecdn.com/13a67247-cb89-417a-92d2-50a7d7aa481c/-/crop/382x382/0,0/-/preview/" - } - ] -} -... -``` -{% endcode %} - -{% hint style="info" %} -**Note**: The `-o` or `--output` flag alters the output format. -{% endhint %} - -The available output display options are: - -* `ui`, a nicely formatted, UI-driven interface, displaying progress information and execution logs. -* `json`, an indented stream of data, **not** displaying progress information or logs, only data output for blocks proceeding the start block. -* `jsonl`, same as `json` showing every individual output on a single line. - -### `gui` - -The `gui` command pops up a terminal-based graphical user interface. - -Its parameters are very similar to those of `run`, but provides a UI to navigate the results instead of a stream of data. - -#### Replay mode - -When you run a `gui` session, a file called `replay.log` gets written with the contents of the streamed data that persists after closing the GUI. - -You can reload the data without hitting the server again using `--replay`. The data is immediately reloaded in the GUI, ready for more inspection. - -### `pack` - -The `pack` command builds a shippable, importable package from a `substreams.yaml` manifest file. - -{% code title="pack command" overflow="wrap" %} -```bash -$ substreams pack ./substreams.yaml -``` -{% endcode %} - -The output of the `pack` command will print a message resembling: - -{% code title="pack output" overflow="wrap" %} -```bash -... -Successfully wrote "your-package-v0.1.0.spkg". -``` -{% endcode %} - -### `info` - -The `info` command prints out the contents of a package for inspection. It works on both local and remote `yaml` or `spkg` configuration files. - -{% code title="info command" overflow="wrap" %} -```bash -$ substreams info ./substreams.yaml -``` -{% endcode %} - -The output of the `info` command will print a message resembling: - -{% code title="info output" overflow="wrap" %} -```bash -Package name: solana_spl_transfers -Version: v0.5.2 -Doc: Solana SPL Token Transfers stream - - Stream SPL token transfers to the nearest human being. - -Modules: ----- -Name: spl_transfers -Initial block: 130000000 -Kind: map -Output Type: proto:solana.spl.v1.TokenTransfers -Hash: 2b59e4e840f814f4154a688c2935da9c3b61dc61 - -Name: transfer_store -Initial block: 130000000 -Kind: store -Value Type: proto:solana.spl.v1.TokenTransfers -Update Policy: UPDATE_POLICY_SET -Hash: 11fd70768029bebce3741b051c15191d099d2436 -``` -{% endcode %} - -### `graph` - -The `graph` command prints out a visual graph of the package in the [mermaid-js format](https://mermaid.js.org/intro/n00b-syntaxReference.html). - -{% hint style="success" %} -**Tip**: [Mermaid Live Editor](https://mermaid.live/) is the visual editor used by Substreams. -{% endhint %} - -{% code title="graph command" overflow="wrap" %} -````bash -$ substreams graph ./substreams.yaml - [±master ●●] -Mermaid graph: - -```mermaid -graph TD; - spl_transfers[map: spl_transfers] - sf.solana.type.v1.Block[source: sf.solana.type.v1.Block] --> spl_transfers - transfer_store[store: transfer_store] - spl_transfers --> transfer_store -``` -```` -{% endcode %} - -The `graph` command will result in a graphic resembling: - -{% embed url="https://mermaid.ink/svg/pako:eNp1kMsKg0AMRX9Fsq5Ct1PootgvaHeOSHBilc6LeRRE_PeOUhe2dBOSm5NLkglaIwgYPBzaPruXJ66zzFvZBIfad-R8pdCyvVSvUFd4I1FjEUZLxetYXKRpn5U30bXE_vXrLM_Pe7vFbSsaH4yjao3sS61_dlu99tDCwAEUOYWDSJdNi8Ih9KSIA0upoA6jDBy4nhMarcBAVzGkcWAdSk8HwBjMbdQtsOAibVA5YHqU-lDzG43ick8" %} -Mermaid generated graph diagram -{% endembed %} - -### `inspect` - -The `inspect` command reaches deep into the file structure of a `yaml` configuration file or `spkg` package and is used mostly for debugging, or if you're curious\_.\_ - -{% code title="inspect command" overflow="wrap" %} -```bash -$ substreams inspect ./substreams.yaml | less -``` -{% endcode %} - -The output of the `inspect` command will print a message resembling: - -{% code title="inspect output" overflow="wrap" %} -```bash -proto_files -... -modules { - modules { - name: "my_module_name" -... -``` -{% endcode %} - -### Help - -To view a list of available commands and brief explanations in the `substreams` CLI, run the `substreams` command in a terminal passing the `-h` flag. You can use this help reference at any time. - -{% code title="help option" overflow="wrap" %} -```bash -substreams -h -``` -{% endcode %} - -The output of the `help` command will print a message resembling: - -{% code title="help output" overflow="wrap" %} -```bash -Usage: - substreams [command] - -Available Commands: - completion Generate the autocompletion script for the specified shell - decode - graph Generate mermaid-js graph document - help Help about any command - info Display package modules and docs - inspect Display low-level package structure - pack Build an .spkg out of a .yaml manifest - protogen Generate Rust bindings from a package - run Stream modules from a given package on a remote endpoint - tools Developer tools related to substreams - -Flags: - -h, --help help for substreams - -v, --version version for substreams - -Use "substreams [command] --help" for more information about a command. -``` -{% endcode %} diff --git a/docs/new/references/faq.md b/docs/new/references/faq.md deleted file mode 100644 index 32b0127f0..000000000 --- a/docs/new/references/faq.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -description: StreamingFast Substreams frequently asked questions ---- - -# FAQ - -## **Substreams FAQ overview** - -You can find answers to common Substreams questions in the FAQ documentation. If the answer you're looking for is not included [contact the StreamingFast team](https://discord.gg/mYPcRAzeVN) through Discord to get help. - -### **What is Substreams?** - -Substreams is an exceptionally powerful processing engine capable of consuming streams of rich blockchain data. Substreams refines and shapes the data for painless digestion by end-user applications, such as decentralized exchanges. - -### **Do I need Firehose to use Substreams?** - -Developers do not need a dedicated installation of Firehose to use Substreams. StreamingFast provides a public Firehose endpoint made available to developers. - -### **Is it possible to use Substreams in my subgraph?** - -Yes, Substreams compliments and extend the capabilities and functionalities of subgraphs. Additional information is available in the [Substreams documentation for graph-node and subgraphs](https://substreams.streamingfast.io/reference-and-specs/graph-node-setup). - -### **Is it possible to use Substreams for production deployments?** - -Yes. Substreams is [generally available](https://streamingfastio.medium.com/substreams-reach-general-availability-48272f6e942e) and used in production at multiple outlets. - -### **What's in the Substreams name?** - -Substreams is the name of the engine, and of the product. It is to be capitalized and kept plural. One can speak of an individual _module_ or individual _stream_ but in general, when speaking about the engine, you would use the word "_Substreams"_. - -### **What is the `substreams` CLI?** - -The [`substreams` command line interface (CLI)](command-line-interface.md) is the main tool developers use to use the Substreams engine. The [`substreams` CLI](command-line-interface.md) provides a range of features, commands, and flags. Additional information for the [`substreams` CLI](command-line-interface.md) is available in the Substreams documentation. - -### **How do I get a Substreams authentication token?** - -Authentication tokens are required to use Substreams and connect to the public Firehose endpoint. Full [instructions for obtaining a StreamingFast authentication token](https://substreams.streamingfast.io/reference-and-specs/authentication) are available in the Substreams documentation. - -### **My Substreams authentication token isn’t working, what do I do?** - -The StreamingFast team is [available on Discord to resolve problems](https://discord.gg/Ugc7KtkA) related to obtaining or by using authentication tokens. - -The Substreams documentation also [provides general instructions surrounding authentication](https://substreams.streamingfast.io/reference-and-specs/authentication) tokens. - -### **How do I create a Substreams module?** - -Developers create their own Substreams implementations in a variety of ways. Use these [examples](reference-and-specs/examples.md) as a reference and starting point. - -The Substreams documentation [provides a Developer's guide](https://substreams.streamingfast.io/developer-guide/overview) to assist you to understand and use Substreams. - -### **What is Substreams used for?** - -Substreams and Firehose work together to index and process blockchain data. Substreams is used for transforming rich blockchain data and exposing it to the needs of application developers. - -### **Is Substreams free?** - -Yes, Substreams is an open-source project and there are free (albeit rate-limited) endpoints available. - -### **How does a developer reach the information returned from a call to Substreams from a web-based UI?** - -Substreams is not meant to be piped to a web UI, it’s a data transformation layer. Substreams reaches Subgraphs, as a data source, and makes GraphQL available for web consumption. Other sinks might expose APIs for web browsers, however, it's not the responsibility of Substreams. - -### Is it possible to listen for new blocks? - -Specifying a stop block value of zero (0), the default enables transparent handoff from historical to real-time blocks. - -### **Does StreamingFast have a Discord?** - -Yes, [join the StreamingFast Discord](https://discord.gg/Ugc7KtkA). - -### **Is StreamingFast on Twitter?** - -Yes, [find StreamingFast on their official Twitter account](https://twitter.com/streamingfastio). - -### **Is StreamingFast on YouTube?** - -Yes, [find StreamingFast on their official YouTube account](https://www.youtube.com/c/streamingfast). - -### **Who is dfuse?** - -StreamingFast was originally called dfuse. The company changed its name and is in the process of rebranding. - -### What is Sparkle? - -Substreams is the successor of [StreamingFast Sparkle](https://github.com/streamingfast/sparkle). Substreams enables greater composability, and provides similar parallelization capabilities. Sparkle is deprecated. - -### **Who is StreamingFast?** - -StreamingFast is a protocol infrastructure company providing a massively scalable architecture for streaming blockchain data. StreamingFast is one of the core developers working alongside The Graph Foundation. - -### Why the `wasm32-unknown-unknown` target? - -The first unknown is the system you are compiling on, and the second is the system you are targeting. - -“Compile on almost any machine, run on almost any machine.” - -Additional information [is available in the Github issue for WASM-bindgen](https://github.com/rustwasm/wasm-bindgen/issues/979). - -### Why does the output show "@unknown" instead of "@type" and the decoding failed only showing "@str" and "@bytes" - -Check to make sure the module's output type matches the protobuf definition. In some cases, the renamed protobuf package isn't updated in the `substreams.yaml` manifest file's `module.output.type` field, creating an incompatibility. - -### Can I retrieve Mempool data with Substreams? - -No, it is currently NOT possible to retrieve Mempool data with Substreams. diff --git a/docs/new/references/glossary.md b/docs/new/references/glossary.md deleted file mode 100644 index 40397dbc5..000000000 --- a/docs/new/references/glossary.md +++ /dev/null @@ -1,92 +0,0 @@ -# Glossary - -## Substreams -[Substreams](https://substreams.streamingfast.io/) is a powerful indexing technology, which allows you to: -1. Extract data from several blockchains (Ethereum, Polygon, BNB, Solana...). -2. Apply custom transformations to the data. -3. Send the data to a place of your choice (for example, a Postgres database or a file). - -## Firehose -[Firehose](https://firehose.streamingfast.io/) is the extraction layer of Substreams (i.e. step number one of the previous glossary entry). Although Firehose is a different project, it is tightly related to Substreams. - -## CLI -`CLI`, which stands for command-line interface, is a text-based interface that allows you to input commands to interact with a computer. -The [Substreams CLI](https://substreams.streamingfast.io/getting-started/installing-the-cli) allows you to deploy and manage your Substreams. - -## Module -[Modules](https://substreams.streamingfast.io/developers-guide/modules) are small pieces of Rust code running in a WebAssembly (WASM) virtual machine. Modules have one or more inputs and an output. -For example, a module could receive an Ethereum block as input and emit a list of transfers for that block as output. - -There are two types of modules: `map` and `store`. - -## map Module -`map` modules receive an input and emit an output (i.e. they perform a transformation). - -## store Module -`store` modules write to key-value stores and are stateful. They are useful in combination with `map` modules to keep track of past data. - -## Directed Acyclic Graph (DAG) -[DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph) are data structures used in many computational models. In Substreams, DAGs are used to define module data flows. - -A DAG is a one-direction, acyclic graph. They are used in a variety of software, such as Git or IPFS. - -## Composability -Modules make Substreams really _composable_. Being composable means that Substreams can be independent, but they can also work together to create powerful streams. - -For example, consider that you have two _map modules_: one emitting `Transfer` objects and another one emitting `AccountInformation` objects. -You could create another module that receives the previous two modules as input and merges the information from both. - -That is why Substreams is so powerful! - -## Protocol Buffers (Protobuf) -[Protocol Buffers](https://protobuf.dev/) is a serializing format used to define module inputs and outputs in Substreams. -For example, a manifest might define a module called `map_tranfers` with an input object, `Transfer` (representing an Ethereum transaction), and an output object `MyTransfer` (representing a reduced version of an Ethereum transaction). - -## Manifest -The [Substreams manifest](https://substreams.streamingfast.io/developers-guide/creating-your-manifest) (called `substreams.yaml`) is a YAML file where you define all the configurations needed. For example, the modules of your Substreams (along with their inputs and outputs), or the Protobuf definitions used. - -## WebAssembly (WASM) -[WebAssembly (WASM)](https://webassembly.org/) is a binary-code format used to run a Substreams. The Rust code used to define your Substreams transformations is packed into a WASM module, which you can use as an independent executable. - -## Block -The `Block` Protobuf object contains all the blockchain information for a specific block number. EVM-compatible chains share the same [Block](https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/type/v2/type.proto) object, but non-EVM-compatible chains must use [their corresponding Block Protobuf definition](https://substreams.streamingfast.io/reference-and-specs/chains-and-endpoints). - -
- -## SPKG (.spkg) -[SPKG files](https://substreams.streamingfast.io/reference-and-specs/packages) contain Substreams definitions. You can create an `.spkg` file from a Substreams manifest using the `substreams pack` command. Then, you can use this file to share or run the Substreams independently. -The `.spkg` file contains everything needed to run a Substreams: Rust code, Protobuf definitions and the manifest. - -## GUI -The CLI includes two commands to run a Substreams: `run` and `gui`. The `substreams run` command prints the output of the execution linearly for every block, while the `substreams gui` allows you to easily jump to the output of a specific block. - -
- -## Subgraph -[Subgraphs](https://thegraph.com/docs/en/developing/creating-a-subgraph/) are another indexing mechanism developed by The Graph. -In Subgraphs, data is indexed and available through a GraphQL endpoint. - -One of the main differences between Subgraphs and Substreams is that Subgraphs rely on _polling_, while Substreams relies on _streaming_. - -## Triggers -In Subgraphs, you define _triggers_ to index your data. These triggers are events that happen in the blockchain (for example, `AccountCreated`). Subgraphs listen for those events, and index the data accordingly. - -## Sink -Substreams allows you to extract blockchain data and apply transformations to it. After that, you should choose **a place to send your transform data, which is called _sink_**. -A sink can be a [SQL database](https://substreams.streamingfast.io/developers-guide/sink-targets/substreams-sink-sql), [a file](https://substreams.streamingfast.io/developers-guide/sink-targets/substreams-sink-files) or a [custom solution of your choice](https://substreams.streamingfast.io/developers-guide/sink-targets/custom-sink-js). - -## Deployable Unit -A deployable unit is a Substreams manifest or package (spkg) that contains all the information about how to run it from sink service. In the manifest, it corresponds to the `network` and `sink` fields. -See [Working with deployable units](https://substreams.streamingfast.io/developers-guide/sink-deployable-units) - - -## Substreams-powered Subgraph -When a Subgraph acts as a sink for your Substreams, you call it a [Substreams-powered Subgraph](https://thegraph.com/docs/en/cookbook/substreams-powered-subgraphs/). - -The Subgraph Sink is one of the _official sinks_ supported by Substreams, and can help you index your Subgraph way faster! - -## Parallel execution -[Parallel execution](https://substreams.streamingfast.io/developers-guide/parallel-execution) is the process of a Substreams module's code executing multiple segments of blockchain data simultaneously in a forward or backward direction. - -## Workers -Workers are the fundamental unit of parallelizing in Substreams. Workers are computer processes that run in parallel to speed up the Substreams computations. diff --git a/docs/new/references/gui.md b/docs/new/references/gui.md deleted file mode 100644 index 7aa26e075..000000000 --- a/docs/new/references/gui.md +++ /dev/null @@ -1,90 +0,0 @@ -# Using the Substreams GUI - -When running Substreams through the CLI, you can use two different commands: `substreams run` and `substreams gui`. The `substreams run` command prints the output of the execution linearly for every block: - -
- -However, this is not a useful approach when dealing with complex Substreams (i.e. with several modules and many blocks). The `substreams gui` command allows you to easily see the progress of a Substreams, move across blocks or search within the output. - -
- -The Substreams GUI is a command-line utility, so you use the keys in your keyboard to interact with it. - -## Cheatsheet - -| Function | Keys | -|---|---| -| Switch screen (`Request`, `Progress`, `Output`) | `tab` | -| Restart | `r` | -| Quit | `q` | -| Navigate Blocks - Forward | `p` | -| Navigate Blocks - Backwards | `o` | -| Navigate Blocks - Go To | `=` + *block number* + `enter` | -| Navigate Modules - Forward | `i` | -| Navigate Modules - Backwards | `u` | -| Search | `/` + *text* + `enter` | -| Commnads information | `?` | - -## Launching the GUI - -In order to showcase the different options of the GUI, the [Ethereum Block Meta Substreams](https://github.com/streamingfast/substreams-eth-block-meta/) will be used as an example. -By running the following command, you are executing the `kv_out` module, which retrieves outputs the data in a key-value format. - -```bash -substreams gui -e mainnet.eth.streamingfast.io:443 https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.5.1/substreams-eth-block-meta-v0.5.1.spkg kv_out --start-block 17712038 --stop-block +100 -``` - -In your command-line terminal, you should see something like: - -
- -The `Progress` screen provides information about the Substreams execution, such as its status or the payload received. Once all the blocks have been consumed, the status is `Stream ended`. -There are two other main screens in the Substreams GUI: `Request` and `Output`. You can move to a different screen by using the `tab` key: - -
- -You can restart the stream by pressing the `s` key. - -
- -To quit the GUI, press the `q` key. - -## The Output Screen - -If you are in the `Progress` screen, press `tab` in your keyboard to move to the `Output` screen. In this screen, you can see the Protobuf output for every block. The image below shows the output for the block number `17712038` (the starting block). - -
- -### Navigating Through Blocks - -You can see the output for other blocks by using the `o` and `p` keys. -The `o` key takes you to the following block, and the `p` takes you to the previous block. - -
- - -If you want to jump to a specific block, you can press the `=` key and specify the block number. Then, just press `enter`. - -
- - -### Navigating Through Modules - -You can see the output of a different module by using the `u` and `i` keys. In the following example, you go from the `kv_out` module (the module specified in the CLI command), to the `store_block_meta_end` module. - -
- - -### Searching in the Output - -To search for a speciifc text in the output: - -1. Press the `/` key. -2. Introduce the text. -3. Press `enter`. - -In the following example, your search for the `SET` text. - -
- - diff --git a/docs/new/references/manifests.md b/docs/new/references/manifests.md deleted file mode 100644 index 6e341d8b4..000000000 --- a/docs/new/references/manifests.md +++ /dev/null @@ -1,406 +0,0 @@ ---- -description: StreamingFast Substreams manifests reference ---- - -# Manifests - -{% hint style="success" %} -**Tip**: When writing and checking your `substreams.yaml` file, it may help to check your manifest against our [JSON schema](https://json-schema.org/) to ensure there are no problems. JSON schemas can be used in [Jetbrains](https://www.jetbrains.com/help/idea/json.html#ws\_json\_schema\_add\_custom) and [VSCode](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml). Our manifest schema can be seen [here](../../schemas/manifest-schema.json). -{% endhint %} - -## Manifests overview - -The manifest is the high-level outline for a Substreams module. The manifest file is used for defining properties specific to the current Substreams module and identifying the dependencies between the `inputs` and `outputs` of modules. - -This reference documentation **provides a guide for all fields and values** used in a Substreams manifest. - -### `specVersion` - -Excerpt pulled from the example Substreams manifest. - -{% code title="manifest excerpt" %} -```yaml -specVersion: v0.1.0 -``` -{% endcode %} - -Use `v0.1.0` for the `specVersion` field. - -### `package` - -Excerpt pulled from the example Substreams manifest. - -{% code title="manifest excerpt" overflow="wrap" %} -```yaml -package: - name: module_name_for_project - version: v0.5.0 - doc: | - Documentation heading for the package. - - More detailed documentation for the package. -``` -{% endcode %} - -#### `package.name` - -The `package.name` field is used to identify the package. - -The `package.name` field infers the filename when the [`pack`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#pack) command is run by using `substreams.yaml` as a flag for the Substreams package. - -The content of the `name` field must match the regular expression: `^([a-zA-Z][a-zA-Z0-9_]{0,63})$`. For consistency, use the `snake_case` naming convention. - -The regular expression ruleset translates to the following: - -* 64 characters maximum -* Separate words by using `_` -* Starts by using `a-z` or `A-Z` and can contain numbers thereafter - -#### `package.version` - -The `package.version` field identifies the package for the Substreams module. - -{% hint style="info" %} -**Note**: The`package.version` **must respect** [Semantic Versioning, version 2.0](https://semver.org/) -{% endhint %} - -#### package.url - -The `package.url` field identifies and helps users discover the source of the Substreams package. - -#### package.doc - -The `package.doc` field is the documentation string of the package. The first line is used by the different UIs as a short-form description. - -This field should be written in Markdown format. - -### `imports` - -The `imports` section references WASM code, compiled protobuf definitions, and module definitions. - -{% hint style="success" %} -**Tip**: Imported modules can be referred to later in the `modules` section of the manifest through the use of a key. -{% endhint %} - -Example: - -```yaml -imports: - ethereum: substreams-ethereum-v1.0.0.spkg - tokens: ../eth-token/substreams.yaml - prices: ../eth-token/substreams.yaml -``` - -The **value is a pointer** to a Substreams manifest or a Substreams [package](packages.md). - -The filename can be absolute or relative or a remote path prefixed by `http://` or `https://`. - -Imports differ across different blockchains. For example, Ethereum-based Substreams modules reference the matching `spkg` file created for the Ethereum blockchain. Solana, and other blockchains, reference a different `spkg` or resources specific to the chosen chain. - -### `protobuf` - -The `protobuf` section points to the Google Protocol Buffer (protobuf) definitions used by the Rust modules in the Substreams module. - -```yaml -protobuf: - files: - - google/protobuf/timestamp.proto - - pcs/v1/pcs.proto - - pcs/v1/database.proto - importPaths: - - ./proto - - ../../external-proto -``` - -The Substreams packager loads files in any of the listed `importPaths`. - -{% hint style="info" %} -**Note**: The `imports` section of the manifest also affects which `.proto` files are used in the final Substreams package. -{% endhint %} - -Protobufs and modules are packaged together to help Substreams clients decode the incoming streams. Protobufs are not sent to the Substreams server in network requests. - -[Learn more about Google Protocol Buffers](https://protobuf.dev/) in the official documentation provided by Google. - -### `binaries` - -The `binaries` field specifies the WASM binary code to use when executing modules. - -The `modules[].binary` field uses a default value of `default`. - -```yaml -binaries: - default: - type: wasm/rust-v1 - file: ./target/wasm32-unknown-unknown/release/my_package.wasm - other: - type: wasm/rust-v1 - file: ./snapshot_of_my_package.wasm -``` - -{% hint style="warning" %} -**Important**_:_ Defining the `default` binary is required when creating a Substreams manifest. -{% endhint %} - -See the [`binary`](manifests.md#module-binary) field under `modules` to see its use. - -#### `binaries[name].type` - -The type of code and implied virtual machine for execution. There is **only one virtual machine available** that uses a value of: **`wasm/rust-v1`**. - -#### `binaries[name].file` - -The `binaries[name].file` field references a locally compiled [WASM module](https://webassembly.github.io/spec/core/syntax/modules.html). Paths for the `binaries[name].file` field are absolute or relative to the manifest's directory. The **standard location** of the compiled WASM module is the **root directory** of the Substreams module. - -{% hint style="success" %} -**Tip**: The WASM file referenced by the `binary` field is picked up and packaged into an `.spkg` when invoking the [`pack`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#pack) and [`run`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#run) commands through the [`substreams` CLI](command-line-interface.md). -{% endhint %} - -### `deriveFrom` -It is possible to override an existing substreams by pointing to an override file in the `run` or `gui` command. This override manifest will have a `deriveFrom` field which points to the original Substreams which is to be overriden. This is useful to port a substreams to one network to another. Example of an override manifest: - -``` -deriveFrom: path/to/mainnet-substreams.spkg # This can also be a remote url - -package: - name: "polygon-substreams" - version: "100.0.0" - -network: polygon - -initialBlocks: - module1: 17500000 -params: - module1: "address=2a75ca72679cf1299936d6104d825c9654489058" -``` - -### sink - -The `sink` field specifies the sink you want to use to consume your data (for example, a database or a subgraph). - -#### Sink `module` -Specifies the name of the module that emits the data to the sink. For example, `db_out` or `graph_out`. - -#### Sink `type` -Specifies the service used to consume the data. For example, `sf.substreams.sink.subgraph.v1.Service` for subgraphs, or `sf.substreams.sink.sql.v1.Service` for databases. - -#### Sink `config` -Specifies the configuration specific to every sink. This field is different for every sink. - -##### Database Config - -``` -sink: - module: db_out - type: sf.substreams.sink.sql.v1.Service - config: - schema: "./schema.sql" - engine: clickhouse - postgraphile_frontend: - enabled: false - pgweb_frontend: - enabled: false - dbt_config: - enabled: true - files: "./path/to/folder" - run_interval_seconds: 300 -``` - -- `schema`: SQL file specifying the schema. -- `engine`: `postgres` or `clickhouse`. -- `postgraphile_frontend.enabled`: enables or disables the Postgraphile portal. -- `pgweb_frontend.enabled`: enables or disables the PGWeb portal. -- `dbt_config`: specifies the configuration of dbt engine. - - `enabled`: enables or disabled the dbt engine. - - `files`: path to the dbt models. - - `run_interval_seconds`: execution intervals in seconds. - -##### Subgraph Config - -``` -sink: - module: graph_out - type: sf.substreams.sink.subgraph.v1.Service - config: - schema: "./schema.graphql" - subgraph_yaml: "./subgraph.yaml" -``` - -- `schema`: path to the GraphQL schema. -- `subgraph_yaml`: path to the Subgraph manifest. - - -### `modules` - -This example shows one map module, named `events_extractor` and one store module, named `totals` : - -{% code title="substreams.yaml" %} -```yaml - - name: events_extractor - kind: map - initialBlock: 5000000 - binary: default # Implicit - inputs: - - source: sf.ethereum.type.v2.Block - - store: myimport:prices - output: - type: proto:my.types.v1.Events - doc: - This module extracts events - - Use in such and such situations - - - name: totals - kind: store - updatePolicy: add - valueType: int64 - inputs: - - source: sf.ethereum.type.v2.Block - - map: events_extractor -``` -{% endcode %} - -#### Module `name` - -The identifier for the module, prefixed by a letter, followed by a maximum of 64 characters of `[a-zA-Z0-9_]`. The [same rules applied to the `package.name`](manifests.md#package.name) field applies to the module `name`, including the convention to use `snake_case` names. - -The module `name` is the reference identifier used on the command line for the `substreams` [`run`](https://substreams.streamingfast.io/reference-and-specs/command-line-interface#run) command. The module `name` is also used in the [`inputs`](manifests.md#modules-.inputs) defined in the Substreams manifest. - -The module `name` also corresponds to the **name of the Rust function** invoked on the compiled WASM code upon execution. The module `name` is the same `#[substreams::handlers::map]` as defined in the Rust code. Maps and stores both work in the same fashion. - -{% hint style="warning" %} -**Important**_:_ When importing another package, all module names are prefixed by the package's name and a colon. Prefixing ensures there are no name clashes across multiple imported packages and almost any name can be safely used for a module `name`. -{% endhint %} - -#### Module `initialBlock` - -The initial block for the module is where Substreams begins processing data for a module. The runtime never processes blocks prior to the one for any given module. - -If all the inputs have the same `initialBlock`, the field can be omitted and its value is inferred by its dependent [`inputs`](manifests.md#modules-.inputs). - -`initialBlock` becomes **mandatory** **when inputs have different values**. - -#### Module `kind` - -There are two module types for `modules[].kind`: - -* `map` -* `store` - -#### Module `updatePolicy` - -Specifies the merge strategy for two contiguous partial stores produced by parallelized operations. - -The values for `modules[].updatePolicy` are defined using specific rules stating: - -* `set`, the last key wins the merge strategy -* `set_if_not_exists`, the first key wins the merge strategy -* `append`, concatenates two keys' values -* `add`, sum the two keys' values -* `min`, min between two keys' values -* `max`, max between two keys' values - -#### Module `valueType` - -{% hint style="success" %} -Tip: The module `updatePolicy` field is only available for modules of `kind: store`. -{% endhint %} - -Specifies the data type of all keys in the `store`, and determines what WASM imports are available to the module and are able to write to the `store`. - -The values for `modules[].valueTypes` can use various types including: - -* `bigfloat` -* `bigint` -* `int64` -* `bytes` -* `string` -* `proto:path.to.custom.protobuf.Model` - -{% hint style="success" %} -Tip: The module `valueType` field is only available for modules of `kind: store`. -{% endhint %} - -#### Module `binary` - -An identifier referring to the [`binaries`](manifests.md#binaries) section of the Substreams manifest. - -The `modules[].binary` field overrides which binary is used from the `binaries` declaration section. This means multiple WASM files can be bundled in the Package. - -```yaml -modules: - name: hello - binary: other - ... -``` - -The default value for `binary` is `default`. Therefore, a `default` binary must be defined under [`binaries`](manifests.md#binaries). - -#### Module `inputs` - -{% code title="substreams.yaml" %} -```yaml -inputs: - - params: string - - source: sf.ethereum.type.v2.Block - - store: my_store - mode: deltas - - store: my_store # defaults to mode: get - - map: my_map -``` -{% endcode %} - -The `inputs` field is a **list of input structures**. One of three keys is required for every object. - -The key types for `inputs` include: - -* `source` -* `store,` used to define `mode` keys -* `map` -* `params` - -You can find more details about inputs in the [Developer Guide's section about Modules](../developers-guide/modules/types.md). - -#### Module `output` - -{% code title="substreams.yaml" %} -```yaml -output: - type: proto:eth.erc721.v1.Transfers -``` -{% endcode %} - -The value for `type` is always prefixed using `proto:` followed by a definition specified in the protobuf definitions, and referenced in the `protobuf` section of the Substreams manifest. - -{% hint style="success" %} -**Tip**: The module `output` field is only available for modules of `kind: map`. -{% endhint %} - -#### Module `doc` - -This field should contain Markdown documentation of the module. Use it to describe how to use the params, or what to expect from the module. - -### `params` - -The `params` mapping changes the default values for modules' parameterizable inputs. - -```yaml -modules: - ... -params: - module_name: "default value" - "imported:module": "overridden value" -``` - -You can override those values with the `-p` parameter of `substreams run`. - -When rolling out your consuming code -- in this example, Python -- you can use something like: - -{% code overflow="wrap" %} -```python -my_mod = [mod for mod in pkg.modules.modules if mod.name == "store_pools"][0] -my_mod.inputs[0].params.value = "myvalue" -``` -{% endcode %} - -which would be inserted just before starting the stream. diff --git a/docs/new/tutorials/ethereum/exploring-ethereum/map_block_meta_module.md b/docs/new/tutorials/ethereum/exploring-ethereum/map_block_meta_module.md deleted file mode 100644 index ab6e8133b..000000000 --- a/docs/new/tutorials/ethereum/exploring-ethereum/map_block_meta_module.md +++ /dev/null @@ -1,130 +0,0 @@ -## Mapping Blocks - -This module takes a raw Ethereum block and returns a reduced version of a block, with just three pieces of information: hash, parent hash, and block number. - -Let's run the Substreams first, and then go through the code. - -### Running the Substreams - -Running a Substreams usually requires three steps: generating the Rust Protobufs, building the WASM container, and using the Substream CLI to start the streaming. Make sure to run the following commands in the `substreams-explorer/ethereum-explorer` folder: - -1. **Generate the Protobuf objects:** The `.proto` files define a data model regardless of any programming language. However, in order to use this model in your Rust application, you must generate the corresponding Rust data structures. - -```bash -make protogen -``` - -2. **Build the WASM module:** The following command generates a WASM container from the Rust application, which you can find at `/target/wasm32-unknown-unknown/release/substreams.wasm`. Note that this is the same path provided in the Substreams manifest (`substreams.yml`). - -```bash -make build -``` - -3. **Streaming data through the CLI:** The following command streams the Ethereum blockchain data, and applies the transformations contained in the `map_block_meta` module to every block. - -```bash -$ substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_block_meta --start-block 17712040 --stop-block +1 -``` - -Let's break down the command into pieces: - -- `mainnet.eth.streamingfast.io:443`: is the StreamingFast Ethereum Mainnet endpoint where you are sending your Substreams for execution. -- `substreams.yaml`: specifies the Substreams manifest. -- `map_block_meta`: specifies the module to execute. Since the Ethereum Explorer application contains several modules, it is necessary to specify which one you want to execute. -- `--start-block 17712040`: specifies the starting block (i.e. the block where Substreams will start streaming). -- `--stop-block +1`: specifies how many blocks after the starting block should be considered. In this example, `+1` means that the streaming will start at `17712040` and finish at `17712041` (just one block). - -The output of the command should be similar to: - -```bash -...output omitted... - ------------ BLOCK #17,712,040 (31ad07fed936990d3c75314589b15cbdec91e4cc53a984a43de622b314c38d0b) --------------- -{ - "@module": "map_block_meta", - "@block": 17712040, - "@type": "eth.block_meta.v1.BlockMeta", - "@data": { - "number": "17712040", - "hash": "31ad07fed936990d3c75314589b15cbdec91e4cc53a984a43de622b314c38d0b", - "parentHash": "1385f853d28b16ad7ebc5d51b6f2ef6d43df4b57bd4c6fe4ef8ccb6f266d8b91" - } -} - -all done -``` - -As you can see, the output is formatted as JSON, and the `@data` field contains the actual output Protobuf of the module (`BlockMeta`). - -The `BlockMeta` definition: - -```protobuf -syntax = "proto3"; - -package eth.block_meta.v1; - -message BlockMeta { - uint64 number = 1; - string hash = 2; - string parent_hash = 3; -} -``` - -The JSON output: - -```json -"@data": { - "number": "17712040", - "hash": "31ad07fed936990d3c75314589b15cbdec91e4cc53a984a43de622b314c38d0b", - "parentHash": "1385f853d28b16ad7ebc5d51b6f2ef6d43df4b57bd4c6fe4ef8ccb6f266d8b91" -} -``` - -### Inspecting the Code - -Although the code (which is in the `map_block_meta.rs` file) for this module is pretty straightforward to understand, let's discuss its main parts. - -Declaration of the module in the manifest (`substreams.yml`): - -```yaml -modules: - - name: map_block_meta - kind: map - inputs: - - source: sf.ethereum.type.v2.Block - output: - type: proto:eth.block_meta.v1.BlockMeta -``` - -Code of the module: - -```rust -#[substreams::handlers::map] -fn map_block_meta(blk: Block) -> Result { - let header = blk.header.as_ref().unwrap(); - - Ok(BlockMeta { - number: blk.number, - hash: Hex::encode(&blk.hash), - parent_hash: Hex::encode(&header.parent_hash), - }) -} -``` - -The `#[substreams::handlers::map]` attribute annotates the `map_block_meta` function as a Substreams mapper. The name of the function must match the name of the module in the Substreams manifest. The input of the function is a raw Ethereum block (`pb::eth::v2::Block`). - -In order to get the block metadata, you use the `header` property. - -```rust -let header = blk.header.as_ref().unwrap(); -``` - -Then, you simply create a `BlockMeta` struct and return it. - -```rust -Ok(BlockMeta { - number: blk.number, - hash: Hex::encode(&blk.hash), - parent_hash: Hex::encode(&header.parent_hash), -}) -``` diff --git a/docs/new/tutorials/ethereum/exploring-ethereum/map_contract_events_module.md b/docs/new/tutorials/ethereum/exploring-ethereum/map_contract_events_module.md deleted file mode 100644 index 73d6bb65d..000000000 --- a/docs/new/tutorials/ethereum/exploring-ethereum/map_contract_events_module.md +++ /dev/null @@ -1,123 +0,0 @@ -## Retrieving Events of a Smart Contract - -Given a smart contract address passed as a parameter, this module returns the logs attached to the contract. - -### Running the Substreams - -First, generate the Protobuf modules and build the Rust code: - -```bash -make protogen -``` - -```bash -make build -``` - -Now, you can run the Substreams. The logs retrieved correspond to the `0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d` (BoredApeYachtClub smart contract). -To avoid iterating over the full blockchain, the following command starts at block `17717995` and finished at block `17718004`. Therefore, only the BoredApeYachtClub smart contract logs that happened within this block range are printed. - -```bash -substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_contract_events --start-block 17717995 --stop-block +10 -``` - -The output of the command should be similar to: - -```bash -...output omitted... - ------------ BLOCK #17,717,995 (bfecb26963a2cd77700754612185e0074fc9589d2d73abb90e362fe9e7969451) --------------- ------------ BLOCK #17,717,996 (7bf431a4f9df67e1d7e385d9a6cba41c658e66a77f0eb926163a7bbf6619ce20) --------------- ------------ BLOCK #17,717,997 (fa5a57231348f1f138cb71207f0cdcc4a0a267e2688aa63ebff14265b8dae275) --------------- -{ - "@module": "map_contract_events", - "@block": 17717997, - "@type": "eth.event.v1.Events", - "@data": { - "events": [ - { - "address": "bc4ca0eda7647a8ab7c2061c2e118a18a936f13d", - "topics": [ - "8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925", - "000000000000000000000000e2a83b15fc300d8457eb9e176f98d92a8ff40a49", - "0000000000000000000000000000000000000000000000000000000000000000", - "00000000000000000000000000000000000000000000000000000000000026a7" - ], - "txHash": "f18291982e955f3c2112de58c1d0a08b79449fb473e58b173de7e0e189d34939" - }, - { - "address": "bc4ca0eda7647a8ab7c2061c2e118a18a936f13d", - "topics": [ - "ddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", - "000000000000000000000000e2a83b15fc300d8457eb9e176f98d92a8ff40a49", - "000000000000000000000000c67db0df922238979da0fd00d46016e8ae14cecb", - "00000000000000000000000000000000000000000000000000000000000026a7" - ], - "txHash": "f18291982e955f3c2112de58c1d0a08b79449fb473e58b173de7e0e189d34939" - } - ] - } -} - ------------ BLOCK #17,717,998 (372ff635821a434c81759b3b23e8dac59393fc27a7ebb88b561c1e5da3c4643a) --------------- ------------ BLOCK #17,717,999 (43f0878e119836cc789ecaf12c3280b82dc49567600cc44f6a042149e2a03779) --------------- ------------ BLOCK #17,718,000 (439efaf9cc0059890a09d34b4cb5a3fe4b61e8ef96ee67673c060d58ff951d4f) --------------- ------------ BLOCK #17,718,001 (c97ca5fd26db28128b0ec2483645348bbfe998e9a6e19e3a442221198254c9ea) --------------- ------------ BLOCK #17,718,002 (9398569e46a954378b16e0e7ce95e49d0f21e6119ed0e3ab84f1c91f16c0c30e) --------------- ------------ BLOCK #17,718,003 (80bcd4c1131c35a413c32903ffa52a14f8c8fe712492a8f6a0feddbb03b10bba) --------------- ------------ BLOCK #17,718,004 (d27309ac29fe47f09fa4987a318818c325403863a53eec6a3676c2c2f8c069d9) --------------- -all done -``` - -The smart contract address is passed as a parameter defined in the Substreams manifest (`substreams.yml`): - -```yml -params: - map_contract_events: "0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d" -``` - -### Inspecting the Code - -Declaration of the module in the manifest (`substreams.yml`): - -```yml -- name: map_contract_events - kind: map - inputs: - - params: string - - source: sf.ethereum.type.v2.Block - output: - type: proto:eth.event.v1.Events -``` - -The module expects two inputs: the parameter as a string, and a raw Ethereum block. -The output is the `Events` object defined in the Protobuf. - -The corresponding Rust function declaration, which matches the name of the module, `map_contract_events`: - -```rust -fn map_contract_events(contract_address: String, blk: Block) -> Result { - verify_parameter(&contract_address)?; // Verify address - - let events: Vec = blk - .logs() // 1. - .filter(|log| log.address().to_vec() == Hex::decode(&contract_address).expect("already validated")) // 2. - .map(|log| Event { // 3. - address: Hex::encode(log.address()), - topics: log.topics().into_iter().map(Hex::encode).collect(), - tx_hash: Hex::encode(&log.receipt.transaction.hash), - }) - .collect(); // 4. - - Ok(Events { events }) -} -``` - -In this example, you do not need to parse the parameters, as `contract_address` is the only string passed and you can use it directly. -However, it is necessary to verify that the parameter is a valid Ethereum; this verification is performed by the `verify_parameter` function. - -Then, you iterate over the events of the contract: -1. The `.logs()` function iterates over the logs of successful transactions within the block. -2. For every log of a successful transaction, you verify if its `address` matches the smart contract address (i.e. you verify if the log was actually emitted by the smart contract). For the comparison, both `log.address()` and `contract_address` are converted to `Vec`. -3. Every filtered log (i.e. every log that belongs to the smart contract) is mapped to a `pb::eth::event::v1::Event` struct, which was specified in the Protobuf definition. -4. Finally, you collect all the events in a vector. \ No newline at end of file diff --git a/docs/new/tutorials/ethereum/exploring-ethereum/map_filter_transactions_module.md b/docs/new/tutorials/ethereum/exploring-ethereum/map_filter_transactions_module.md deleted file mode 100644 index 69e00a40c..000000000 --- a/docs/new/tutorials/ethereum/exploring-ethereum/map_filter_transactions_module.md +++ /dev/null @@ -1,221 +0,0 @@ -## Filtering Transactions - -This module iterates over all the blockchain transactions and filters them by some of their fields (the `from` and `to` fields). -For example, if you want to retrieve all the transactions initiated by the address `0xb6692f7ae54e89da0269c1bfd685ccdfd41d2bf7`, you set the filter `from = 0xb6692f7ae54e89da0269c1bfd685ccdfd41d2bf7`. - -### Running the Substreams - -First, generate the Protobuf modules and build the Rust code: - -```bash -make protogen -``` - -```bash -make build -``` - -Now, you can run the Substreams: - -```bash -substreams run -e mainnet.eth.streamingfast.io:443 substreams.yaml map_filter_transactions --start-block 17712038 --stop-block +3 -``` - -The output of the command should be similar to: - -```bash -...output omitted... - ------------ BLOCK #17,712,038 (b96fc7e71c0daf69b19211c45fbb5c201f4356fb2b5607500b7d88d298599f5b) --------------- -{ - "@module": "map_filter_transactions", - "@block": 17712038, - "@type": "eth.transaction.v1.Transactions", - "@data": { - "transactions": [ - { - "from": "b6692f7ae54e89da0269c1bfd685ccdfd41d2bf7", - "to": "dac17f958d2ee523a2206206994597c13d831ec7", - "hash": "933b74565234ac9ca8389f7a49fad80099abf1be77e4bef5af69ade30127f30e" - }, - -...output omitted... - - { - "from": "4c8e30406f5dbedfaa18cb6b9d0484cd5390490a", - "to": "dac17f958d2ee523a2206206994597c13d831ec7", - "hash": "558031630b43c8c61e36d742a779f967f3f0102fa290111f6f6f9c2acaadf3ea" - } - ] - } -} - ------------ BLOCK #17,712,039 (1385f853d28b16ad7ebc5d51b6f2ef6d43df4b57bd4c6fe4ef8ccb6f266d8b91) --------------- -{ - "@module": "map_filter_transactions", - "@block": 17712039, - "@type": "eth.transaction.v1.Transactions", - "@data": { - "transactions": [ - { - "from": "75e89d5979e4f6fba9f97c104c2f0afb3f1dcb88", - "to": "dac17f958d2ee523a2206206994597c13d831ec7", - "hash": "43e0e1b6315c4cc1608d876f98c9bbf09f2a25404aabaeac045b5cc852df0e85" - }, - -...output omitted... - - { - "from": "e41febca31f997718d2ddf6b21b9710c5c7a3425", - "to": "dac17f958d2ee523a2206206994597c13d831ec7", - "hash": "45c03fcbefcce9920806dcd7d638cef262ad405f8beae383fbc2695ad4bc9b1b" - } - ] - } -} - ------------ BLOCK #17,712,040 (31ad07fed936990d3c75314589b15cbdec91e4cc53a984a43de622b314c38d0b) --------------- -{ - "@module": "map_filter_transactions", - "@block": 17712040, - "@type": "eth.transaction.v1.Transactions", - "@data": { - "transactions": [ - { - "from": "48c04ed5691981c42154c6167398f95e8f38a7ff", - "to": "dac17f958d2ee523a2206206994597c13d831ec7", - "hash": "137799eea9fa8ae410c913e16ebc5cc8a01352a638f3ce6f3f29a283ad918987" - }, - -...output omitted... - - { - "from": "f89d7b9c864f589bbf53a82105107622b35eaa40", - "to": "dac17f958d2ee523a2206206994597c13d831ec7", - "hash": "0544143b459969c9ed36741533fba70d6ea7069f156d2019d5362c06bf8d887f" - } - ] - } -} - -all done -``` - -In the previous command, you are filtering all the transactions from blocks `17712038` to `17712040`, where `to = 0xdac17f958d2ee523a2206206994597c13d831ec7` (the USDT smart contract address). The filters are specified in the `params` section of the Substreams manifest (`substreams.yml`): - -```yml -map_filter_transactions: "to=0xdAC17F958D2ee523a2206206994597C13D831ec7" -``` - -### Applying Filters - -The filters are specified as a query-encoded string (`param1=value1¶m2=value2¶m3=value3`). In this example, only two parameters are supported, `from` and `to`, which you can use to create filters, such as: - -```yml -map_filter_transactions: "from=0x89e51fa8ca5d66cd220baed62ed01e8951aa7c40&to=0xdAC17F958D2ee523a2206206994597C13D831ec7" -``` - -Retrieve all transactions where `from=0x89e51fa8ca5d66cd220baed62ed01e8951aa7c40` and `to=0xdAC17F958D2ee523a2206206994597C13D831ec7`. - -```yml -map_filter_transactions: "from=0x89e51fa8ca5d66cd220baed62ed01e8951aa7c40" -``` - -Retrieve all transactions where `from=0x89e51fa8ca5d66cd220baed62ed01e8951aa7c40`. - -```yml -map_filter_transactions: "" -``` - -Retrieve all transactions. Without applying any filter. - -### Inspecting the Code - -Declaration of the module in the manifest (`substreams.yml`): - -```yml -- name: map_filter_transactions - kind: map - inputs: - - params: string - - source: sf.ethereum.type.v2.Block - output: - type: proto:eth.transaction.v1.Transactions -``` - -The module expects two inputs: the parameters string, which contains the filters, plus a raw Ethereum block. -The output is the `Transactions` object declared in the Protobuf. - -Now, let's take a look at the actual Rust code: - -```rust -#[derive(Deserialize)] -struct TransactionFilterParams { - to: Option, - from: Option, -} - -#[substreams::handlers::map] -fn map_filter_transactions(params: String, blk: Block) -> Result> { - let filters = parse_filters_from_params(params)?; - - let transactions: Vec = blk - .transactions() - .filter(|trans| apply_filter(&trans, &filters)) - .map(|trans| Transaction { - from: Hex::encode(&trans.from), - to: Hex::encode(&trans.to), - hash: Hex::encode(&trans.hash), - }) - .collect(); - - Ok(Transactions { transactions }) -} -``` - -The function name, `map_filter_transactions` matches the name given in the Substreams manifest. Two parameters are passed: `params: String, blk: Block`. -For Substreams, the parameter specified in the manifest is a simple String. The query-encoded format is just an abstraction that you must parse. -The `parse_filters_from_params` parses the string and creates a `TransactionFilterParams` struct. - -```rust -let filters = parse_filters_from_params(params)?; -``` - -```rust -fn parse_filters_from_params(params: String) -> Result> { - let parsed_result = serde_qs::from_str(¶ms); - if parsed_result.is_err() { - return Err(Vec::from([anyhow!("Unexpected error while parsing parameters")])); - } - - let filters = parsed_result.unwrap(); - verify_filters(&filters)?; - - Ok(filters) -} -``` - -The `serde_qs::from_str(¶ms)` from the [Serde QS Rust library](https://docs.rs/serde_qs/latest/serde_qs/) parses the parameters and returns the filters struct. Then, you call the `verify_filters(&filters)?` function, which ensures that the filters provided are valid Ethereum addresses. -If there are errors while parsing the parameters, they are collected in a `substreams::errors::Error` vector and returned. - -Back in the main function, if the parameters parsing is correct, you start filtering the transactions: - -```rust - let filters = parse_filters_from_params(params)?; - - // At this point, the filters are correct. If not, a Vec object is returned. - let transactions: Vec = blk - .transactions() // 1. - .filter(|trans| apply_filter(&trans, &filters)) // 2. - .map(|trans| Transaction { // 3. - from: Hex::encode(&trans.from), - to: Hex::encode(&trans.to), - hash: Hex::encode(&trans.hash), - }) - .collect(); // 4. -``` -1. The `transactions()` method iterates over all the **successful** transactions of the block. -2. Then, for every successful transaction, the previously parsed filters are applied. -3. Every transaction that complies with the filters provided is mapped into a `pb::eth::transaction::v1::Transaction` struct. -This struct is part of the Protobuf declarations and is part of the output of the Substreams module. -4. Finally, all the transactions are collected into a vector of type `pb::eth::transaction::v1::Transaction`. \ No newline at end of file diff --git a/docs/new/tutorials/ethereum/exploring-ethereum/overview.md b/docs/new/tutorials/ethereum/exploring-ethereum/overview.md deleted file mode 100644 index fd583bdfe..000000000 --- a/docs/new/tutorials/ethereum/exploring-ethereum/overview.md +++ /dev/null @@ -1,79 +0,0 @@ -# Substreams Explorers - -Getting started with Substreams might feel challenging, but you are not alone! The Substreams Explorers are a set of projects, modules, and code samples that allow you to _explore_ and discover the main features of Substreams. - -As you already know, Substreams supports streaming from several blockchains, and every blockchain holds a different data model. This is why the data structures used change from one blockchain to the other. For example, an Ethereum block is represented by the [`sf.ethereum.type.v2.Block`](https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/type/v2/type.proto) Rust struct, while a Near block is represented by [`sf.near.type.v1.Block`](https://github.com/streamingfast/firehose-near/blob/develop/proto/sf/near/type/v1/type.proto). - -## Before You Begin - -Before you start coding, there are several dependencies you must install on your computer. - -### The GitHub Repository -The `https://github.com/streamingfast/substreams-explorers` GitHub repository contains all the Substreams Explorers currently available. You can simply clone the repository: - -``` -$ git clone https://github.com/streamingfast/substreams-explorers -``` - -### The Substreams CLI - -The Substreams CLI allows you to run, package, and visualize your Substreams. Make sure you have the CLI installed by following this [simple tutorial](/getting-started/installing-the-cli.md). - -### Substreams Basics - -You should be familiar with the basic Substreams terminology, which includes: -- Modules (understanding the difference between a `map` and a `store` module) -- Protobufs (understanding what they are) - - -## Ethereum Explorer - -The Ethereum Explorer consists of several Substreams modules showcasing the most basic operations that you can perform with Substreams on the Ethereum blockchain. - -You can find the Ethereum Explorer at [https://github.com/streamingfast/substreams-explorers](https://github.com/streamingfast/substreams-explorers) - -### Modules - -The modules in this repository answer some interesting questions when developing a blockchain application: - -#### How Can You Get the Basic Information of a Block? - -For every block, the `map_block_meta` module retrieves the most relevant information of the block (number, hash, and parent hash). - -#### How Can You Retrieve Transactions By Their From or To Fields? - -Given any combination of two parameters (`from` and `to`), the `map_filter_transactions` filters a transaction among all transactions in the blockchain. This involves: - -1. Providing the filters (only the `from` fields, only the `to` field, both `from` and `to` fields, or none) -1. Iterating over all the transactions. -2. Filtering the transactions, according to the parameters provided. For example, `from == tx_from`, `from == tx_from and to == tx_to`. - -#### How Can You Retrieve All the Events For a Specific Smart Contract? - -Given a smart contract address parameter (`contract_address`), the `map_contract_events` module retrieves all the events related to a specific smart contract. This involves: - -1. Iterating over all the logs of a block. -2. Filtering the log, where the `address` field is equal to the smart contract address parameter (`address == contract_address`). - -In the following sections, you will go through every module, run the corresponding Substreams, and understand every piece of code. Let's go! - -### The Project Structure - -

Ethereum Explorer Project Structure

- -1. The `proto` folder contains the Protobuf definitions for the transformations. -In this example, there are three Protobuf objects, which are the outputs of the Substreams module mentioned in the previous section: BlockMeta (which represents the information of an Ethereum block), Transaction (which is an abstraction for an Ethereum transaction), and Event (an abstraction for an Ethereum event). -2. The `src` folder contains the source code of the Substreams transformations. Every module has its corresponding Rust file. -3. The `substreams.yml` is the Substreams manifest, which defines relevant information, such as the inputs/outputs of every module or the Protobuf files. - -#### The Substreams Manifest - -Let's take a closer look at the Substreams manifest (`substreams.yml`): - -

Ethereum Explorer Manifest

- -1. The `protobuf` section specifies the location of the Protobuf files used in the Substreams (i.e. where are the files defining the objects that you are going to use as output). In this example, the files are under the `proto` folder. -2. When you run Substreams, you are really executing a Rust application inside a WASM container. Therefore, Substreams needs to know where is the WASM executable. The `binaries` section specifies the location of the WASM executable. -3. Every module must be defined in the manifest, along with its `kind`, `inputs` and `outputs`. -In this example, the `map_block_meta` module is a mapper that takes a raw Ethereum block as input ([`sf.ethereum.type.v2.Block`](https://github.com/streamingfast/firehose-ethereum/blob/develop/proto/sf/ethereum/type/v2/type.proto)) and outputs the `BlockMeta` protobuf. Basically, the `map_block_meta` module returns a reduced version of the Ethereum block. - diff --git a/docs/new/tutorials/overview.md b/docs/new/tutorials/overview.md deleted file mode 100644 index e69de29bb..000000000 diff --git a/docs/new/tutorials/rust/option.md b/docs/new/tutorials/rust/option.md deleted file mode 100644 index 29767a72f..000000000 --- a/docs/new/tutorials/rust/option.md +++ /dev/null @@ -1,105 +0,0 @@ -# The Option struct - -## The Problem - -Consider that you want to implement a function that returns a username, given the corresponding user identifier. The signature of the function could be as follows: - -```rust -fn get_username_by_id(id: u32) -> String { - // function body -} -``` - -In a success case, you pass the user identifier as a parameter and the function returns the corresponding username. However, **what happens if the function does not have a username for the given identifier?** A possible solution is to return an empty string: -- If the function **is able to retrieve the data**, then the string returned is the username. -- If the function **is NOT able to retrieve the data**, then the string returned is the empty string (`''`). - -Although this is a valid approach, it creates hidden logic that is not visible unless you deep dive into the function code. - -## The Solution - -Rust provides a better way of dealing with these situations by using the `Option` enum. This enum has two possible values: `Some(T)` (used when the returned value is present) and `None` (used when the returned value is not present). Therefore, the previous function can be refactored to: - -```rust -fn get_username_by_id(id: u32) -> Option { - // function body -} -``` - -Now, the function works as follows: -- If the function **is able to retrieve the data**, then a `Some` value containing the string is returned. -- If the function **is NOT able to retrieve the data**, then a `None` value is returned. - -Let's complete the body of the function: - -```rust -fn get_username_by_id(id: u32) -> Option { // 1. - match(id) { - 1 => Some(String::from("Susan")), // 2. - 2 => Some(String::from("John")), // 3. - _ => None // 4. - } -} -``` -1. Given a user identifier, return the corresponding username if it exists. -2. If `id == 1`, then a `Some` struct containing the string is returned. -3. If `id == 2`, then a `Some` struct containing the string is returned. -4. If `id` does not match with any of the provided identifiers, then a `None` struct is returned. - -## Using Options - -The `Option` struct contains two helper methods to check if the returned type is `Some` or `None`: the `.is_some()` and `.is_none()` methods. Let's see how to use these methods: - -```rust -fn get_username_by_id(id: u32) -> Option { - match(id) { - 1 => Some(String::from("Susan")), - 2 => Some(String::from("John")), - _ => None - } -} - -fn main() { - let user1 = get_username_by_id(1); // 1. - let user10 = get_username_by_id(10); // 2. - - if (user1.is_some()) { // 3. - println!("User with id = 1 holds username {}", user1.unwrap()) - } - - if (user10.is_none()) { // 4. - println!("User with id = 10 does not exist") - } -} -``` -1. Get the user with `id == 1`. -1. Get the user with `id == 10`. -3. If the function returned a name for `id == 1`, then `user1.is_some()` returns `true`. -4. If the function did NOT return a name for `id == 10`, then `user1.is_none()` returns `true`. - -You can also use [pattern matching](https://doc.rust-lang.org/book/ch18-03-pattern-syntax.html) instead of the helper methods: - -```rust -fn get_username_by_id(id: u32) -> Option { - match(id) { - 1 => Some(String::from("Susan")), - 2 => Some(String::from("John")), - _ => None - } -} - -fn main() { - let user1 = get_username_by_id(1); - let user10 = get_username_by_id(10); - - match (&user1) { - Some(name) => println!("User with id = 1 holds username {}", &user1.unwrap()), - None => println!("No user with id = 1 found") - } - - match (&user10) { - Some(name) => println!("User with id = 10 holds username {}", &user10.unwrap()), - None => println!("No user with id = 10 found") - } -} -``` diff --git a/docs/new/tutorials/rust/overview.md b/docs/new/tutorials/rust/overview.md deleted file mode 100644 index 9ad55ff04..000000000 --- a/docs/new/tutorials/rust/overview.md +++ /dev/null @@ -1,7 +0,0 @@ -# Rust Tutorials - -Currently, the only programming language supported to build Substreams is Rust, although more might be added in the future. - -If you have experience with typed programming languages, such as Go or Java, you should be able to understand and learn Rust pretty easily. However, there some features and standards that are specific to the Rust programming language. - -The Substreams team recommends following the official [Rust by Example tutorial](https://doc.rust-lang.org/rust-by-example/), which pretty much includes everything you should know about Rust. At the same time, the following sections cover some features and standards that are important when developing Substreams. \ No newline at end of file diff --git a/docs/new/tutorials/rust/result.md b/docs/new/tutorials/rust/result.md deleted file mode 100644 index ba47c248b..000000000 --- a/docs/new/tutorials/rust/result.md +++ /dev/null @@ -1,175 +0,0 @@ -# The Result struct - -In Rust, the `Result` struct is used to abstract both a successful response (if it exists) and an error (if it occurs). Let's better understand through an example. - -## Basic Usage - -Consider that you have a function `divide(num1, num2)`, which executes the division between two numbers. As you already know, dividing by 0 is undefined, and generates an error in Rust. You can use `Result` to return a controlled error. - -```rust -fn divide(num1: u32, num2: u32) -> Result { - if num2 == 0 { - return Err(String::from("You can't divide by 0")); - } - - return Ok(num1 / num2); -} - -fn main() { - let result = divide(6, 0); - if result.is_ok() { - println!("This is the happy path: {}", result.unwrap()) - } else { - println!("This is the error: {}", result.err().unwrap()) - } -} -``` - -Let's inspect the `divide` function: - -```rust -fn divide(num1: u32, num2: u32) -> Result { // 1. - if num2 == 0 { - return Err(String::from("You can't divide by 0")); // 2. - } - - return Ok(num1 / num2); // 3. -} -``` - -1. Declaration of the function. Two unsigned numbers of 32-bit length are passed as parameters. - The return type is `Result`: the first type (`u32`) is for the successful response, and the second type (`String`) is for the error response. -2. If dividing by 0, you return an error String. -3. If not, you return the result of the division (`u32`). - -The `Result` is really an enum that can take two values: `Ok(T)` (success) and `Err(E)` (error). - -In the previous code, when you return `Err(String)`, the success part is automatically empty. At the same time, when you return `Ok(u32)`, the error part is empty. - -Now, let's see how you can interact with this result. - -```rust -fn main() { - let result = divide(6, 0); // 1. - if result.is_ok() { // 2. - println!("This is the happy path: {}", result.unwrap()) - } else { // 3. - println!("This is the error: {}", result.err().unwrap()) - } -} -``` - -1. You invoke the function and store the `Result` enum in a variable. -2. If the result _is ok_ (i.e. the happy path has been returned), you can take its value by using the `result.unwrap()` method. -3. If the error has been returned, you can return the error string by using the `result.err().unwrap()` method. - -The output of the program for `divide(6,2)` (happy path) is: - -```bash -This is the happy path: 3 -``` - -The output of the program for `divide(6,0)` (error) is: - -```bash -This is the error: You can't divide by 0 -``` - -## The Shortcut - -Checking with an `if` condition whether the result contains an error is a valid approach. However, Rust includes a shortcut to improve this. - -In the previous example, consider that you want to invoke the `divide` function from another function that performs other computations. - -```rust -fn divide(num1: u32, num2: u32) -> Result { - if num2 == 0 { - return Err(String::from("You can't divide by 0")); - } - - return Ok(num1 / num2); -} - -fn computations() -> Result { - let result = divide(6, 0); // Performing the division - - if result.is_err() { // If the division returns an error, then you return an error. - return Err(result.err().unwrap()); - } - - let division_result = result.unwrap(); - return Ok(division_result + 5); -} - -fn main() { - let result = computations(); - if result.is_ok() { - println!("This is the happy path: {}", result.unwrap()) - } else { - println!("This is the error: {}", result.err().unwrap()) - } -} -``` - -Now, the Rust program adds `5` to the result of the division, checking that the division is correct first. -Although this approach is correct, Rust provides a `?` symbol that simplifies the logic: - -```rust -fn divide(num1: u32, num2: u32) -> Result { - if num2 == 0 { - return Err(String::from("You can't divide by 0")); - } - - return Ok(num1 / num2); -} - -fn computations() -> Result { - let division_result = divide(6, 0)?; - - return Ok(division_result + 5); -} - -fn main() { - let result = computations(); - if result.is_ok() { - println!("This is the happy path: {}", result.unwrap()) - } else { - println!("This is the error: {}", result.err().unwrap()) - } -} -``` - -The `?` symbol after a `Result` enum does two things: - -1. If successful, it unwraps the result (in this case, a `u32` number), and stores it in a variable - `let division_result = divide(6, 0)?;` -2. If an error occurs, it returns the error directly. In this example, the error type of the `divide` and the `computations` function is the same (a `String`). - -## In Substreams - -The `Result` enum is used in Substreams to return the data (or the errors) of a module. For example, if you take the `map_filter_transactions` module from the [Ethereum Explorer tutorial](/tutorials/ethereum/exploring-ethereum/map_filter_transactions_module): - -```rust -[...] - -#[substreams::handlers::map] -fn map_filter_transactions(params: String, blk: Block) -> Result> { - let filters = parse_filters_from_params(params)?; - - let transactions: Vec = blk - .transactions() - .filter(|trans| apply_filter(&trans, &filters)) - .map(|trans| Transaction { - from: Hex::encode(&trans.from), - to: Hex::encode(&trans.to), - hash: Hex::encode(&trans.hash), - }) - .collect(); - - Ok(Transactions { transactions }) -} - -[...] -``` - -This module returns a `Result> ` enum. If successful, it returns the transactions filtered; in the case of an error, it returns the `substreams::errors::Error` error, which is a Substreams wrapper for a generic [anyhow Rust error](https://docs.rs/anyhow/latest/anyhow/). diff --git a/docs/reference-and-specs/change-log.md b/docs/reference-and-specs/change-log.md deleted file mode 100644 index 432ae113d..000000000 --- a/docs/reference-and-specs/change-log.md +++ /dev/null @@ -1,974 +0,0 @@ -# Change log - -{% hint style="info" %} -Substreams builds upon Firehose.\ -Keep track of [Firehose releases and Data model updates](https://firehose.streamingfast.io/release-notes/change-logs) in the Firehose documentation. -{% endhint %} - -All notable changes to this project will be documented in this file. - -The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## Unreleased - -### Changed - -* Codegen: substreams-database-change to v1.3, properly generates primary key to support chain reorgs in postgres sink. -* Sink server: support for deploying sinks with DBT configuration, so that users can deploy their own DBT models (supported on postgres and clickhouse sinks). Example manifest file segment: - - ```yaml - [...] - - sink: - module: db_out - type: sf.substreams.sink.sql.v1.Service - config: - schema: "./schema.sql" - wire_protocol_access: true - postgraphile_frontend: - enabled: true - pgweb_frontend: - enabled: true - dbt: - files: "./dbt" - run_interval_seconds: 60 - ``` - - where "./dbt" is a folder containing the dbt project. -* Sink server: added REST interface support for clickhouse sinks. Example manifest file segment: - - ```yaml - [...] - - sink: - module: db_out - type: sf.substreams.sink.sql.v1.Service - config: - schema: "./schema.clickhouse.sql" - wire_protocol_access: true - engine: clickhouse - postgraphile_frontend: - enabled: false - pgweb_frontend: - enabled: false - rest_frontend: - enabled: true - ``` - -### Fixed - -* Fix `substreams info` cli doc field which wasn't printing any doc output - -## v1.1.20 - -* Optimized start of output stream in developer mode when start block is in reversible segment and output module does not have any stores in its dependencies. -* Fixed bug where the first streamable block of a chain was not processed correctly when the start block was set to the default zero value. - -## v1.1.19 - -### Changed - -* Codegen: Now generates separate substreams.{target}.yaml files for sql, clickhouse and graphql sink targets. - -### Added - -* Codegen: Added support for clickhouse in schema.sql - -### Fixed - -* Fixed metrics for time spent in eth\_calls within modules stats (server and GUI) -* Fixed `undo` json message in 'run' command -* Fixed stream ending immediately in dev mode when start/end blocks are both 0. -* Sink-serve: fix missing output details on docker-compose apply errors -* Codegen: Fixed pluralized entity created for db\_out and graph\_out - -## v1.1.18 - -### Fixed - -* Fixed a regression where start block was not resolved correctly when it was in the reversible segment of the chain, causing the substreams to reprocess a segment in tier 2 instead of linearly in tier 1. - -## v1.1.17 - -### Fixed - -* Missing decrement on metrics `substreams_active_requests` - -## v1.1.16 - -### Added - -* `substreams_active_requests` and `substreams_counter` metrics to `substreams-tier1` - -### Changed - -* `evt_block_time` in ms to timestamp in `lib.rs`, proto definition and `schema.sql` - -## v1.1.15 - -### Highlights - -* This release brings the `substreams init` command out of alpha! You can quickly generate a Substreams from an Ethereum ABI: ![init-flow](../assets/init-flow.gif) -* New Alpha feature: deploy your Substreams Sink as a deployable unit to a local docker environment! ![sink-deploy-flow](../assets/sink-deploy-flow.gif) -* See those two new features in action in this [tutorial](https://substreams.streamingfast.io/tutorials/from-ethereum-address-to-sql) - -### Added - -* Sink configs can now use protobuf annotations (aka Field Options) to determine how the field will be interpreted in substreams.yaml: - - * `load_from_file` will put the content of the file directly in the field (string and bytes contents are supported). - * `zip_from_folder` will create a zip archive and put its content in the field (field type must be bytes). - - Example protobuf definition: - - ``` - import "sf/substreams/v1/options.proto"; - - message HostedPostgresDatabase { - bytes schema = 1 [ (sf.substreams.v1.options).load_from_file = true ]; - bytes extra_config_files = 2 [ (sf.substreams.v1.options).zip_from_folder = true ]; - } - ``` - - Example manifest file: - - ```yaml - [...] - network: mainnet - - sink: - module: main:db_out - type: sf.substreams.sink.sql.v1.Service - config: - schema: "./schema.sql" - wire_protocol_access: true - postgraphile_frontend: - enabled: true - pgweb_frontend: - enabled: true - ``` -* `substreams info` command now properly displays the content of sink configs, optionally writing the fields that were bundled from files to disk with `--output-sinkconfig-files-path=` - -### Changed - -* `substreams alpha init` renamed to `substreams init`. It now includes `db_out` module and `schema.sql` to support the substreams-sql-sink directly. -* The override feature has been overhauled. Users may now override an existing substreams by pointing to an override file in `run` or `gui` command. This override manifest will have a `deriveFrom` field which points to the original substreams which is to be overriden. This is useful to port a substreams to one network to another. Example of an override manifest: - - ``` - deriveFrom: path/to/mainnet-substreams.spkg #this can also be a remote url - - package: - name: "polygon-substreams" - version: "100.0.0" - - network: polygon - - initialBlocks: - module1: 17500000 - params: - module1: "address=2a75ca72679cf1299936d6104d825c9654489058" - ``` -* The `substreams run` and `substreams gui` commands now determine the endpoint from the 'network' field in the manifest if no value is passed in the `--substreams-endpoint` flag. -* The endpoint for each network can be set by using an environment variable `SUBSTREAMS_ENDPOINTS_CONFIG_`, ex: `SUBSTREAMS_ENDPOINTS_CONFIG_MAINNET=my-endpoint:443` -* The `substreams alpha init` has been moved to `substreams init` - -### Fixed - -* fixed the `substreams gui` command to correctly compute the stop-block when given a relative value (ex: '-t +10') - -## v1.1.14 - -### Bug fixes - -* Fixed (bumped) substreams protobuf definitions that get embedded in `spkg` to match the new progress messages from v1.1.12. -* Regression fix: fixed a bug where negative start blocks would not be resolved correctly when using `substreams run` or `substreams gui`. -* In the request plan, the process previously panicked when errors related to block number validation occurred. Now the error will be returned to the client. - -## v1.1.13 - -### Bug fixes - -* If the initial block or start block is less than the first block in the chain, the substreams will now start from the first block in the chain. Previously, setting the initial block to a block before the first block in the chain would cause the substreams to hang. -* Fixed a bug where the substreams would fail if the start block was set to a future block. The substreams will now wait for the block to be produced before starting. - -## v1.1.12 - -### Highlights - -* Complete redesign of the progress messages: - * Tier2 internal stats are aggregated on Tier1 and sent out every 500ms (no more bursts) - * No need to collect events on client: a single message now represents the current state - * Message now includes list of running jobs and information about execution stages - * Performance metrics has been added to show which modules are executing slowly and where the time is spent (eth calls, store operations, etc.) - -### Upgrading client and server - -> \[!IMPORTANT] The client and servers will both need to be upgraded at the same time for the new progress messages to be parsed: -> -> * The new Substreams servers will _NOT_ send the old `modules` field as part of its `progress` message, only the new `running_jobs`, `modules_stats`, `stages`. -> * The new Substreams clients will _NOT_ be able to decode the old progress information when connecting to older servers. - -However, the actual data (and cursor) will work correctly between versions. Only incompatible progress information will be ignored. - -### CLI - -#### Changed - -* Bumped `substreams` and `substreams-ethereum` to latest in `substreams alpha init`. -* Improved error message when `` is not received, previously this would lead to weird error message, now, if the input is likely a manifest, the error message will be super clear. - -#### Fixed - -* Fixed compilation errors when tracking some contracts when using `substreams alpha init`. - -#### Added - -* `substreams info` now takes an optional second parameter `` to show how the substreams modules can be divided into stages -* Pack command: added `-c` flag to allow overriding of certain substreams.yaml values by passing in the path of a yaml file. example yaml contents: - - ```yaml - package: - name: my_custom_package_name - - network: arbitrum-one - initialBlocks: - module_name_1: 123123123 - params: - mod1: "custom_parameter" - ``` - -### Backend - -#### Removed - -* Removed `Config.RequestStats`, stats are now always enabled. - -## v1.1.11 - -### Fixes - -* Added metering of live blocks - -## v1.1.10 - -### Backend changes - -* Fixed/Removed: jobs would hang when config parameter `StateBundleSize` was different from `SubrequestsSize`. The latter has been removed completely: Subrequests size will now always be aligned with bundle size. -* Auth: added support for _continuous authentication_ via the grpc auth plugin (allowing cutoff triggered by the auth system). - -### CLI changes - -* Fixed params handling in `gui` mode - -## v1.1.9 - -### Backend changes - -* Massive refactoring of the scheduler: prevent excessive splitting of jobs, grouping them into stages when they have the same dependencies. This should reduce the required number of `tier2` workers (2x to 3x, depending on the substreams). -* The `tier1` and `tier2` config have a new configuration `StateStoreDefaultTag`, will be appended to the `StateStoreURL` value to form the final state store URL, ex: `StateStoreURL="/data/states"` and `StateStoreDefaultTag="v2"` will make `/data/states/v2` the default state store location, while allowing users to provide a `X-Sf-Substreams-Cache-Tag` header (gated by auth module) to point to `/data/states/v1`, and so on. -* Authentication plugin `trust` can now specify an exclusive list of `allowed` headers (all lowercase), ex: `trust://?allowed=x-sf-user-id,x-sf-api-key-id,x-real-ip,x-sf-substreams-cache-tag` -* The `tier2` app no longer has customizable auth plugin (or any Modules), `trust` will always be used, so that `tier` can pass down its headers (e.g. `X-Sf-Substreams-Cache-Tag`). The `tier2` instances should not be accessible publicly. - -### GUI changes - -* Color theme is now adapted to the terminal background (fixes readability on 'light' background) -* Provided parameters are now shown in the 'Request' tab. - -### CLI changes - -#### Added - -* `alpha init` command: replace `initialBlock` for generated manifest based on contract creation block. -* `alpha init` prompt Ethereum chain. Added: Mainnet, BNB, Polygon, Goerli, Mumbai. - -#### Fixed - -* `alpha init` reports better progress specially when performing ABI & creation block retrieval. -* `alpha init` command without contracts fixed Protogen command invocation. - -## v1.1.8 - -### Backend changes - -#### Added - -* Max-subrequests can now be overridden by auth header `X-Sf-Substreams-Parallel-Jobs` (note: if your auth plugin is 'trust', make sure that you filter out this header from public access -* Request Stats logging. When enable it will log metrics associated to a Tier1 and Tier2 request -* On request, save "substreams.partial.spkg" file to the state cache for debugging purposes. -* Manifest reader can now read 'partial' spkg files (without protobuf and metadata) with an option. - -#### Fixed - -* Fixed a bug which caused "live" blocks to be sent while the stream previously received block(s) were historic. - -### CLI changes - -#### Fixed - -* In GUI, module output now shows fields with default values, i.e. `0`, `""`, `false` - -## v1.1.7 (https://github.com/streamingfast/substreams/releases/tag/v1.1.7) - -### Highlights - -Now using `plugin: buf.build/community/neoeinstein-prost-crate:v0.3.1` when generating the Protobuf Rust `mod.rs` which fixes the warning that remote plugins are deprecated. - -Previously we were using `remote: buf.build/prost/plugins/crate:v0.3.1-1`. But remote plugins when using https://buf.build (which we use to generate the Protobuf) are now deprecated and will cease to function on July 10th, 2023. - -The net effect of this is that if you don't update your Substreams CLI to `1.1.7`, on July 10th 2023 and after, the `substreams protogen` will not work anymore. - -## v1.1.6 (https://github.com/streamingfast/substreams/releases/tag/v1.1.6) - -### Backend changes - -* `substreams-tier1` and `substreams-tier2` are now standalone **Apps**, to be used as such by server implementations (_firehose-ethereum_, etc.) -* `substreams-tier1` now listens to [Connect](https://buf.build/blog/connect-a-better-grpc) protocol, enabling browser-based substreams clients -* **Authentication** has been overhauled to take advantage of https://github.com/streamingfast/dauth, allowing the use of a GRPC-based sidecar or reverse-proxy to provide authentication. -* **Metering** has been overhauled to take advantage of https://github.com/streamingfast/dmetering plugins, allowing the use of a GRPC sidecar or logs to expose usage metrics. -* The **tier2 logs** no longer show a `parent_trace_id`: the `trace_id` is now the same as tier1 jobs. Unique tier2 jobs can be distinguished by their `stage` and `segment`, corresponding to the `output_module_name` and `startblock:stopblock` - -### CLI changes - -* The `substreams protogen` command now uses this Buf plugin https://buf.build/community/neoeinstein-prost to generate the Rust code for your Substreams definitions. -* The `substreams protogen` command no longer generate the `FILE_DESCRIPTOR_SET` constant which generates an unsued warning in Rust. We don't think nobody relied on having the `FILE_DESCRIPTOR_SET` constant generated, but if it's the case, you can provide your own `buf.gen.yaml` that will be used instead of the generated one when doing `substreams protogen`. -* Added `-H` flag on the `substreams run` command, to set HTTP Headers in the Substreams request. - -### Fixed - -* Fixed generated `buf.gen.yaml` not being deleted when an error occurs while generating the Rust code. - -## [v1.1.5](https://github.com/streamingfast/substreams/releases/tag/v1.1.5) - -### Highlights - -This release fixes data determinism issues. This comes at a 20% performance cost but is necessary for integration with The Graph ecosystem. - -#### Operators - -* When upgrading a substreams server to this version, you should delete all existing module caches to benefit from deterministic output - -### Added - -* Tier1 now records deterministic failures in wasm, "blacklists" identical requests for 10 minutes (by serving them the same InvalidArgument error) with a forced incremental backoff. This prevents accidental bad actors from hogging tier2 resources when their substreams cannot go passed a certain block. -* Tier1 now sends the ResolvedStartBlock, LinearHandoffBlock and MaxJobWorkers in SessionInit message for the client and gui to show -* Substreams CLI can now read manifests/spkg directly from an IPFS address (subgraph deployment or the spkg itself), using `ipfs://Qm...` notation - -### Fixed - -* When talking to an updated server, the gui will not overflow on a negative start block, using the newly available resolvedStartBlock instead. -* When running in development mode with a start-block in the future on a cold cache, you would sometimes get invalid "updates" from the store passed down to your modules that depend on them. It did not impact the caches but caused invalid output. -* The WASM engine was incorrectly reusing memory, preventing deterministic output. It made things go faster, but at the cost of determinism. Memory is now reset between WASM executions on each block. -* The GUI no longer panics when an invalid output-module is given as argument - -### Changed - -* Changed default WASM engine from `wasmtime` to `wazero`, use `SUBSTREAMS_WASM_RUNTIME=wasmtime` to revert to prior engine. Note that `wasmtime` will now run a lot slower than before because resetting the memory in `wasmtime` is more expensive than in `wazero`. -* Execution of modules is now done in parallel within a single instance, based on a tree of module dependencies. -* The `substreams gui` and `substreams run` now accept commas inside a `param` value. For example: `substreams run --param=p1=bar,baz,qux --param=p2=foo,baz`. However, you can no longer pass multiple parameters using an ENV variable, or a `.yaml` config file. - -## [v1.1.4](https://github.com/streamingfast/substreams/releases/tag/v1.1.4) - -### HIGHLIGHTS - -* Module hashing changed to fix cache reuse on substreams use imported modules -* Memory leak fixed on rpc-enabled servers -* GUI more responsive - -### Fixed - -* BREAKING: The module hashing algorithm wrongfully changed the hash for imported modules, which made it impossible to leverage caches when composing new substreams off of imported ones. - * Operationally, if you want to keep your caches, you will need to copy or move the old hashes to the new ones. - * You can obtain the prior hashes for a given spkg with: `substreams info my.spkg`, using a prior release of the `substreams` - * With a more recent `substreams` release, you can obtain the new hashes with the same command. - * You can then `cp` or `mv` the caches for each module hash. - * You can also ignore this change. This will simply invalidate your cache. -* Fixed a memory leak where "PostJobHooks" were not always called. These are used to hook in rpc calls in Ethereum chain. They are now always called, even if no block has been processed (can be called with `nil` value for the clock) -* Jobs that fail deterministically (during WASM execution) on tier2 will fail faster, without retries from tier1. -* `substreams gui` command now handles params flag (it was ignored) -* Substeams GUI responsiveness improved significantly when handling large payloads - -### Added - -* Added Tracing capabilities, using https://github.com/streamingfast/sf-tracing . See repository for details on how to enable. - -### Known issues - -* If the cached substreams states are missing a 'full-kv' file in its sequence (not a normal scenario), requests will fail with `opening file: not found` https://github.com/streamingfast/substreams/issues/222 - -## [v1.1.3](https://github.com/streamingfast/substreams/releases/tag/v1.1.3) - -### Highlights - -This release contains fixes for race conditions that happen when multiple request tries to sync the same range using the same `.spkg`. Those fixes will avoid weird state error at the cost of duplicating work in some circumstances. A future refactor of the Substreams engine scheduler will come later to fix those inefficiencies. - -Operators, please read the operators section for upgrade instructions. - -#### Operators - -> **Note** This upgrade procedure is applies if your Substreams deployment topology includes both `tier1` and `tier2` processes. If you have defined somewhere the config value `substreams-tier2: true`, then this applies to you, otherwise, if you can ignore the upgrade procedure. - -This release includes a small change in the internal RPC layer between `tier1` processes and `tier2` processes. This change requires an ordered upgrade of the processes to avoid errors. - -The components should be deployed in this order: - -1. Deploy and roll out `tier1` processes first -2. Deploy and roll out `tier2` processes in second - -If you upgrade in the wrong order or if somehow `tier2` processes start using the new protocol without `tier1` being aware, user will end up with backend error(s) saying that some partial file are not found. Those will be resolved only when `tier1` processes have been upgraded successfully. - -### Fixed - -* Fixed a race when multiple Substreams request execute on the same `.spkg`, it was causing races between the two executors. -* GUI: fixed an issue which would slow down message consumption when progress page was shown in ascii art "bars" mode -* GUI: fixed the display of blocks per second to represent actual blocks, not messages count - -### Changed - -* \[`binary`]: Commands `substreams <...>` that fails now correctly return an exit code 1. -* \[`library`]: The `manifest.NewReader` signature changed and will now return a `*Reader, error` (previously `*Reader`). - -### Added - -* \[`library`]: The `manifest.Reader` gained the ability to infer the path if provided with input `""` based on the current working directory. -* \[`library`]: The `manifest.Reader` gained the ability to infer the path if provided with input that is a directory. - -## [v1.1.2](https://github.com/streamingfast/substreams/releases/tag/v1.1.2) - -### Highlights - -This release contains bug fixes and speed/scaling improvements around the Substreams engine. It also contains few small enhancements for `substreams gui`. - -This release contains an important bug that could have generated corrupted `store` state files. This is important for developers and operators. - -#### Sinkers & Developers - -The `store` state files will be fully deleted on the Substreams server to start fresh again. The impact for you as a developer is that Substreams that were fully synced will now need to re-generate from initial block the store's state. So you might see long delays before getting a new block data while the Substreams engine is re-computing the `store` states from scratch. - -### Operators - -You need to clear the state store and remove all the files that are stored under `substreams-state-store-url` flag. You can also make it point to a brand new folder and delete the old one after the rollout. - -### Fixed - -* Fix a bug where not all extra modules would be sent back on debug mode -* Fixed a bug in tier1 that could result in corrupted state files when getting close to chain HEAD -* Fixed some performance and stalling issues when using GCS for blocks -* Fixed storage logs not being shown properly -* GUI: Fixed panic race condition -* GUI: Cosmetic changes - -### Added - -* GUI: Added traceID - -## [v1.1.1](https://github.com/streamingfast/substreams/releases/tag/v1.1.1) - -### Highlights - -This release introduces a new RPC protocol and the old one has been removed. The new RPC protocol is in a new Protobuf package `sf.substreams.rpc.v2` and it drastically changes how chain re-orgs are signaled to the user. Here the highlights of this release: - -* Getting rid of `undo` payload during re-org -* `substreams gui` Improvements -* Substreams integration testing -* Substreams Protobuf definitions updated - -#### Getting rid of `undo` payload during re-org - -Previously, the GRPC endpoint `sf.substreams.v1.Stream/Blocks` would send a payload with the corresponding "step", NEW or UNDO. - -Unfortunately, this led to some cases where the payload could not be deterministically generated for old blocks that had been forked out, resulting in a stalling request, a failure, or in some worst cases, incomplete data. - -The new design, under `sf.substreams.rpc.v2.Stream/Blocks`, takes care of these situations by removing the 'step' component and using these two messages types: - -* `sf.substreams.rpc.v2.BlockScopedData` when chain progresses, with the payload -* `sf.substreams.rpc.v2.BlockUndoSignal` during a reorg, with the last valid block number + block hash - -The client now has the burden of keeping the necessary means of performing the undo actions (ex: a map of previous values for each block). The BlockScopedData message now includes the `final_block_height` to let you know when this "undo data" can be discarded. - -With these changes, a substreams server can even handle a cursor for a block that it has never seen, provided that it is a valid cursor, by signaling the client to revert up to the last known final block, trading efficiency for resilience in these extreme cases. - -### `substreams gui` Improvements - -* Added key 'f' shortcut for changing display encoding of bytes value (hex, pruned string, base64) -* Added `jq` search mode (hit `/` twice). Filters the output with the `jq` expression, and applies the search to match all blocks. -* Added search history (with `up`/`down`), similar to `less`. -* Running a search now applies it to all blocks, and highlights the matching ones in the blocks bar (in red). -* Added `O` and `P`, to jump to prev/next block with matching search results. -* Added module search with `m`, to quickly switch from module to module. - -#### Substreams integration testing - -Added a basic Substreams testing framework that validates module outputs against expected values. The testing framework currently runs on `substreams run` command, where you can specify the following flags: - -* `test-file` Points to a file that contains your test specs -* `test-verbose` Enables verbose mode while testing. - -The test file, specifies the expected output for a given substreams module at a given block. - -#### Substreams Protobuf definitions updated - -We changed the Substreams Protobuf definitions making a major overhaul of the RPC communication. This is a **breaking change** for those consuming Substreams through gRPC. - -> **Note** The is no breaking changes for Substreams developers regarding your Rust code, Substreams manifest and Substreams package. - -* Removed the `Request` and `Response` messages (and related) from `sf.substreams.v1`, they have been moved to `sf.substreams.rpc.v2`. You will need to update your usage if you were consuming Substreams through gRPC. -* The new `Request` excludes fields and usages that were already deprecated, like using multiple `module_outputs`. -* The `Response` now contains a single module output -* In `development` mode, the additional modules output can be inspected under `debug_map_outputs` and `debug_store_outputs`. - -**Separating Tier1 vs Tier2 gRPC protocol (for Substreams server operators)** - -Now that the `Blocks` request has been moved from `sf.substreams.v1` to `sf.substreams.rpc.v2`, the communication between a substreams instance acting as tier1 and a tier2 instance that performs the background processing has also been reworked, and put under `sf.substreams.internal.v2.Stream/ProcessRange`. It has also been stripped of parameters that were not used for that level of communication (ex: `cursor`, `logs`...) - -### Fixed - -* The `final_blocks_only: true` on the `Request` was not honored on the server. It now correctly sends only blocks that are final/irreversible (according to Firehose rules). -* Prevent substreams panic when requested module has unknown value for "type" - -### Added - -* The `substreams run` command now has flag `--final-blocks-only` - -## [1.0.3](https://github.com/streamingfast/substreams/releases/tag/v1.0.3) - -This should be the last release before a breaking change in the API and handling of the reorgs and UNDO messages. - -### Highlights - -* Added support for resolving a negative start-block on server -* CHANGED: The `run` command now resolves a start-block=-1 from the head of the chain (as supported by the servers now). Prior to this change, the `-1` value meant the 'initialBlock' of the requested module. The empty string is now used for this purpose, -* GUI: Added support for search, similar to `less`, with `/`. -* GUI: Search and output offset is conserved when switching module/block number in the "Output" tab. -* Library: protobuf message descriptors now exposed in the `manifest/` package. This is something useful to any sink that would need to interpret the protobuf messages inside a Package. -* Added support for resolving a negative start-block on server (also added to run command) -* The `run` and `gui` command no longer resolve a `start-block=-1` to the 'initialBlock' of the requested module. To get this behavior, simply assign an empty string value to the flag `start-block` instead. -* Added support for search within the Substreams gui `output` view. Usage of search within `output` behaves similar to the `less` command, and can be toggled with "/". - -## [1.0.2](https://github.com/streamingfast/substreams/releases/tag/v1.0.2) - -* Release was retracted because it contained the refactoring expected for 1.1.0 by mistake, check https://github.com/streamingfast/substreams/releases/tag/v1.0.3 instead. - -## [1.0.1](https://github.com/streamingfast/substreams/releases/tag/v1.0.1) - -### Fixed - -* Fixed "undo" messages incorrectly contained too many module outputs (all modules, with some duplicates). -* Fixed status bar message cutoff bug -* Fixed `substreams run` when `manifest` contains unknown attributes -* Fixed bubble tea program error when existing the `run` command - -## [1.0.0](https://github.com/streamingfast/substreams/releases/tag/v1.0.0) - -### Highlights - -* Added command `substreams gui`, providing a terminal-based GUI to inspect the streamed data. Also adds `--replay` support, to save a stream to `replay.log` and load it back in the UI later. You can use it as you would `substreams run`. Feedback welcome. -* Modified command `substreams protogen`, defaulting to generating the `mod.rs` file alongside the rust bindings. Also added `--generate-mod-rs` flag to toggle `mod.rs` generation. -* Added support for module parameterization. Defined in the manifest as: - -``` -module: - name: my_module - inputs: - params: string - ... - -params: - my_module: "0x123123" - "imported:module": override value from imported module -``` - -and on the command-line as: - -* `substreams run -p module=value -p "module2=other value" ...` - -Servers need to be updated for packages to be able to be consumed this way. - -This change keeps backwards compatibility. Old Substreams Packages will still work the same, with no changes to module hashes. - -### Added - -* Added support for `{version}` template in `--output-file` flag value on `substreams pack`. -* Added fuel limit to wasm execution as a server-side option, preventing wasm process from running forever. -* Added 'Network' and 'Sink{Type, Module, Config}' fields in the manifest and protobuf definition for future bundling of substreams sink definitions within a substreams package. - -## [0.2.0](https://github.com/streamingfast/substreams/releases/tag/v0.2.0) - -### Highlights - -* Improved execution speed and module loading speed by bumping to WASM Time to version 4.0. -* Improved developer experience on the CLI by making the `` argument optional. - - The CLI when `` argument is not provided will now look in the current directory for a `substreams.yaml` file and is going to use it if present. So if you are in your Substreams project and your file is named `substreams.yaml`, you can simply do `substreams pack`, `substreams protogen`, etc. - - Moreover, we added to possibility to pass a directory containing a `substreams.yaml` directly so `substreams pack path/to/project` would work as long as `path/to/project` contains a file named `substreams.yaml`. -* Fixed a bug that was preventing production mode to complete properly when using a bounded block range. -* Improved overall stability of the Substreams engine. - -#### Operators Notes - -* **Breaking** Config values `substreams-stores-save-interval` and `substreams-output-cache-save-interval` have been merged together into `substreams-cache-save-interval` in the `firehose-` repositories. Refer to chain specific `firehose-` repository for further details. - -### Added - -* The `` can point to a directory that contains a `substreams.yaml` file instead of having to point to the file directly. -* The `` parameter is now optional in all commands requiring it. - -### Fixed - -* Fixed valuetype mismatch for stores -* Fixed production mode not completing when block range was specified -* Fixed tier1 crashing due to missing context canceled check. -* Fixed some code paths where locking could have happened due to incorrect checking of context cancellation. -* Request validation for blockchain's input type is now made only against the requested module it's transitive dependencies. - -### Updated - -* Updated WASM Time library to 4.0.0 leading to improved execution speed. - -### Changed - -* Remove distinction between `output-save-interval` and `store-save-interval`. -* `substreams init` has been moved under `substreams alpha init` as this is a feature included by mistake in latest release that should not have been displayed in the main list of commands. -* `substreams codegen` has been moved under `substreams alpha codegen` as this is a feature included by mistake in latest release that should not have been displayed in the main list of commands. - -## [0.1.0](https://github.com/streamingfast/substreams/releases/tag/v0.1.0) - -This upcoming release is going to bring significant changes on how Substreams are developed, consumed and speed of execution. Note that there is **no** breaking changes related to your Substreams' Rust code, only breaking changes will be about how Substreams are run and available features/flags. - -Here the highlights of elements that will change in next release: - -* [Production vs Development Mode](change-log.md#production-vs-development-mode) -* [Single Output Module](change-log.md#single-module-output) -* [Output Module must be of type `map`](change-log.md#output-module-must-be-of-type-map) -* [`InitialSnapshots` is now a `development` mode feature only](change-log.md#initialsnapshots-is-now-a-development-mode-feature-only) -* [Enhanced Parallel Execution](change-log.md#enhanced-parallel-execution) - -In this rest of this post, we are going to go through each of them in greater details and the implications they have for you. Full changelog is available after. - -> **Warning** Operators, refer to [Operators Notes](change-log.md#operators-notes) section for specific instructions of deploying this new version. - -### Production vs development mode - -We introduce an execution mode when running Substreams, either `production` mode or `development` mode. The execution mode impacts how the Substreams get executed, specifically: - -* The time to first byte -* The module logs and outputs sent back to the client -* How parallel execution is applied through the requested range - -The difference between the modes are: - -* In `development` mode, the client will receive all the logs of the executed `modules`. In `production` mode, logs are not available at all. -* In `development` mode, module's are always re-executed from request's start block meaning now that logs will always be visible to the user. In `production` mode, if a module's output is found in cache, module execution is skipped completely and data is returned directly. -* In `development` mode, only backward parallel execution can be effective. In `production` mode, both backward parallel execution and forward parallel execution can be effective. See [Enhanced parallel execution](change-log.md#enhanced-parallel-execution) section for further details about parallel execution. -* In `development` mode, every module's output is returned back in the response but only root module is displayed by default in `substreams` CLI (configurable via a flag). In `production` mode, only root module's output is returned. -* In `development` mode, you may request specific `store` snapshot that are in the execution tree via the `substreams` CLI `--debug-modules-initial-snapshots` flag. In `production` mode, this feature is not available. - -The execution mode is specified at that gRPC request level and is the default mode is `development`. The `substreams` CLI tool being a development tool foremost, we do not expect people to activate production mode (`-p`) when using it outside for maybe testing purposes. - -If today's you have `sink` code making the gRPC request yourself and are using that for production consumption, ensure that field `production_mode` in your Substreams request is set to `true`. StreamingFast provided `sink` like [substreams-sink-postgres](https://github.com/streamingfast/substreams-sink-postgres), [substreams-sink-files](https://github.com/streamingfast/substreams-sink-files) and others have already been updated to use `production_mode` by default. - -Final note, we recommend to run the production mode against a compiled `.spkg` file that should ideally be released and versioned. This is to ensure stable modules' hashes and leverage cached output properly. - -### Single module output - -We now only support 1 output module when running a Substreams, while prior this release, it was possible to have multiple ones. - -* Only a single module can now be requested, previous version allowed to request N modules. -* Only `map` module can now be requested, previous version allowed `map` and `store` to be requested. -* `InitialSnapshots` is now forbidden in `production` mode and still allowed in `development` mode. -* In `development` mode, the server sends back output for all executed modules (by default the CLI displays only requested module's output). - -> **Note** We added `output_module` to the Substreams request and kept `output_modules` to remain backwards compatible for a while. If an `output_module` is specified we will honor that module. If not we will check `output_modules` to ensure there is only 1 output module. In a future release, we are going to remove `output_modules` altogether. - -With the introduction of `development` vs `production` mode, we added a change in behavior to reduce frictions this changes has on debugging. Indeed, in `development` mode, all executed modules's output will be sent be to the user. This includes the requested output module as well as all its dependencies. The `substreams` CLI has been adjusted to show only the output of the requested output module by default. The new `substreams` CLI flag `-debug-modules-output` can be used to control which modules' output is actually displayed by the CLI. - -> **Migration Path** If you are currently requesting more than one module, refactor your Substreams code so that a single `map` module aggregates all the required information from your different dependencies in one output. - -### Output module must be of type `map` - -It is now forbidden to request a `store` module as the output module of the Substreams request, the requested output module must now be of kind `map`. Different factors have motivated this change: - -* Recently we have seen incorrect usage of `store` module. A `store` module was not intended to be used as a persistent long term storage, `store` modules were conceived as a place to aggregate data for later steps in computation. Using it as a persistent storage make the store unmanageable. -* We had always expected users to consume a `map` module which would return data formatted according to a final `sink` spec which will then permanently store the extracted data. We never envisioned `store` to act as long term storage. -* Forward parallel execution does not support a `store` as its last step. - -> **Migration Path** If you are currently using a `store` module as your output store. You will need to create a `map` module that will have as input the `deltas` of said `store` module, and return the deltas. - -#### Examples - -Let's assume a Substreams with these dependencies: `[block] --> [map_pools] --> [store_pools] --> [map_transfers]` - -* Running `substreams run substreams.yaml map_transfers` will only print the outputs and logs from the `map_transfers` module. -* Running `substreams run substreams.yaml map_transfers --debug-modules-output=map_pools,map_transfers,store_pools` will print the outputs of those 3 modules. - -### `InitialSnapshots` is now a `development` mode feature only - -Now that a `store` cannot be requested as the output module, the `InitialSnapshots` did not make sense anymore to be available. Moreover, we have seen people using it to retrieve the initial state and then continue syncing. While it's a fair use case, we always wanted people to perform the synchronization using the streaming primitive and not by using `store` as long term storage. - -However, the `InitialSnapshots` is a useful tool for debugging what a store contains at a given block. So we decided to keep it in `development` mode only where you can request the snapshot of a `store` module when doing your request. In the Substreams' request/response, `initial_store_snapshot_for_modules` has been renamed to `debug_initial_store_snapshot_for_modules`, `snapshot_data` to `debug_snapshot_data` and `snapshot_complete` to `debug_snapshot_complete`. - -> **Migration Path** If you were relying on `InitialSnapshots` feature in production. You will need to create a `map` module that will have as input the `deltas` of said `store` module, and then synchronize the full state on the consuming side. - -#### Examples - -Let's assume a Substreams with these dependencies: `[block] --> [map_pools] --> [store_pools] --> [map_transfers]` - -* Running `substreams run substreams.yaml map_transfers -s 1000 -t +5 --debug-modules-initial-snapshot=store_pools` will print all the entries in store\_pools at block 999, then continue with outputs and logs from `map_transfers` in blocks 1000 to 1004. - -### Enhanced parallel execution - -There are 2 ways parallel execution can happen either backward or forward. - -Backward parallel execution consists of executing in parallel block ranges from the module's start block up to the start block of the request. If the start block of the request matches module's start block, there is no backward parallel execution to perform. Also, this is happening only for dependencies of type `store` which means that if you depends only on other `map` modules, no backward parallel execution happens. - -Forward parallel execution consists of executing in parallel block ranges from the start block of the request up to last known final block (a.k.a the irreversible block) or the stop block of the request, depending on which is smaller. Forward parallel execution significantly improves the performance of the Substreams as we execute your module in advanced through the chain history in parallel. What we stream you back is the cached output of your module's execution which means essentially that we stream back to you data written in flat files. This gives a major performance boost because in almost all cases, the data will be already for you to consume. - -Forward parallel execution happens only in `production` mode is always disabled when in `development` mode. Moreover, since we read back data from cache, it means that logs of your modules will never be accessible as we do not store them. - -Backward parallel execution still occurs in `development` and `production` mode. The diagram below gives details about when parallel execution happen. - -![parallel processing](../assets/substreams\_processing.png) - -You can see that in `production` mode, parallel execution happens before the Substreams request range as well as within the requested range. While in `development` mode, we can see that parallel execution happens only before the Substreams request range, so between module's start block and start block of requested range (backward parallel execution only). - -### Operators Notes - -The state output format for `map` and `store` modules has changed internally to be more compact in Protobuf format. When deploying this new version, previous existing state files should be deleted or deployment updated to point to a new store location. The state output store is defined by the flag `--substreams-state-store-url` flag parameter on chain specific binary (i.e. `fireeth`). - -### Library - -* Added `production_mode` to Substreams Request -* Added `output_module` to Substreams Request - -### CLI - -* Fixed `Ctrl-C` not working directly when in TUI mode. -* Added `Trace ID` printing once available. -* Added command `substreams tools analytics store-stats` to get statistic for a given store. -* Added `--debug-modules-output` (comma-separated module names) (unavailable in `production` mode). -* **Breaking** Renamed flag `--initial-snapshots` to `--debug-modules-initial-snapshots` (comma-separated module names) (unavailable in `production` mode). - -## [0.0.21](https://github.com/streamingfast/substreams/releases/tag/v0.0.21) - -* Moved Rust modules to `github.com/streamingfast/substreams-rs` - -### Library - -* Gained significant execution time improvement when saving and loading stores, during the squashing process by leveraging [vtprotobuf](https://github.com/planetscale/vtprotobuf) -* Added XDS support for tier 2s -* Added intrinsic support for type `bigdecimal`, will deprecate `bigfloat` -* Significant improvements in code-coverage and full integration tests. - -### CLI - -* Added `substreams tools proxy ` subcommand to allow calling substreams with a pre-defined package easily from a web browser using bufbuild/connect-web -* Lowered GRPC client keep alive frequency, to prevent "Too Many Pings" disconnection issue. -* Added a fast failure when attempting to connect to an unreachable substreams endpoint. -* CLI is now able to read `.spkg` from `gs://`, `s3://` and `az://` URLs, the URL format must be supported by our [dstore](https://github.com/streamingfast/dstore) library). -* Command `substreams pack` is now restricted to local manifest file. -* Added command `substreams tools module` to introspect a store state in storage. -* Made changes to allow for `substreams` CLI to run on Windows OS (thanks @robinbernon). -* Added flag `--output-file