Docs revamp (#379)

streamingfast · Jan 19, 2024 · 6e2962b · 6e2962b
1 parent 21d3604
commit 6e2962b
Show file tree

Hide file tree

Showing 73 changed files with 6,469 additions and 126 deletions.
diff --git a/.gitbook.yaml b/.gitbook.yaml
@@ -2,3 +2,40 @@ root: ./docs/
 
 structure:
   summary: SUMMARY.md
+
+redirects:
+  concepts-and-fundamentals/modules: ./new/common/manifest-modules.md
+  developers-guide/creating-protobuf-schema: ./new/develop/creating-protobuf-schemas.md
+  developers-guide/creating-your-manifest: ./new/common/manifest-modules.md
+  developers-guide/gui: ./new/common/manifest-modules.md
+  developers-guide/installation-requirements: ./new/common/installing-the-cli.md
+  developers-guide/overview: ./new/common/introduction.md
+  developers-guide/parallel-execution: ./new/develop/architecture.md
+  developers-guide/running-substreams: ./new/common/running-substreams.md
+  developers-guide/modules/inputs: ./new/develop/modules/inputs.md
+  developers-guide/modules/outputs: ./new/develop/modules/outputs.md
+  developers-guide/modules/setting-up-handlers: ./new/develop/modules/setting-up-handlers.md
+  developers-guide/modules/types: ./new/develop/modules/types.md
+  developers-guide/modules/writing-module-handlers: ./new/develop/modules/writing-module-handlers.md
+  developers-guide/cookbook/advanced-params: ./new/develop/parameterized-modules.md
+  developers-guide/cookbook/aggregation-windows: ./new/develop/modules/aggregation-windows.md
+  developers-guide/cookbook/dynamic-data-sources: ./new/develop/modules/dynamic-data-sources.md
+  developers-guide/cookbook/factory-params: ./new/develop/parameterized-modules.md
+  developers-guide/cookbook/keys-in-stores: ./new/develop/modules/keys-in-stores.md
+  developers-guide/sink-targets/substreams-powered-subgraph: ./new/consume/subgraph/quickstart.md
+  developers-guide/sink-targets/substreams-sink-sql: ./new/consume/sql/introduction.md
+  getting-started/installing-the-cli: ./new/common/installing-the-cli.md
+  getting-started/quickstart: ./new/common/introduction.md
+  references-and-specs/chains-and-endpoints: ./new/references/chains-and-endpoints.md
+  references-and-specs/command-line-interface: ./new/references/command-line-interface.md
+  references-and-specs/examples: ./new/tutorials/overview.md
+  references-and-specs/gui: ./new/references/gui.md
+  references-and-specs/faq: ./new/references/faq.md
+  references-and-specs/manifests: ./new/references/manifests.md
+  references-and-specs/packages: ./new/common/packages.md
+  references-and-specs/rust-crates: ./new/develop/rust-crates.md
+  glossary/glossary: ./new/references/glossary.md
+  quick-access/change-log: ./new/references/change-log.md
+
+
+
diff --git a/docs/.gitbook/assets/chains-endpoints.png b/docs/.gitbook/assets/chains-endpoints.png
diff --git a/docs/.gitbook/assets/consume/consume-services-only.png b/docs/.gitbook/assets/consume/consume-services-only.png
diff --git a/docs/.gitbook/assets/consume/consume-services.png b/docs/.gitbook/assets/consume/consume-services.png
diff --git a/docs/.gitbook/assets/consume/featured-packages.png b/docs/.gitbook/assets/consume/featured-packages.png
diff --git a/docs/.gitbook/assets/consume/service-sql.png b/docs/.gitbook/assets/consume/service-sql.png
diff --git a/docs/.gitbook/assets/consume/service-stream.png b/docs/.gitbook/assets/consume/service-stream.png
diff --git a/docs/.gitbook/assets/consume/service-subgraph.png b/docs/.gitbook/assets/consume/service-subgraph.png
diff --git a/docs/.gitbook/assets/develop/provider-gif.gif b/docs/.gitbook/assets/develop/provider-gif.gif
diff --git a/docs/.gitbook/assets/develop/rust-package-gif.gif b/docs/.gitbook/assets/develop/rust-package-gif.gif
diff --git a/docs/.gitbook/assets/develop/sending-gif.gif b/docs/.gitbook/assets/develop/sending-gif.gif
diff --git a/docs/.gitbook/assets/develop/transformations-gif.gif b/docs/.gitbook/assets/develop/transformations-gif.gif
diff --git a/docs/.gitbook/assets/intro/consume-flow.png b/docs/.gitbook/assets/intro/consume-flow.png
diff --git a/docs/.gitbook/assets/intro/develop-flow.png b/docs/.gitbook/assets/intro/develop-flow.png
diff --git a/docs/.gitbook/assets/intro/supported-chains.png b/docs/.gitbook/assets/intro/supported-chains.png
diff --git a/docs/.gitbook/assets/packages/arch.png b/docs/.gitbook/assets/packages/arch.png
diff --git a/docs/.gitbook/assets/packages/erc20-balance-changes.png b/docs/.gitbook/assets/packages/erc20-balance-changes.png
diff --git a/docs/.gitbook/assets/sql/explore-pgweb.png b/docs/.gitbook/assets/sql/explore-pgweb.png
diff --git a/docs/README.md b/docs/README.md
@@ -1,68 +1,42 @@
----
-description: StreamingFast Substreams documentation
----
+Substreams is a powerful indexing technology, which allows you to:
+- Extract data from several blockchains (Ethereum, Polygon, BNB, Solana...).
+- Apply custom transformations to the data.
+- Send the data to a place of your choice (for example, a Postgres database or a file).
 
-# Substreams
+<figure><img src=".gitbook/assets/intro/supported-chains.png" width="100%" /></figure>
 
-## Welcome to Substreams documentation
+**You can use Substreams packages to define which specific data you want to extract from the blockchain**. For example, consider that you want to retrieve data from the Uniswap v3 smart contract. You can simply use the [Uniswap v3 Substreams Package](https://substreams.dev/streamingfast/uniswap-v3/v0.2.7) and send that data wherever you want!
 
-Substreams is a powerful blockchain indexing technology, developed for [The Graph Network](https://thegraph.com).
+## Consume Substreams
 
-> Substreams enables developers to write Rust modules, composing data streams alongside the community, and provides extremely high-performance indexing by virtue of parallelization, in a streaming-first fashion.
->
-> Substreams have all the benefits of StreamingFast Firehose, like low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling.
+There are many ready-to-use Substreams packages, so you can simply consume them. Use the **([Substreams.dev Registry](https://substreams.dev)) to explore packages**.
 
-### Where to start
+Once you find a package that fits your needs, you only have choose **how you want to consume the data**. Send the data to a SQL database, configure a webhook or stream directly from your application!
 
-Learn about Substreams in a short, dense 25-minute intro and understand its impact on the blockchain ecosystem.
+<figure><img src=".gitbook/assets/intro/consume-flow.png" width="100%" /></figure>
 
-{% embed url="https://www.youtube.com/watch?v=K-nhC2FCB5k" %}
-A walkthrough of Firehose features, Substreams modules, including a sample Rust module and StreamingFast's vision.
-{% endembed %}
+## Develop Substreams
 
-Learn about the benefits of Substreams, and how it compares to otheressential facts about Substreams through [reading the Benefits and comparison](concepts-and-fundamentals/benefits.md).
+If you can't find a Substreams package that retrieves exactly the data you need, **you can develop your own Substreams**.
 
-The primary ways to use Substreams include:
+You can write your own Rust function to extract data from the blockchain:
 
-* [Installing the `substreams` CLI](getting-started/installing-the-cli.md)
-* [Going through the Quickstart](getting-started/quickstart.md)
+```rust
+fn get_usdt_transaction(block: eth::Block) -> Result<Vec<Transaction>, substreams:error:Error> {
+    let my_transactions = block.transactions().
+        .filter(|transaction| transaction.to == USDT_CONTRACT_ADDRESS)
+        .map(|transaction| MyTransaction(transaction.hash, transaction.from, transaction.to))
+        .collect();
+    Ok(my_transactions)
+}
+```
 
-After installing Substreams and reviewing the Quickstart:
+<figure><img src=".gitbook/assets/intro/develop-flow.png" width="100%" /></figure>
 
-* You can [learn more about ](developers-guide/modules/)modules, and then [study the Developer's guide](developers-guide/overview.md).
+## How Does It Work?
 
-Find pre-built Substreams by using the following resources:
+The following video covers how Substreams works in less than 2 minutes:
 
-* The [Substreams Template](https://github.com/streamingfast/substreams-template) helps expedite the process of getting you up and running.
-* A [list of maintained Substreams examples](reference-and-specs/examples.md)
-
-### Network model diagram
-
-<img src=".gitbook/assets/substreams.excalidraw.svg" alt="" class="gitbook-drawing">
-
-**You can view Substreams from two perspectives** as illustrated in the high-level visual diagram. It can be viewed through the perspective of the **Substreams engine** itself and also the perspective of the **end-user developer and consumer**.
-
-### Community
-
-Substreams is an open source community effort, so feel free to suggest new topics, report issues, and provide feedback. Contribute through GitHub [pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests).
-
-* [StreamingFast Discord](https://discord.gg/mYPcRAzeVN)
-* [The Graph Discord](https://discord.gg/vtvv7FP)
-* [StreamingFast on Twitter](https://twitter.com/streamingfastio)
-* [StreamingFast on YouTube](https://www.youtube.com/c/streamingfast)
-
-### Contributing
-
-For additional information, [refer to the general StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md).
-
-### License
-
-Substreams uses the [Apache 2.0](../LICENSE/) license.
-
-### Disclaimer
-
-The content in the Substreams documentation was created through StreamingFast's full effort. It is up to the reader to validate the accuracy of all content presented. Substreams is in active development and, at times, the associated documentation becomes outdated. [Contact StreamingFast](https://discord.gg/mYPcRAzeVN) to report problems or service interruptions.
-
-{% hint style="info" %}
-**Note**: The Substreams documentation uses the [Google developer documentation style guide](https://developers.google.com/style) for its style and formatting.
-{% endhint %}
+{% embed url="https://www.youtube.com/watch?v=gVqGCqKVM08" %}
+Get an overview of Substreams
+{% endembed %}
diff --git a/docs/README_old.md b/docs/README_old.md
@@ -0,0 +1,68 @@
+---
+description: StreamingFast Substreams documentation
+---
+
+# Substreams
+
+## Welcome to Substreams documentation
+
+Substreams is a powerful blockchain indexing technology, developed for [The Graph Network](https://thegraph.com).
+
+> Substreams enables developers to write Rust modules, composing data streams alongside the community, and provides extremely high-performance indexing by virtue of parallelization, in a streaming-first fashion.
+>
+> Substreams have all the benefits of StreamingFast Firehose, like low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling.
+
+### Where to start
+
+Learn about Substreams in a short, dense 25-minute intro and understand its impact on the blockchain ecosystem.
+
+{% embed url="https://www.youtube.com/watch?v=K-nhC2FCB5k" %}
+A walkthrough of Firehose features, Substreams modules, including a sample Rust module and StreamingFast's vision.
+{% endembed %}
+
+Learn about the benefits of Substreams, and how it compares to otheressential facts about Substreams through [reading the Benefits and comparison](concepts-and-fundamentals/benefits.md).
+
+The primary ways to use Substreams include:
+
+* [Installing the `substreams` CLI](getting-started/installing-the-cli.md)
+* [Going through the Quickstart](getting-started/quickstart.md)
+
+After installing Substreams and reviewing the Quickstart:
+
+* You can [learn more about ](developers-guide/modules/)modules, and then [study the Developer's guide](developers-guide/overview.md).
+
+Find pre-built Substreams by using the following resources:
+
+* The [Substreams Template](https://github.com/streamingfast/substreams-template) helps expedite the process of getting you up and running.
+* A [list of maintained Substreams examples](reference-and-specs/examples.md)
+
+### Network model diagram
+
+<img src=".gitbook/assets/substreams.excalidraw.svg" alt="" class="gitbook-drawing">
+
+**You can view Substreams from two perspectives** as illustrated in the high-level visual diagram. It can be viewed through the perspective of the **Substreams engine** itself and also the perspective of the **end-user developer and consumer**.
+
+### Community
+
+Substreams is an open source community effort, so feel free to suggest new topics, report issues, and provide feedback. Contribute through GitHub [pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests).
+
+* [StreamingFast Discord](https://discord.gg/mYPcRAzeVN)
+* [The Graph Discord](https://discord.gg/vtvv7FP)
+* [StreamingFast on Twitter](https://twitter.com/streamingfastio)
+* [StreamingFast on YouTube](https://www.youtube.com/c/streamingfast)
+
+### Contributing
+
+For additional information, [refer to the general StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md).
+
+### License
+
+Substreams uses the [Apache 2.0](../LICENSE/) license.
+
+### Disclaimer
+
+The content in the Substreams documentation was created through StreamingFast's full effort. It is up to the reader to validate the accuracy of all content presented. Substreams is in active development and, at times, the associated documentation becomes outdated. [Contact StreamingFast](https://discord.gg/mYPcRAzeVN) to report problems or service interruptions.
+
+{% hint style="info" %}
+**Note**: The Substreams documentation uses the [Google developer documentation style guide](https://developers.google.com/style) for its style and formatting.
+{% endhint %}
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -2,80 +2,59 @@
 
 * [Substreams](README.md)
 
-## Quick Access
-
-* [Glossary](glossary/glossary.md)
-* [Change log](release-notes/change-log.md)
-
-## Concepts & Fundamentals
-
-* [Benefits and comparisons](concepts-and-fundamentals/benefits.md)
-* [Modules basics](concepts-and-fundamentals/modules.md)
-* [Fundamentals](concepts-and-fundamentals/fundamentals.md)
-
-## Getting Started
-
-* [Installing the Substreams CLI](getting-started/installing-the-cli.md)
-* [Quickstart](getting-started/quickstart.md)
-
-## Cheatsheet
-
-* [EVM](cheatsheet/evm/overview.md)
-  * [Blocks](cheatsheet/evm/blocks.md)
-  * [Transactions](cheatsheet/evm/transactions.md)
-  * [Logs](cheatsheet/evm/logs.md)
+## Documentation
+
+* [Consume Substreams](new/consume/consume.md)
+  * [Install the CLI](new/common/installing-the-cli.md)
+  * [Authentication](new/common/authentication.md)
+  * [Packages](new/common/packages.md)
+  * [Deployable Services](new/common/deployable-services.md)
+  * [Substreams:SQL](new/consume/sql/sql.md)
+    * [Deployable Services](new/consume/sql/deployable-services/introduction.md)
+      * [Remote Service](new/consume/sql/deployable-services/remote-service.md)
+      * [Local Service](new/consume/sql/deployable-services/local-service.md)
+    * [SQL Sink](new/consume/sql/sql-sink.md)
+  * [Substreams:Stream](new/consume/stream/stream.md)
+    * [JavaScript](new/consume/stream/javascript.md)
+  * [Substreams:Subgraph](new/consume/subgraph/subgraph.md)
+* [Develop Substreams](new/develop/develop.md)
+  * [Install the CLI](new/common/installing-the-cli.md)
+  * [Authentication](new/common/authentication.md)
+  * [Quickstart](new/develop/init-project.md)
+  * [Manifest & Modules](new/common/manifest-modules.md)
+    * [Module types](new/develop/modules/types.md)
+    * [Inputs](new/develop/modules/inputs.md)
+    * [Output](new/develop/modules/outputs.md)
+    * [Module handlers](new/develop/modules/setting-up-handlers.md)
+    * [Module handler creation](new/develop/modules/writing-module-handlers.md)
+    * [Keys in stores](new/develop/modules/keys-in-stores.md)
+    * [Dynamic data sources](new/develop/modules/dynamic-data-sources.md)
+    * [Aggregation Windows](new/develop/modules/aggregation-windows.md)
+  * [Protobuf Schemas](new/develop/creating-protobuf-schemas.md)
+  * [Packages](new/common/packages.md)
+  * [Run a Substreams](new/common/running-substreams.md)
+  * [Deployable Services](new/common/deployable-services.md)
+  * [Rust Dependencies](new/develop/rust-crates.md)
+  * [Parameterized Modules](new/develop/parameterized-modules.md)
+  * [Chain-Specific Extensions](new/develop/chain-specific/chain-specifi.md)
+    * [EVM Chain-Specific Extensions](new/develop/chain-specific/evm/evm.md)
+      * [Making eth_calls](new/develop/chain-specific/evm/eth-calls.md)
+  * [Architecture](new/develop/architecture.md)
 
 ## Tutorials
-
-* [From Ethereum address to SQL](tutorials/from-ethereum-address-to-SQL.md)
-* [Substreams:SQL Deployable Service](tutorials/substreams-sql.md)
-* [Exploring Ethereum](tutorials/exploring-ethereum/overview.md)
-  * [Mapping Blocks](tutorials/exploring-ethereum/map_block_meta_module.md)
-  * [Filtering Transactions](tutorials/exploring-ethereum/map_filter_transactions_module.md)
-  * [Retrieving Events of a Smart Contract](tutorials/exploring-ethereum/map_contract_events_module.md)
-* [Making eth_calls](tutorials/eth-calls.md)
-* [Rust](tutorials/rust/overview.md)
-  * [The Option struct](tutorials/rust/option.md)
-  * [The Result struct](tutorials/rust/result.md)
-
-## Developer's Guide
-
-* [Overview](developers-guide/overview.md)
-* [Dependency installation](developers-guide/installation-requirements.md)
-* [Manifest](developers-guide/creating-your-manifest.md)
-* [Protobuf schemas](developers-guide/creating-protobuf-schemas.md)
-* [Modules](developers-guide/modules/README.md)
-  * [Module types](developers-guide/modules/types.md)
-  * [Inputs](developers-guide/modules/inputs.md)
-  * [Output](developers-guide/modules/outputs.md)
-  * [Module handlers](developers-guide/modules/setting-up-handlers.md)
-  * [Module handler creation](developers-guide/modules/writing-module-handlers.md)
-* [Running Substreams](developers-guide/running-substreams.md)
-* [GUI](developers-guide/gui.md)
-* [Parallel Execution](developers-guide/parallel-execution.md)
-* [Sink targets](developers-guide/sink-targets/README.md)
-  * [Substreams-powered subgraph](developers-guide/sink-targets/substreams-powered-subgraph.md)
-  * [Files](developers-guide/sink-targets/substreams-sink-files.md)
-  * [Key/value store](developers-guide/sink-targets/substreams-sink-kv.md)
-  * [SQL (Postgres, Clickhouse, others)](developers-guide/sink-targets/substreams-sink-sql.md)
-  * [Prometheus](developers-guide/sink-targets/substreams-sink-prometheus.md)
-  * [MongoDB](developers-guide/sink-targets/substreams-sink-mongodb.md)
-  * [Custom Sink Using JavaScript](developers-guide/sink-targets/custom-sink-js.md)
-* [Cookbook](developers-guide/cookbook.md)
-  * [Dynamic data sources](developers-guide/cookbook/dynamic-data-sources.md)
-  * [Factory parameterization](developers-guide/cookbook/factory-params.md)
-  * [Advanced parameters](developers-guide/cookbook/advanced-params.md)
-  * [Keys in stores](developers-guide/cookbook/keys-in-stores.md)
-  * [Aggregation windows](developers-guide/cookbook/aggregation-windows.md)
-  * [Subgraph entity changes](developers-guide/cookbook/entity-changes.md)
+* [EVM](new/tutorials/evm/evm.md)
+  * [Exploring Ethereum](new/tutorials/evm/exploring-ethereum/exploring-ethereum.md)
+    * [Mapping Blocks](new/tutorials/evm/exploring-ethereum/map_block_meta_module.md)
+    * [Filtering Transactions](new/tutorials/evm/exploring-ethereum/map_filter_transactions_module.md)
+    * [Retrieving Events of a Smart Contract](new/tutorials/evm/exploring-ethereum/map_contract_events_module.md)
+* [Rust](new/tutorials/rust/overview.md)
 
 ## Reference & Specs
 
-* [Chains and endpoints](reference-and-specs/chains-and-endpoints.md)
-* [Substreams CLI reference](reference-and-specs/command-line-interface.md)
-* [Authentication](reference-and-specs/authentication.md)
-* [Manifests](reference-and-specs/manifests.md)
-* [Packages](reference-and-specs/packages.md)
-* [Rust crates](reference-and-specs/rust-crates.md)
-* [Examples](reference-and-specs/examples.md)
-* [FAQ](reference-and-specs/faq.md)
+* [Chains and endpoints](new/references/chains-and-endpoints.md)
+* [Substreams CLI reference](new/references/command-line-interface.md)
+* [Manifests Reference](new/references/manifests.md)
+* [GUI Reference](new/references/gui.md)
+* [Glossary](new/references/glossary.md)
+* [Change log](new/references/change-log.md)
+* [FAQ](new/references/faq.md)