Skip to content

Commit

Permalink
Merge pull request #569 from yash251/yash251/cleanup
Browse files Browse the repository at this point in the history
cleanup
  • Loading branch information
Giuliano-1 authored Dec 18, 2024
2 parents 1d67284 + d059920 commit 54a4055
Show file tree
Hide file tree
Showing 35 changed files with 100 additions and 106 deletions.
8 changes: 4 additions & 4 deletions docs/new/explanation/getting-started.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Getting Started with Substreams

Integrating Substreams can be quick and easy. This guide will help you get started with consuming ready-made Substreams packages or developing your own. Substreams are permissionaless. Grab a key [here](https://thegraph.market/), no personal information required, and start streaming on-chain data.
Integrating Substreams can be quick and easy. This guide will help you get started with consuming ready-made Substreams packages or developing your own. Substreams are permissionless. Grab a key [here](https://thegraph.market/), no personal information required, and start streaming on-chain data.

# Build

Expand All @@ -25,10 +25,10 @@ If you can't find a Substreams package that meets your specific needs, you can d
- [Injective](../tutorials/cosmos-compatible/injective.md)
- [Mantra](../tutorials/cosmos-compatible/mantra.md)

To build and optimize your Substreams from zero, use the minimal path within the [Dev Container](../references/devcontainer-ref.md) to setup your environment and follow the [How-To Guides](./how-to-guides/develop-your-own-substreams/develop-your-own-substreams.md).
To build and optimize your Substreams from zero, use the minimal path within the [Dev Container](../references/devcontainer-ref.md) to setup your environment and follow the [How-To Guides](../how-to-guides/develop-your-own-substreams/develop-your-own-substreams.md).

## Learn

- **Substreams Reliability Guarantees**: With a simple reconnection policy, Substreams guarantees you'll _[Never Miss Data](../references/reliability-guarantees.md)_.
- **Substreams Architecture**: For a deeper understanding of how Substreams works, explore the [architectural overview](architecture.md) of the data service.
- **Supported Networks**: Check-out which endpoints are supported [here](./references/chains-and-endpoints.md).
- **Substreams Architecture**: For a deeper understanding of how Substreams works, explore the [architectural overview](../references/architecture.md) of the data service.
- **Supported Networks**: Check-out which endpoints are supported [here](../references/chains-and-endpoints.md).
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
The [BlockStats Susbtreams](https://github.com/streamingfast/substreams-cosmos-block-stats) is a very basic Substreams, extracting data from the Injective blockchain.
The [BlockStats Substreams](https://github.com/streamingfast/substreams-cosmos-block-stats) is a very basic Substreams, extracting data from the Injective blockchain.

{% hint style="success" %}
**Tip**: This tutorial teaches you how to build a Substreams from scratch.

Remember that you can auto-generate your Substreams module by usig the [code-generation tools](../../../getting-started/injective/injective-first-sql.md).
Remember that you can auto-generate your Substreams module by using the [code-generation tools](../../../getting-started/injective/injective-first-sql.md).
{% endhint %}

## Before You Begin
Expand Down Expand Up @@ -58,7 +58,7 @@ modules:
type: proto:cosmos.v1.BlockStats # 6.
```
1. The `network` field specifies which network is the Substreams going to be executed on.
2. Import the [Cosmos Block Protobuf](https://github.com/streamingfast/firehose-cosmos/blob/develop/cosmos/pb/sf/cosmos/type/v1/block.pb.go#L75), which gives you access to the blockchain data.
2. Import the [Cosmos Block Protobuf](https://github.com/streamingfast/firehose-cosmos/blob/develop/cosmos/pb/sf/cosmos/type/v2/block.pb.go#L75), which gives you access to the blockchain data.
3. Import the user-defined Protobuf schemas (i.e. the outputs of your Substreams).
4. Define a module. `block_to_stats`, which will be mapped to the `block_to_stats` Rust function in the source code.
5. Define the inputs of the module. In this case, the `Block` Cosmos Protobuf.
Expand Down Expand Up @@ -92,7 +92,7 @@ substreams gui substreams.yaml block_to_stats \
--start-block=64987400 --stop-block=+1000
```

Review the [GUI Reference](../../../references/gui.md) to get more information on how to use this utility.
Review the [GUI Reference](https://docs.substreams.dev/reference-material/installing-the-cli/command-line-interface#gui) to get more information on how to use this utility.

## Inspect the Code

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
The [Injective Foundational Substreams](https://github.com/streamingfast/substreams-foundational-modules/injective-common) contains Substreams modules, which retrieve fundammental data on the Injective blockchain.
The [Injective Foundational Substreams](https://github.com/streamingfast/substreams-foundational-modules/injective-common) contains Substreams modules, which retrieve fundamental data on the Injective blockchain.

You can use the Injective Foundational Modules as the input for your Substreams or subgraph.

Expand Down Expand Up @@ -59,15 +59,15 @@ modules:
output:
type: proto:sf.substreams.cosmos.v1.EventList
doc: |
`filtered_events` reads from `all_events` and applies a filter on the event types, only outputing the events that match the filter.
`filtered_events` reads from `all_events` and applies a filter on the event types, only outputting the events that match the filter.
The only operator that you should need to use this filter is the logical or `||`, because each event can only match one type.
```
1. The `all_transactions` module provides access to all the transactions of the Injective blockchain.
It receives a raw Injective block object as input (`sf.cosmos.type.v2.Block`), and outputs a list of transactions object (`sf.substreams.cosmos.v1.TransactionList`).
2. The `all_events` module provides access to all the events in the Injective blockchain.
It receives a raw Injective block as input (`sf.cosmos.type.v2.Block`), and outputs a list of events object (`sf.substreams.cosmos.v1.EventList`).
3. The `index_events` module uses the `all_events` module to create a cache where events are sorted based on their `type` field. This cache helps in the performance of the module. You can read more about _index modules_ in the [correspoding documentation](../../../develop/indexes).
4. The `filtered_events` allows you to use the `index_events` module (i.e. using the cache of events), to filter only the event types you are interested in.
1. The `index_events` module uses the `all_events` module to create a cache where events are sorted based on their `type` field. This cache helps in the performance of the module. You can read more about _index modules_ in the [corresponding documentation](../../../../references/substreams-components/modules/indexes.md).
2. The `filtered_events` allows you to use the `index_events` module (i.e. using the cache of events), to filter only the event types you are interested in.
The string parameter passed as input is used to specify which events you want to consume.

## Use The Foundational Modules
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ The [USDT Exchanges Volume Subgraph](https://github.com/streamingfast/injective-
{% hint style="success" %}
**Tip**: This tutorial teaches you how to build a Substreams from scratch.

Remember that you can auto-generate your Substreams module by usig the [code-generation tools](../../../getting-started/injective/injective-first-sps.md).
Remember that you can auto-generate your Substreams module by using the [code-generation tools](../../../getting-started/injective/injective-first-sps.md).
{% endhint %}

The subgraph uses the [Substreams triggers](../../../consume/subgraph/triggers.md) to import data from the Injective foundational modules.
Expand Down Expand Up @@ -136,15 +136,15 @@ export function handleEvents(bytes: Uint8Array): void { // 1.
3. Load the `USDTExchangeVolume` subgraph entity, which will store the historical volume.
If it is the first trade, then the entity will not exist, and it must be created.
4. Iterate over the events and verify that the event type is `wasm` (`type == wasm`). This should be already filtered by the Substreams, but it is also nice to re-check it.
5. Iterate over the attributes of every event, finding out the neccesary information (contract address, action, ask amount, offer amount...).
5. Iterate over the attributes of every event, finding out the necessary information (contract address, action, ask amount, offer amount...).
6. Verify that the contract where the event is executed corresponds to the `INJ-USDT` pair in the Dojo DEX.
7. Update the entity.

## Deploy to a Local Graph Node

You can test your Substreams-powered Subgraph by deploying to a local Graph Node set-up. Take a look at the the [Graph Node Local Development tutorial](../../graph-node/local-development.md), which provides information on how to spin up a local environment for Graph Node.
You can test your Substreams-powered Subgraph by deploying to a local Graph Node set-up. Take a look at the the [Graph Node Local Development tutorial](../../../../references/graph-node/local-development.md), which provides information on how to spin up a local environment for Graph Node.

First, clone the [Substreams Development Environment GitHub respository](https://github.com/streamingfast/substreams-dev-environment) and move to the `graph-node` folder. Execute the `start.sh` command with the Injective information (make sure you have Docker running in your computer).
First, clone the [Substreams Development Environment GitHub repository](https://github.com/streamingfast/substreams-dev-environment) and move to the `graph-node` folder. Execute the `start.sh` command with the Injective information (make sure you have Docker running in your computer).

```bash
./start.sh injective-mainnet https://mainnet.injective.streamingfast.io:443
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
These how-to guides walk through creating a Substreams that uses raw blockchain data to index your dapp. The application includes using Rust and Protobuf to extract, transform, and load the data.

{% hint style="warning" %}
**Important**_:_ These how-to guides are in-depth walkthroughs of builing highly performance indexers for your dapp. Less experienced users may want to reference the [Tutorials](../../tutorials/intro-to-tutorials.md) for a quick start.
**Important**_:_ These how-to guides are in-depth walkthroughs of building highly performance indexers for your dapp. Less experienced users may want to reference the [Tutorials](../../tutorials/intro-to-tutorials.md) for a quick start.
{% endhint %}

Choose your ecosystem to get started:

- [Solana](./solana/solana.md)
- [EVM](./evm/exploring-ethereum/exploring-ethereum.md)
- [Solana](./solana/solana.md)
- [Cosmos](./cosmos/injective/block-stats.md)
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Before moving forward, make sure to reference the minimal path within the [Dev C
The `https://github.com/streamingfast/substreams-explorers` GitHub repository contains all the Substreams Explorers currently available. You can simply clone the repository:

```
$ git clone https://github.com/streamingfast/substreams-explorers
git clone https://github.com/streamingfast/substreams-explorers
```
### Substreams Basics

Expand Down Expand Up @@ -83,5 +83,4 @@ The [CLI reference](../../../../references/cli/command-line-interface.md) lets y

### Substreams Components Reference

The [Components Reference](../../../../references/substreams-components/) dives deeeper into navigating the `substreams.yaml`.

The [Components Reference](../../../../references/substreams-components/packages.md) dives deeper into navigating the `substreams.yaml`.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Let's run the Substreams first, and then go through the code.

### Running the Substreams

Running a Substreams usually requires three steps: generating the Rust Protobufs, building the WASM container, and using the Substream CLI to start the streaming. Make sure to run the following commands in the `substreams-explorer/ethereum-explorer` folder:
Running a Substreams usually requires three steps: generating the Rust Protobufs, building the WASM container, and using the Substreams CLI to start the streaming. Make sure to run the following commands in the `substreams-explorer/ethereum-explorer` folder:

1. **Generate the Protobuf objects:** The `.proto` files define a data model regardless of any programming language. However, in order to use this model in your Rust application, you must generate the corresponding Rust data structures. Note that running `make protogen` is only necessary when making updates to any file in the proto folder.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,3 @@ Take a look at the _Develop Substreams_ section for more information on how to s
## The Solana Explorer

The Solana explorer includes several modules showcasing what Solana data you can extract with Substreams (it's easy and fast!). In the following sections, you will find out about the different functions you can use to easily get started with Solana.

Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,9 @@ fn map_filter_instructions(params: String, blk: Block) -> Result<Instructions, s
1. The `parse_filters_from_params` function parses the parameters passed to the module.
In this example, the parameter passed is defined in the `substreams.yaml` file as `program_id=Stake11111111111111111111111111111111111111`.
2. Iterate over the transactions of the blocks.
3. Extract the [Message](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto#L32) object, which contains relevant information, such as the instructions of the trasaction.
3. Extract the [Message](https://github.com/streamingfast/firehose-solana/blob/develop/proto/sf/solana/type/v1/type.proto#L32) object, which contains relevant information, such as the instructions of the transaction.
4. Get accounts of the transaction (the `resolved_accounts()` method contains also accounts stored in the [Address Lookup Tables](https://docs.solana.com/developing/lookup-tables)).
5. Iterave over the instructions.
5. Iterate over the instructions.
6. Use the `apply_filter` function to only keep instruction where `program_id=Stake11111111111111111111111111111111111111`.
7. Create an `Instruction` object, which will be the output of the Substreams.
This object is declared as a Protobuf in the `proto` folder of the project.
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Now, you can run the Substreams by using the `substreams gui` command. To avoid

```bash
substreams gui ./substreams.yaml \
map_filter_instructions -e mainnet.sol.streamingfast.io:443 \
map_filter_transactions -e mainnet.sol.streamingfast.io:443 \
--start-block 153000028 --stop-block +1
```

Expand All @@ -31,7 +31,7 @@ params:
### Run the Package From the Substreams Registry
The Solana Explorer is also available as a Substreams package in the [Substreams Regitry](https://substreams.dev). You can simply run it:
The Solana Explorer is also available as a Substreams package in the [Substreams Registry](https://substreams.dev). You can simply run it:
```bash
substreams gui https://spkg.io/streamingfast/solana-explorer-v0.2.0.spkg \
Expand All @@ -41,7 +41,7 @@ substreams gui https://spkg.io/streamingfast/solana-explorer-v0.2.0.spkg \

## Inspect the Code

The `map_flter_transaction.rs` file contains the source of the module. The output emitted by the module is defined as a Protobuf in the `/proto/transactions.proto` file of the project.
The `map_filter_transaction.rs` file contains the source of the module. The output emitted by the module is defined as a Protobuf in the `/proto/transactions.proto` file of the project.

```protobuf
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ The Solana Token Tracker Substreams allows you to extract transfers from Solana

The Solana Token Tracker Substreams requires medium to advanced Substreams knowledge. If this is the first time you are using Substreams, make sure you:

- Read the [Develop Substreams](../../../develop/develop.md) section, which will teach you the basics of the developing Substreams modules.
- Read the [Develop Substreams](../../../../tutorials/intro-to-tutorials.md) section, which will teach you the basics of the developing Substreams modules.
- Complete the [Explore Solana](../explore-solana/explore-solana.md) tutorial, which will assist you in understanding the main pieces of the Solana Substreams.

If you already have the required knowledge, clone the [Solana Token Tracker GitHub repository](https://github.com/streamingfast/solana-token-tracker). You will go through the code in the following steps.
Expand All @@ -27,7 +27,7 @@ params:
map_solana_token_events: "token_contract=EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v&token_decimals=6"
```
The module receives two inputs (defined in the `intputs` section of the YAML):
The module receives two inputs (defined in the `inputs` section of the YAML):
- A string containing a couple of parameters: this parameter is defined in the `params` section of the YAML, and defines the token that you want to extract data from: `token_contract=EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v&token_decimals=6`
`token_contract=EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v` is the address of the USDC contract in Solana mainnet and `token_decimals=6` is the number of decimals used the USDC token.
- A raw Solana block.
Expand Down Expand Up @@ -105,7 +105,7 @@ pub fn map_solana_token_events(params: String, block: Block) -> Result<Output, E
4. _Unwrap_ the transaction if it is available.
5. _Unwrap_ the transaction message.
6. _Unwrap_ the transaction metadata.
7. Iterave over the instructions contained within the transaction.
7. Iterate over the instructions contained within the transaction.
8. For every instruction, call the `process_compiled_instruction(...)` function to process the instruction further.

- The `process_compiled_instruction(...)` function is defined in the `util.rs` file.
Expand Down Expand Up @@ -193,7 +193,7 @@ Essentially, you are matching every top-level instruction with its corresponding
5. You only keep Token Program inner instructions.
6. You process every Token Program inner instruction found further by calling the `process_token_instruction(...)`.

Once you have identified all the Token Program instructions, the `proces_token_instruction(...)` function extracts transfer or mint data from these instructions. To easily extract data from a Token Program instruction, the Substreams relies on the `substreams-solana-program-instructions` Rust crate, which provides useful helper functions.
Once you have identified all the Token Program instructions, the `process_token_instruction(...)` function extracts transfer or mint data from these instructions. To easily extract data from a Token Program instruction, the Substreams relies on the `substreams-solana-program-instructions` Rust crate, which provides useful helper functions.

```rust
fn process_token_instruction(
Expand Down
Loading

0 comments on commit 54a4055

Please sign in to comment.