Skip to content

Latest commit

 

History

History
127 lines (90 loc) · 9 KB

new-chain-integration.mdx

File metadata and controls

127 lines (90 loc) · 9 KB
Raw
title
New Chain Integration

Chains can bring subgraph support to their ecosystem by starting a new graph-node integration. Subgraphs are a powerful indexing tool opening a world of possibilities for developers. Graph Node already indexes data from the chains listed here. If you are interested in a new integration, there are 2 integration strategies:

  1. EVM JSON-RPC
  2. Firehose: All Firehose integration solutions include Substreams, a large scale streaming engine based off Firehose with native graph-node support, allowing for parallelized transforms.

Note that while the recommended approach is to developing a new Firehose for all new chains, it is a requirement for non-EVM chains.

Integration Strategies

1. EVM JSON-RPC

If the blockchain is EVM equivalent and the client/node exposes the standard EVM JSON-RPC API, Graph Node should be able to index the new chain.

Testing an EVM JSON-RPC

For Graph Node to be able to ingest data from an EVM chain, the RPC node must expose the following EVM JSON RPC methods:

  • eth_getLogs
  • eth_call (for historical blocks, with EIP-1898 - requires archive node)
  • eth_getBlockByNumber
  • eth_getBlockByHash
  • net_version
  • eth_getTransactionReceipt, in a JSON-RPC batch request
  • *trace_filter* (optionally required for Graph Node to support call handlers)

2. Firehose Integration

Firehose is a next generation extraction layer. It collects history in flat files and streams in real-time. Firehose technology replaces those polling API calls with a stream of data utilizing a push model that sends data to the indexing node faster. This helps increase the speed of syncing and indexing.

The primary method to integrate the Firehose into chains is to use an RPC polling strategy. Our polling algorithm will predict when a new block will arrive and increase the rate at which it checks for a new block near that time, making it a very low-latency and efficient solution. For help with the integration and maintenance of the Firehose contact the StreamingFast team. New chains and their integrators will appreciate the fork awareness and massive parallelized indexing capabilities that Firehose and Substreams brings to their ecosystem.

NOTE: All integrations done by the StreamingFast team include maintenance for the Firehose replication protocol into the chain's codebase. StreamingFast tracks any changes and releases binaries when you change code and when StreamingFast changes code. This includes releasing Firehose/Substreams binaries for the protocol, maintaining Substreams modules for the block model of the chain, and releasing binaries for the blockchain node with instrumentation if need be.

Specific Firehose Instrumentation for EVM (geth) chains

For the EVM chains, there exists a deeper level of data that can be achieved through the geth live-tracer, a collaboration between Go-Ethereum and StreamingFast, in building a high-throughput and rich transaction tracing system. The Live Tracer is the most comprehensive solution, resulting in EXTENDED block details. This enables new indexing paradigms, like pattern matching of events based on state changes, calls, parent call trees, or triggering of events based on changes to the actual variables in a smart contract.

NOTE: This improvement upon the Firehose requires chains make use of the EVM engine geth version 1.13.0 and up.

EVM considerations - Difference between JSON-RPC & Firehose

While the JSON-RPC and Firehose are both suitable for subgraphs, a Firehose is always required for developers wanting to build with Substreams. Supporting Substreams allows developers to build Substreams-powered subgraphs for the new chain, and has the potential to improve the performance of your subgraphs. Additionally, Firehose — as a drop-in replacement for the JSON-RPC extraction layer of graph-node — reduces by 90% the number of RPC calls required for general indexing.

  • All those getLogs call and roundtrips get replaced by a single stream arriving into the heart of graph-node and by running filters on the in-memory Firehose Block; a single block for all subgraphs it processes.

NOTE: A Firehose-based integration for EVM chains will still require Indexers to run the chain's archive RPC node to properly index subgraphs. This is due to the Firehose's inability to provide smart contract state typically accessible by the eth_call RPC method. (It's worth reminding that eth_calls are not a good practice for developers)

Graph Node Configuration

Configuring Graph Node is as easy as preparing your local environment. Once your local environment is set, you can test the integration by locally deploying a subgraph.

  1. Clone Graph Node

  2. Modify this line to include the new network name and the EVM JSON RPC compliant URL

    Do not change the env var name itself. It must remain ethereum even if the network name is different.

  3. Run an IPFS node or use the one used by The Graph: https://api.thegraph.com/ipfs/

Testing an EVM JSON-RPC by locally deploying a subgraph
  1. Install graph-cli
  2. Create a simple example subgraph. Some options are below:
    1. The pre-packed Gravitar smart contract and subgraph is a good starting point
    2. Bootstrap a local subgraph from any existing smart contract or solidity dev environment using Hardhat with a Graph plugin
  3. Adapt the resulting subgraph.yaml by changing dataSources.network to the same name previously passed on to Graph Node.
  4. Create your subgraph in Graph Node: graph create $SUBGRAPH_NAME --node $GRAPH_NODE_ENDPOINT
  5. Publish your subgraph to Graph Node: graph deploy $SUBGRAPH_NAME --ipfs $IPFS_ENDPOINT --node $GRAPH_NODE_ENDPOINT

Graph Node should be syncing the deployed subgraph if there are no errors. Give it time to sync, then send some GraphQL queries to the API endpoint printed in the logs.

Substreams-powered Subgraphs

For StreamingFast led Firehose/Substreams integrations, basic support for foundational Substreams modules (e.g. decoded transactions, logs and smart-contract events) and Substreams-powered-Subgraph codegen tools are included (check out Injective for an example).

There are two options to consume Substreams data through a subgraph:

  • Using Substreams triggers: Consume from any Substreams module by importing the Protobuf model and move all your logic into the subgraph. This method creates the subgraph entities directly in the subgraph
  • Using EntityChanges: By writing more of the logic into Substreams, you can consume the module-output directly into graph-node. In graph-node, you can use the Substreams data to create your subgraph entities.

It is really a matter of where you put your logic, in the subgraph or the Substreams. Consider the following example using a subgraph handler:

export function handleTransactions(bytes: Uint8Array): void {
    let transactions = assembly.eth.transaction.v1.Transactions.decode(bytes.buffer).trasanctions; // 1.
    if (transactions.length == 0) {
        log.info("No transactions found", []);
        return;
    }

    for (let i = 0; i < transactions.length; i++) { // 2.
        let transaction = transactions[i];

        let entity = new Transaction(transaction.hash); // 3.
        entity.from = transaction.from;
        entity.to = transaction.to;
        entity.save();
    }
}

The “handleTransactions” function is a subgraph handler that receives the raw Substreams bytes as parameter and decodes them into a “Transactions” object. Then, for every transaction, a new subgraph entity is created. For more information about Substreams triggers, visit the StreamingFast documentation or checkout community modules at substreams.dev.

NOTE: Keep in mind that having your logic in Substreams benefits from a parallelized model, whereas, Substreams triggers will be linearly consumed.