In this example, we will launch the block-meta substream, sink it to a key-value store and launch a Generic service gRPC API and connect it to a webpage.
Learn about WasmEdge from its Quick Start Guide, or simply run the following to install.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- --version 0.11.2Note If you use
zsh, the final installation instructions talks about sourcing"$HOME/.zprofilebut it seems this file is not created properly in all cases. If it's the case, addsource "$HOME/.wasmedge/env"at the end of your.zshrcfile.
buf command v1.11.10 or later https://docs.buf.build/installation
brew install bufbuild/buf/bufSee https://nodejs.org for installation instructions.
Get from the Releases tab, or from source:
go install -v github.com/streamingfast/substreams-sink-kv/cmd/substreams-sink-kv@latestThe block-meta Substreams tracks the first and last block of every month since genesis block. The Substreams has a map module with an output type of sf.substreams.sink.kv.v1.KVOperations
...
- name: kv_out
kind: map
inputs:
- store: store_block_meta_start
mode: deltas
- store: store_block_meta_end
mode: deltas
output:
type: proto:sf.substreams.sink.kv.v1.KVOperations
...You can see the full substreams.yaml here
The module outputs KVOperations that the substreams-sink-kv will apply to key-value-store. The implementation details can be found here
use substreams::proto;
use substreams::store::{self, DeltaProto};
use substreams_sink_kv::pb::kv::KvOperations;
use crate::pb::block_meta::BlockMeta;
pub fn block_meta_to_kv_ops(ops: &mut KvOperations, deltas: store::Deltas<DeltaProto<BlockMeta>>) {
use substreams::pb::substreams::store_delta::Operation;
for delta in deltas.deltas {
match delta.operation {
Operation::Create | Operation::Update => {
let val = proto::encode(&delta.new_value).unwrap();
ops.push_new(delta.key, val, delta.ordinal);
}
Operation::Delete => ops.push_delete(&delta.key, delta.ordinal),
x => panic!("unsupported opeation {:?}", x),
}
}
}Note To connect to Substreams you will need an authentication token, follow this guide to obtain one,
You can run the substreams-sink-kv inject mode.
# Required only on MacOS to properly instruct the 'substreams-sink-kv' where to find the WasmEdge library
export DYLD_LIBRARY_PATH=$LIBRARY_PATH
substreams-sink-kv inject mainnet.eth.streamingfast.io:443 "badger3://$(pwd)/badger_data.db" substreams.yamlNote You can also use the
inject.shscripts which contains the call above
The inject mode is running the block-meta substream and applying the KVOperation to a local badger -b that is located here ./badger_data.db.
After a few minutes of sinking your local badger-db should contains keys. You can close the inject process.
We can introspect the store with our kvdb CLI
kvdb read prefix kmonth:first --dsn "badger3://$(pwd)/badger_data.db" --decoder="proto://./proto/[email protected]_meta.v1.BlockMeta"You should get an output like this
keys with prefix: kmonth:first
kmonth:first:197001 -> {"hash":"1OVnQPh2rvjAELhqQNX1Z0WhGNCQajTmmuyMDbHLj6M=","parentHash":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","timestamp":"1970-01-01T00:00:00Z"}
kmonth:first:201507 -> {"number":"1","hash":"iOltRTe+pNnAXRJUmQezJWHTvzH0Wq5zTNwRnxNAbLY=","parentHash":"1OVnQPh2rvjAELhqQNX1Z0WhGNCQajTmmuyMDbHLj6M=","timestamp":"2015-07-30T15:26:28Z"}
kmonth:first:201508 -> {"number":"13775","hash":"Lc7K1M8gedGBacoFvCHnugrdcTK5OCmEdg9D8nYb2CI=","parentHash":"q6q7j4t/f6B2aPs4/VoI2pgUzYrRink+VO72+pt5SrQ=","timestamp":"2015-08-01T00:00:03Z"}
kmonth:first:201509 -> {"number":"170395","hash":"BcQ5Q4WDvwWinOLmvmBRow+93ncq4UPzlyaoFzETBas=","parentHash":"PrACXrHJMjr8l/zN8E4mucPsxTEVMnAj9uog69E75jo=","timestamp":"2015-09-01T00:00:20Z"}
kmonth:first:201510 -> {"number":"314573","hash":"Po8RfewImJMcUEK0KDdDzORKt3mqx2VihhfA3EyRvAI=","parentHash":"Sek9sYxwxk2MuiJI4V8H/D6IX/unSxICfVYVBP9hpx0=","timestamp":"2015-10-01T00:00:17Z"}
kmonth:first:201511 -> {"number":"470668","hash":"1AB3cXZtMx5JL2KLz4dCBpjq+lsZEf6zgHd14+u+z1k=","parentHash":"6Kq+SBHeXyRdZ+/rC6he5QZWKHVPCoo0ebbgyUuL6H4=","timestamp":"2015-11-01T00:00:08Z"}
kmonth:first:201512 -> {"number":"622214","hash":"fw3ZOpMrUo8mqZReGkt+SBfnpv0aiPkKF2qrdmZn27o=","parentHash":"cPTq4v4Q7Ys5ivJjdiaxEjES4SIKRkZV238e3LhbQFU=","timestamp":"2015-12-01T00:00:01Z"}
...The Generic Query service is a Connect-Web protocol (gRPC-compatible). It exposes a browser and gRPC-compatible APIs.
The API is defined in protobuf here.
Launch the API, this starts the Connect-Web server
export DYLD_LIBRARY_PATH=$LIBRARY_PATH
substreams-sink-kv serve "badger3://$(pwd)/badger_data.db" substreams.yaml --listen-addr=":8080"Note You can also use the
serve.shscripts which contains the call above
We create a straightforward web app template:
npm create vite@latest -- connect-web-example --template react-ts
cd connect-web-example/
npm install
npm install --save-dev @bufbuild/protoc-gen-connect-web @bufbuild/protoc-gen-es
npm install @bufbuild/connect-web @bufbuild/protobufCreate the buf.gen.yaml, that will configure buf to generate our typescript files based on the proto definitions
version: v1
plugins:
- plugin: es
- plugin: connect-web- Add script line to generate
typescriptfiles from theprotofiles
# package.json
# "scripts": {
# ...
"buf:generate": "buf generate ../proto/substreams/sink/kv/v1 && buf generate ./proto",- Generate code:
npm run buf:generate to generate the following files under gen/ : kv_pb.ts, read_connectweb.ts, read_pb.ts
- Create client from KV in App.tsx:
import {
createConnectTransport,
createPromiseClient,
} from "@bufbuild/connect-web";
import { Kv } from "../gen/read_connectweb";
const transport = createConnectTransport({
baseUrl: "http://localhost:8000",
});
const client = createPromiseClient(Kv, transport);- Call client from button action:
const response = await client.get({
key: inputValue,
});
// then display `response.value`, typed as 'Uint8Array'- Use our generated 'block_meta_pb' protobuf bindings to decode the value:
import { BlockMeta } from "../gen/block_meta_pb";
const blkmeta = BlockMeta.fromBinary(response.value);
output = JSON.stringify(blkmeta, null, 2);The rest is just formatting, error-handling and front-end stuff ...
Lastly, we need to run our frontend in another terminal from our API
npm run dev- Connect-web code generation https://connect.build/docs/web/generating-code