-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4203974
commit 71024b8
Showing
5 changed files
with
190 additions
and
61 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| project_name: substreams-sink-kv | ||
|
|
||
| release: | ||
| draft: true | ||
| github: | ||
| owner: streamingfast | ||
| name: substreams-sink-kv | ||
| name_template: '{{.Tag}}' | ||
| builds: | ||
| - id: substreams-sink-kv | ||
| goos: | ||
| - linux | ||
| - darwin | ||
| goarch: | ||
| - arm64 | ||
| - amd64 | ||
| targets: | ||
| - linux_amd64 | ||
| - darwin_amd64 | ||
| - darwin_arm64 | ||
| main: ./cmd/substreams-sink-kv | ||
| ldflags: -s -w -X main.version={{.Version}} | ||
| binary: substreams-sink-kv | ||
|
|
||
| archives: | ||
| # fireeth | ||
| - name_template: '{{ .Binary }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}' | ||
| id: substreams-sink-kv | ||
| builds: | ||
| - substreams-sink-kv | ||
| replacements: | ||
| amd64: x86_64 | ||
| darwin: macOS | ||
| linux: linux | ||
| format: tar.gz | ||
| files: | ||
| - LICENSE | ||
| - README.md | ||
| snapshot: | ||
| name_template: '{{ .Tag }}-next' | ||
| checksum: | ||
| name_template: checksums.txt | ||
| changelog: | ||
| filters: | ||
| exclude: | ||
| - '^docs:' | ||
| - '^test:' | ||
| sort: asc | ||
| dist: dist | ||
| signs: | ||
| - cmd: keybase | ||
| args: | ||
| - sign | ||
| - --infile | ||
| - $artifact | ||
| - --binary | ||
| - --outfile | ||
| - $signature | ||
| - --detached | ||
| signature: ${artifact}.sig | ||
| artifacts: checksum | ||
| env_files: | ||
| github_token: ~/.config/goreleaser/github_token | ||
| brews: | ||
| - name: substreams-sink-kv | ||
| ids: | ||
| - substreams-sink-kv | ||
| tap: | ||
| owner: streamingfast | ||
| name: homebrew-tap | ||
| commit_author: | ||
| name: goreleaserbot | ||
| email: [email protected] | ||
| homepage: "https://github.com/streamingfast/substreams-sink-kv" | ||
| description: "Substreams Sink tool to extract information to a kv store" | ||
| license: "Apache-2.0" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,73 +1,126 @@ | ||
| # substreams-sink-kv | ||
| # Substreams sink key-value store | ||
|
|
||
| This is a command line tool to quickly sync a substreams with a kv database. | ||
| It exposes this data through Connect-Web protocol (compatible with GRPC) | ||
| ## Description | ||
|
|
||
| `substreams-sink-kv` is a tool that allows developers to pipe data extracted from a blockchain into a key-value store and expose it through Connect-Web protocol (compatible with GRPC) | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Go installation and compiler | ||
| - Rust installation and compiler | ||
| - Cloned `substreams-sink-kv` repository | ||
| - A Substreams module prepared for a kv-sink | ||
| - Knowledge of blockchain and substreams development | ||
|
|
||
| ## Installation | ||
|
|
||
| * Clone this repository, then run `go install -v ./cmd/substreams-sink-kv` | ||
| * Check your installation by running `substreams-sink-kv --version` | ||
|
|
||
| > **Note** the binary file will be installed in your *GO_PATH*, usually `$HOME/go/bin`. Ensure that this folder is included in your *PATH* environment variable. | ||
| ## Running with an example substreams | ||
|
|
||
| > **Note** To connect to substreams you will need an authentication token, follow this [guide](https://substreams.streamingfast.io/reference-and-specs/authentication) to obtain one, | ||
| ### Quickstart | ||
| ```bash | ||
| substreams-sink-kv \ | ||
| run \ | ||
| "badger3://$(pwd)/badger_data.db" \ | ||
| mainnet.eth.streamingfast.io:443 \ | ||
| https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.4.0/substreams-eth-block-meta-v0.4.0.spkg \ | ||
| kv_out | ||
| ``` | ||
|
|
||
| 1. Install `substreams-sink-kv` (Installation from source required for now): | ||
| You should see output similar to this one: | ||
| ```bash | ||
| 2023-01-12T10:08:31.803-0500 INFO (sink-kv) starting prometheus metrics server {"listen_addr": "localhost:9102"} | ||
| 2023-01-12T10:08:31.803-0500 INFO (sink-kv) sink to kv {"dsn": "badger3:///Users/stepd/repos/substreams-sink-kv/badger_data.db", "endpoint": "mainnet.eth.streamingfast.io:443", "manifest_path": "https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.4.0/substreams-eth-block-meta-v0.4.0.spkg", "output_module_name": "kv_out", "block_range": ""} | ||
| 2023-01-12T10:08:31.803-0500 INFO (sink-kv) starting pprof server {"listen_addr": "localhost:6060"} | ||
| 2023-01-12T10:08:31.826-0500 INFO (sink-kv) reading substreams manifest {"manifest_path": "https://github.com/streamingfast/substreams-eth-block-meta/releases/download/v0.4.0/substreams-eth-block-meta-v0.4.0.spkg"} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) validating output store {"output_store": "kv_out"} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) resolved block range {"start_block": 0, "stop_block": 0} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) starting to listen on {"addr": "localhost:8000"} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) starting stats service {"runs_each": "2s"} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) no block data buffer provided. since undo steps are possible, using default buffer size {"size": 12} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) starting stats service {"runs_each": "2s"} | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) ready, waiting for signal to quit | ||
| 2023-01-12T10:08:32.186-0500 INFO (sink-kv) launching server {"listen_addr": "localhost:8000"} | ||
| 2023-01-12T10:08:32.187-0500 INFO (sink-kv) serving plaintext {"listen_addr": "localhost:8000"} | ||
| 2023-01-12T10:08:32.278-0500 INFO (sink-kv) session init {"trace_id": "a3c59bd7992c433402b70f9541565d2d"} | ||
| 2023-01-12T10:08:34.186-0500 INFO (sink-kv) substreams sink stats {"db_flush_rate": "10.500 flush/s (21 total)", "data_msg_rate": "0.000 msg/s (0 total)", "progress_msg_rate": "0.000 msg/s (0 total)", "block_rate": "0.000 blocks/s (0 total)", "flushed_entries": 0, "last_block": "None"} | ||
| 2023-01-12T10:08:34.186-0500 INFO (sink-kv) substreams sink stats {"progress_msg_rate": "16551.500 msg/s (33103 total)", "block_rate": "10941.500 blocks/s (21883 total)", "last_block": "#291883 (66d03f819dde948b297c8d582889246d7ba11a5b947335497f8716a7b608f78e)"} | ||
| ``` | ||
|
|
||
| ```bash | ||
| go install ./cmd/substreams-sink-kv | ||
| ``` | ||
| > **Note** Alternatively, you can simply run `./devel/local/start.sh` which runs this command for you. | ||
| 2. Add a 'map' module to your `substreams.yaml` with an output type of `proto:substreams.kv.v1.KVOperations`: | ||
| ## Running with your own substreams | ||
|
|
||
| ```yaml | ||
| modules: | ||
| - name: kv_out | ||
| kind: map | ||
| initialBlock: 0 | ||
| inputs: | ||
| - store: store_something | ||
| output: | ||
| type: proto:substreams.kv.v1.KVOperations | ||
| ``` | ||
| 1. Add a 'map' module to your `substreams.yaml` with an output type of `proto:sf.substreams.kv.v1.KVOperations`: | ||
|
|
||
| 3. Run the sink to a local 'badger' database | ||
| ```yaml | ||
| modules: | ||
| - name: kv_out | ||
| kind: map | ||
| inputs: | ||
| - store: your_store | ||
| output: | ||
| type: proto:substreams.kv.v1.KVOperations | ||
| ``` | ||
| > To connect to substreams you will need an authentication token, follow this [guide](https://substreams.streamingfast.io/reference-and-specs/authentication) to obtain one, | ||
| 2. Write the `kv_out` module in your substreams code (and recompile) | ||
|
|
||
| ```shell | ||
| substreams-sink-kv run \ | ||
| "badger3:///home/user/sf-data/my-badger.db" \ | ||
| "mainnet.eth.streamingfast.io:443" \ | ||
| "substreams.yaml" \ | ||
| kv_out | ||
| ``` | ||
| * See [substreams-eth-block-meta](https://github.com/streamingfast/substreams-eth-block-meta) for an example integration | ||
|
|
||
| ### Even quicker iteration | ||
|
|
||
| * Run `devel/local/start.sh` | ||
| 3. Run your substreams with the sink-kv, writing a local 'badger' database | ||
|
|
||
| ### Querying the data | ||
| > **Note** To connect to substreams you will need an authentication token, follow this [guide](https://substreams.streamingfast.io/reference-and-specs/authentication) to obtain one, | ||
|
|
||
| #### In command-line | ||
| ```shell | ||
| substreams-sink-kv run \ | ||
| "badger3:///$(pwd)/my-badger.db" \ | ||
| "mainnet.eth.streamingfast.io:443" \ | ||
| "substreams.yaml" \ | ||
| kv_out | ||
| ``` | ||
|
|
||
| (these commands have been tested while running the 'devel/local/start.sh' in another terminal) | ||
| ## Querying the data | ||
|
|
||
| ### In command-line | ||
|
|
||
| * Get single block data: `grpcurl --plaintext -d '{"key":"month:last:201511"}' localhost:8000 sf.substreams.sink.kv.v1.Kv/GetMany` | ||
| * Get many: `grpcurl --plaintext -d '{"keys":["day:first:20151201","day:first:20151202"]}' localhost:8000 sf.substreams.sink.kv.v1.Kv/GetMany` | ||
| * By prefix: `grpcurl --plaintext -d '{"prefix": "day:first:201511", "limit":31}' localhost:8000 sf.substreams.sink.kv.v1.Kv/GetByPrefix` | ||
| * Scan: `grpcurl --plaintext -d '{"begin": "day:first:201501", "exclusive_end": "day:first:2016", "limit":400}' localhost:8000 sf.substreams.sink.kv.v1.Kv/Scan` | ||
|
|
||
| #### From your browser | ||
|
|
||
| (this has been tested while running the 'devel/local/start.sh' in a terminal) | ||
| > **Note** These commands have been tested with `devel/local/start.sh` running in another terminal, which runs [substreams-eth-block-meta](https://github.com/streamingfast/substreams-eth-block-meta) | ||
| ### From your browser | ||
|
|
||
| * Run `npm run dev` from within `/connect-web-example` to see a Vite JS example connecting to the KV GRPC Connect-Web interface | ||
|
|
||
| ### Protobuf generation | ||
| > **Note** The 'connect-web-version' works best with `devel/local/start.sh` running in another terminal, which runs [substreams-eth-block-meta](https://github.com/streamingfast/substreams-eth-block-meta) | ||
|
|
||
|
|
||
| ## Protobuf generation | ||
|
|
||
| * Requires 'buf.build' with protoc-gen-go and protoc-gen-go-grpc (https://docs.buf.build/tour/generate-go-code) | ||
| * Run `cd proto && buf generate` | ||
|
|
||
| ## Cargo crate | ||
|
|
||
| Note: this crate is a stub, mostly to contain the protobuf bindings, feel free to use it or not. | ||
| A library containing the protobuf bindings and a few helpers has been published as a crate for your convenience. | ||
|
|
||
| * https://crates.io/crates/substreams-sink-kv | ||
|
|
||
| ### Example integration | ||
| See [substreams-eth-block-meta](https://github.com/streamingfast/substreams-eth-block-meta) for an example integration | ||
|
|
||
| ## Contributing | ||
|
|
||
| * See https://github.com/streamingfast/substreams-eth-block-meta | ||
| For additional information, [refer to the general StreamingFast contribution guide](https://github.com/streamingfast/streamingfast/blob/master/CONTRIBUTING.md). | ||
|
|
||
| ## License | ||
|
|
||
| The `substreams-sink-kv` tool [uses the Apache 2.0 license](https://github.com/streamingfast/substreams/blob/develop/LICENSE/README.md). | ||
|
|
||
| This is a command line tool to quickly sync a substreams with a kv database. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters