v1.1.12

Highlights

• Complete redesign of the progress messages:
  • Tier2 internal stats are aggregated on Tier1 and sent out every 500ms (no more bursts)
  • No need to collect events on client: a single message now represents the current state
  • Message now includes list of running jobs and information about execution stages
  • Performance metrics has been added to show which modules are executing slowly and where the time is spent (eth calls, store operations, etc.)

Upgrading client and server

Important

The client and servers will both need to be upgraded at the same time for the new progress messages to be parsed:

• The new Substreams servers will NOT send the old modules field as part of its progress message, only the new running_jobs, modules_stats, stages.
• The new Substreams clients will NOT be able to decode the old progress information when connecting to older servers.

However, the actual data (and cursor) will work correctly between versions. Only incompatible progress information will be ignored.

CLI

Changed

• Bumped substreams and substreams-ethereum to latest in substreams alpha init.

• Improved error message when <module_name> is not received, previously this would lead to weird error message, now, if the input is likely a manifest, the error message will be super clear.

Fixed

• Fixed compilation errors when tracking some contracts when using substreams alpha init.

Added

• substreams info now takes an optional second parameter <output-module> to show how the substreams modules can be divided into stages
• Pack command: added -c flag to allow overriding of certain substreams.yaml values by passing in the path of a yaml file.
  example yaml contents:

  package:
    name: my_custom_package_name 
  
  network: arbitrum-one
  initialBlocks:
    module_name_1: 123123123
  params:
    mod1: "custom_parameter"

Backend

Removed

• Removed Config.RequestStats, stats are now always enabled.

v1.1.11

Fixes

• Added metering of live blocks

v1.1.10

Backend changes

• Fixed/Removed: jobs would hang when config parameter StateBundleSize was different from SubrequestsSize. The latter has been removed completely: Subrequests size will now always be aligned with bundle size.
• Auth: added support for continuous authentication via the grpc auth plugin (allowing cutoff triggered by the auth system).

CLI changes

• Fixed params handling in gui mode

v1.1.9

Backend changes

• Massive refactoring of the scheduler: prevent excessive splitting of jobs, grouping them into stages when they have the same dependencies. This should reduce the required number of tier2 workers (2x to 3x, depending on the substreams).

• The tier1 and tier2 config have a new configuration StateStoreDefaultTag, will be appended to the StateStoreURL value to form the final state store URL, ex: StateStoreURL="/data/states" and StateStoreDefaultTag="v2" will make /data/states/v2 the default state store location, while allowing users to provide a X-Sf-Substreams-Cache-Tag header (gated by auth module) to point to /data/states/v1, and so on.

• Authentication plugin trust can now specify an exclusive list of allowed headers (all lowercase), ex: trust://?allowed=x-sf-user-id,x-sf-api-key-id,x-real-ip,x-sf-substreams-cache-tag

• The tier2 app no longer has customizable auth plugin (or any Modules), trust will always be used, so that tier can pass down its headers (e.g. X-Sf-Substreams-Cache-Tag). The tier2 instances should not be accessible publicly.

GUI changes

• Color theme is now adapted to the terminal background (fixes readability on 'light' background)

• Provided parameters are now shown in the 'Request' tab.

CLI changes

Added

• alpha init command: replace initialBlock for generated manifest based on contract creation block.

• alpha init prompt Ethereum chain. Added: Mainnet, BNB, Polygon, Goerli, Mumbai.

Fixed

• alpha init reports better progress specially when performing ABI & creation block retrieval.

• alpha init command without contracts fixed Protogen command invocation.

v1.1.8

Backend changes

Added

• Max-subrequests can now be overridden by auth header X-Sf-Substreams-Parallel-Jobs (note: if your auth plugin is 'trust', make sure that you filter out this header from public access
• Request Stats logging. When enable it will log metrics associated to a Tier1 and Tier2 request
• On request, save "substreams.partial.spkg" file to the state cache for debugging purposes.
• Manifest reader can now read 'partial' spkg files (without protobuf and metadata) with an option.

Fixed

• Fixed a bug which caused "live" blocks to be sent while the stream previously received block(s) were historic.

CLI changes

Fixed

• In GUI, module output now shows fields with default values, i.e. 0, "", false

v1.1.7

Highlights

Now using plugin: buf.build/community/neoeinstein-prost-crate:v0.3.1 when generating the Protobuf Rust mod.rs which fixes the warning that remote plugins are deprecated.

Previously we were using remote: buf.build/prost/plugins/crate:v0.3.1-1. But remote plugins when using https://buf.build (which we use to generate the Protobuf) are now deprecated and will cease to function on July 10th, 2023.

The net effect of this is that if you don't update your Substreams CLI to 1.1.7, on July 10th 2023 and after, the substreams protogen will not work anymore.

v1.1.6

Backend changes

• substreams-tier1 and substreams-tier2 are now standalone Apps, to be used as such by server implementations (firehose-ethereum, etc.)

• substreams-tier1 now listens to Connect protocol, enabling browser-based substreams clients

• Authentication has been overhauled to take advantage of https://github.com/streamingfast/dauth, allowing the use of a GRPC-based sidecar or reverse-proxy to provide authentication.

• Metering has been overhauled to take advantage of https://github.com/streamingfast/dmetering plugins, allowing the use of a GRPC sidecar or logs to expose usage metrics.

• The tier2 logs no longer show a parent_trace_id: the trace_id is now the same as tier1 jobs. Unique tier2 jobs can be distinguished by their stage and segment, corresponding to the output_module_name and startblock:stopblock

CLI changes

• The substreams protogen command now uses this Buf plugin https://buf.build/community/neoeinstein-prost to generate the Rust code for your Substreams definitions.

• The substreams protogen command no longer generate the FILE_DESCRIPTOR_SET constant which generates an unsued warning in Rust. We don't think nobody relied on having the FILE_DESCRIPTOR_SET constant generated, but if it's the case, you can provide your own buf.gen.yaml that will be used instead of the generated one when doing substreams protogen.

• Added -H flag on the substreams run command, to set HTTP Headers in the Substreams request.

Fixed

• Fixed generated buf.gen.yaml not being deleted when an error occurs while generating the Rust code.

v1.1.5

Highlights

This release fixes data determinism issues. This comes at a 20% performance cost but is necessary for integration with The Graph ecosystem.

Operators

• When upgrading a substreams server to this version, you should delete all existing module caches to benefit from deterministic output

Added

• Tier1 now records deterministic failures in wasm, "blacklists" identical requests for 10 minutes (by serving them the same InvalidArgument error) with a forced incremental backoff. This prevents accidental bad actors from hogging tier2 resources when their substreams cannot go passed a certain block.
• Tier1 now sends the ResolvedStartBlock, LinearHandoffBlock and MaxJobWorkers in SessionInit message for the client and gui to show
• Substreams CLI can now read manifests/spkg directly from an IPFS address (subgraph deployment or the spkg itself), using ipfs://Qm... notation

Fixed

• When talking to an updated server, the gui will not overflow on a negative start block, using the newly available resolvedStartBlock instead.
• When running in development mode with a start-block in the future on a cold cache, you would sometimes get invalid "updates" from the store passed down to your modules that depend on them. It did not impact the caches but caused invalid output.
• The WASM engine was incorrectly reusing memory, preventing deterministic output. It made things go faster, but at the cost of determinism. Memory is now reset between WASM executions on each block.
• The GUI no longer panics when an invalid output-module is given as argument

Changed

• Changed default WASM engine from wasmtime to wazero, use SUBSTREAMS_WASM_RUNTIME=wasmtime to revert to prior engine. Note that wasmtime will now run a lot slower than before because resetting the memory in wasmtime is more expensive than in wazero.
• Execution of modules is now done in parallel within a single instance, based on a tree of module dependencies.
• The substreams gui and substreams run now accept commas inside a param value. For example: substreams run --param=p1=bar,baz,qux --param=p2=foo,baz. However, you can no longer pass multiple parameters using an ENV variable, or a .yaml config file.

v1.1.4

HIGHLIGHTS

• Module hashing changed to fix cache reuse on substreams use imported modules
• Memory leak fixed on rpc-enabled servers
• GUI more responsive

Fixed

• BREAKING: The module hashing algorithm wrongfully changed the hash for imported modules, which made it impossible to leverage caches when composing new substreams off of imported ones.

  • Operationally, if you want to keep your caches, you will need to copy or move the old hashes to the new ones.
    • You can obtain the prior hashes for a given spkg with: substreams info my.spkg, using a prior release of the substreams
    • With a more recent substreams release, you can obtain the new hashes with the same command.
    • You can then cp or mv the caches for each module hash.
  • You can also ignore this change. This will simply invalidate your cache.

• Fixed a memory leak where "PostJobHooks" were not always called. These are used to hook in rpc calls in ethereum chain. They are now always called, even if no block has been processed (can be called with nil value for the clock)

• Jobs that fail deterministically (during WASM execution) on tier2 will fail faster, without retries from tier1.

• substreams gui command now handles params flag (it was ignored)

• Substeams GUI responsiveness improved significantly when handling large payloads

Added

• Added Tracing capabilities, using https://github.com/streamingfast/sf-tracing . See repository for details on how to enable.

Known issues

• If the cached substreams states are missing a 'full-kv' file in its sequence (not a normal scenario), requests will fail with opening file: not found #222

v1.1.3

Highlights

This release contains fixes for race conditions that happen when multiple request tries to sync the same range using the same .spkg. Those fixes will avoid weird state error at the cost of duplicating work in some circumstances. A future refactor of the Substreams engine scheduler will come later to fix those inefficiencies.

Operators, please read the operators section for upgrade instructions.

Operators

Note This upgrade procedure to you applies if your Substreams deployment topology includes both tier1 and tier2 processes. If you have defined somewhere the config value substreams-tier2: true, then this applies to you, otherwise, if you can ignore the upgrade procedure.

This release includes a small change in the internal RPC layer between tier1 processes and tier2 processes. This change requires an ordered upgrade of the processes to avoid errors.

The components should be deployed in this order:

1. Deploy and roll out tier1 processes first
2. Deploy and roll out tier2 processes in second

If you upgrade in the wrong order or if somehow tier2 processes start using the new protocol without tier1 being aware, user will end up with backend error(s) saying that some partial file are not found. Those will be resolved only when tier1 processes have been upgraded successfully.

Fixed

• Fixed a race when multiple Substreams request execute on the same .spkg, it was causing races between the two executors.
• GUI: fixed an issue which would slow down message consumption when progress page was shown in ascii art "bars" mode
• GUI: fixed the display of blocks per second to represent actual blocks, not messages count

Changed

• [binary]: Commands substreams <...> that fails now correctly return an exit code 1.

• [library]: The manifest.NewReader signature changed and will now return a *Reader, error (previously *Reader).

Added

• [library]: The manifest.Reader gained the ability to infer the path if provided with input "" based on the current working directory.

• [library]: The manifest.Reader gained the ability to infer the path if provided with input that is a directory.