Conversion testing #442
Merged
Conversion testing #442
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ArnaudBger
added a commit
that referenced
this pull request
Mar 29, 2024
* Conversion testing * fix SET_MIN_INT64 not using new conversion helper
sduchesneau
added a commit
that referenced
this pull request
May 9, 2024
* First commit * prevent panic on loadImports when imported package isn't reachable * Sink Examples Update (#439) * Minor docs improvements (#440) * enable reflection api * fix buf generation * partial (unsquashed) stores no longer contain traceID in filename, so they can be reused, squasher will handle this gracefully (#441) the execouts for stores are now only written on a PartialKV, containing a list of operations to re-apply. They are only output as deltas when applied to a fullKV. * First working version of index query language (still in package sqe for now) * fix dstore metering when reading output cache on tier2 * adjust scheduler to reduce reprocessing of same modules with new cached outputs * fix tier1 storage read-bytes metering, fix 'gui' header * Fix conflict * fix byte copy in store outputs saving * fix work plan: with new model, don't run all stages together on first segment * fix encoding/decoding of bigint/bigdecimal * fix corruption in new cached store implementation: keep deltas for proper ordinal lookup * Conversion testing (#442) * Conversion testing * fix SET_MIN_INT64 not using new conversion helper * touch changelog a bit * bump changelog to v1.4.0 * fix broken scheduler test with new behavior * when scanning store files, find old partial files with a traceID and delete them on sight (they are leftovers, possibly slowing down next runs) * improve log message when retrying tier2 request * Add sinks table (#443) * Handling timeout when getting block type from stream factory * Update decode tool * revent creating wrong store outputs on previous stages * First commit * Fixing tests * ensure the 'decode' tool can skip package validation by default * Handle BlockIndex type in computeStages * WIP * Add TODO * Fix compilation error * add blockfilter info on module in 'substreams info' command * add tool to decode index files * fix typo in decode tools * fix index handling in outputgraph * enable skipping some module executions when blockfilter exists (does not work when blockfilter is not already existing) * Add blockKeys parser * Add NewFromIndexKeys * prevent blockfilter fail if index don't exist, prepare skipping the whole run, clean up * add "on-the-fly" block filtering from index module's outputs * Disable not operation in query * remove println * support modules with non-0 initial blocks with indexes * remove checks of outputs on 'use' modules -> they're not accepted anyway * Fix bitmap_test * cleanup: most go linter warnings and remove dead code * go mod tidy * Handle index in integration test and improve test * Remove not operation tests and fix tier1 sending outpout to client * wip, take it away Arnaud * WIP * wip comments * WIP move stuff around, prepare refactor * some more comments * linear handoff calculation updated according to new spec * fix build and a few tests * skip indice module if it exists already, cleanup old stuff * WIP * Removing evaluateModuleRequireToRun * fix different modules outputs * fix read End Block on unbounded queries * fix request plan when storeInitialBlock > linearHandoffBlock, also fix integration tests stalling * fix some integration tests, remove useless ones * Refactor test and add lowestInitBlock func in graph * prevent executing modules before their initialBlock * allow some nil inputs when running a module * Add complex substreams * fix complex_substreams * safer skipping of inputs that have initialBlock in the future * simplify blockindex lookups * validate that each module has at least one valid input on its initialBlock * Add complex_substreams testing * Fix test * Rename and refactor testing * Fix test * fix another test path * Add Deltas into testing * Renaming tests * fix stages.getState on store modules with initialBlock in the future, fixing tests * add a few test cases to complex integration substreams * ensure tier2 "tries" to save fullKV if it has them * WIP * Fix test * Fix test again * fix test case stores_with_different_initial_blocks_on_the_same_stage * tweak 'different-initial-blocks-on-same-stage' test to be between boundaries. It now fails on a case that should work * Modify init block to 52 * Fix previous changes * Fix again test * Removing error when setting initial block to block index * Fix test * WIP * Add testing and fix others tests * Fix all the tests * Change scheduler logic to reduce reexecution in the first segments by running later stages until the other jobs can complete * fix new scheduler pattern in some cases * refactor checkValidBlockFilter * fix reader and run module func * Fix tests * Change loading file logic * Fix indexes * tier2: skip full block range when excluded by index * improve tier1 max output speed by preloading next file(s) in parallel * disable otelcol:// tracing which affected performance. Add 'clock' output to `run` * Apply index skipping for output * WIP * add support for blockFilterQuery from params * Change index logic * WIP * WIP * Remove clocks * Remove unecessary changes * Handle prefix for filtered imported modules * fix misaligned readexec on tier1 reader * make FileWalker on tier1 poll more aggressively on local filesystem * fix varying hashmodule when blockFilter is imported with another name * do not "meter" the cache store * allow index to be queried directly and terminate when index is created * fix total read bytes in case data already cache * prevent panic on empty match in index bitmap * fix panic on StreamTerminate for tier2 * allow skipping existing output ranges when running an index directly * fix writer test * remove debug println * bump changelog for v1.6.0 --------- Co-authored-by: Stéphane Duchesneau <[email protected]> Co-authored-by: Enol <[email protected]> Co-authored-by: YaroShkvorets <[email protected]> Co-authored-by: billettc <[email protected]> Co-authored-by: Stéphane Duchesneau <[email protected]> Co-authored-by: Matthieu Vachon <[email protected]> Co-authored-by: colindickson <[email protected]> Co-authored-by: Alexandre Bourget <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.