Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(wasm-gen, runtime-fuzzer): Increase successful messages execution rate in runtime-fuzzer. #3383

Merged
merged 5 commits into from
Oct 2, 2023

Conversation

techraed
Copy link
Member

@techraed techraed commented Oct 1, 2023

This task was a hot one from the beginning of runtime-fuzzer launch when first production coverages were available. The issue was that there were too many traps, which caused a very weak coverage for messaging logic in the pallet_gear.

After researching it thoroughly, I have found out that there were 2 many trap reasons: Possibly 'unreachable' instruction reached and Not enough gas to continue execution.
The former trap reason was caused not only by unreachable instruction occurrences, but there were some other internal wasmi errors causing that termination reason. I have turned off unreachable instructions here #3307.
The latter one was mainly due to large loops in wasm - each function body consisted of about 100_000 instructions and these instructions could call other functions with the same size in a loop. Turning off Control instructions can be a solution to that, but it brings some other problems described here #3382. This seems to be the best solution to the problem - #3313.

Basically, there are 3 available params that have a significant effect on execution:

  1. Turning off Control (if, loop, try, catch and etc.) instructions. This makes the execution of the ~8gb corpus directory really fast (3 minutes), there are no Not enough gas to continue execution errors, but it there is a still a huge number of traps.
  2. Instructions number in the function body.
  3. Amount of internal functions generated by wasm-smith.

After running 2 corpuses sets each about 8 gb with different values for these params, I have found out that amount of instructions doesn't have a huge effect on execution time or execution results. In theory, less instructions make less gas errors and more frequent syscalls invocations. In practice, this has a very weak effect when amount of functions is large. The third param caused the most effect on increasing the corpus execution speed and amount of successfull executions. Even if there are 100_000 instructions, but less functions, it empirically shows great results.

@techraed techraed added A0-pleasereview PR is ready to be reviewed by the team D4-test Autotests, and examples D5-tooling Helper tools and utilities labels Oct 1, 2023
@techraed techraed self-assigned this Oct 1, 2023
utils/runtime-fuzzer/src/gear_calls.rs Outdated Show resolved Hide resolved
@techraed techraed added A2-mergeoncegreen PR is ready to merge after CI passes A5-dontmerge PR should not be merged yet and removed A0-pleasereview PR is ready to be reviewed by the team A2-mergeoncegreen PR is ready to merge after CI passes labels Oct 2, 2023
@techraed
Copy link
Member Author

techraed commented Oct 2, 2023

DON'T MERGE, some performance checks are running

@techraed techraed added A2-mergeoncegreen PR is ready to merge after CI passes and removed A5-dontmerge PR should not be merged yet labels Oct 2, 2023
@techraed
Copy link
Member Author

techraed commented Oct 2, 2023

This task was a hot one from the beginning of runtime-fuzzer launch when first production coverages were available. The issue was that there were too many traps, which caused a very weak coverage for messaging logic in the pallet_gear.

After researching it thoroughly, I have found out that there were 2 many trap reasons: Possibly 'unreachable' instruction reached and Not enough gas to continue execution. The former trap reason was caused not only by unreachable instruction occurrences, but there were some other internal wasmi errors causing that termination reason. I have turned off unreachable instructions here #3307. The latter one was mainly due to large loops in wasm - each function body consisted of about 100_000 instructions and these instructions could call other functions with the same size in a loop. Turning off Control instructions can be a solution to that, but it brings some other problems described here #3382. This seems to be the best solution to the problem - #3313.

Basically, there are 3 available params that have a significant effect on execution:

  1. Turning off Control (if, loop, try, catch and etc.) instructions. This makes the execution of the ~8gb corpus directory really fast (3 minutes), there are no Not enough gas to continue execution errors, but it there is a still a huge number of traps.
  2. Instructions number in the function body.
  3. Amount of internal functions generated by wasm-smith.

After running 2 corpuses sets each about 8 gb with different values for these params, I have found out that amount of instructions doesn't have a huge effect on execution time or execution results. In theory, less instructions make less gas errors and more frequent syscalls invocations. In practice, this has a very weak effect when amount of functions is large. The third param caused the most effect on increasing the corpus execution speed and amount of successfull executions. Even if there are 100_000 instructions, but less functions, it empirically shows great results.

After additional performance checks it occurred that changing amount of functions and lowering instructions doesn't actually give much improvement to successful execution rate. Anyway the PR will be merged as it brings some execution improvements (speed).

@techraed techraed merged commit a4763d8 into master Oct 2, 2023
10 checks passed
@techraed techraed deleted the st-increase-successful-msgs-rate branch October 2, 2023 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A2-mergeoncegreen PR is ready to merge after CI passes D4-test Autotests, and examples D5-tooling Helper tools and utilities
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants