Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added bench/roulette.lua #9

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Added bench/roulette.lua #9

wants to merge 2 commits into from

Conversation

lukego
Copy link

@lukego lukego commented Oct 8, 2016

This benchmark exhibits non-deterministic performance. See LuaJIT/LuaJIT#218.

I think that it is important to include tests that can demonstrate non-deterministic performance. The challenge I see is to make performance as consistent as possible while including as much entropy as possible on the inputs.

Paraphrasing Dijkstra:

I used to think that benchmarks have to be deterministic. Now I realize that deterministic benchmarks are only a special case of non-deterministic benchmarks, and not a particularly interesting one at that.

Requires a test harness that can account for variation. More on that to follow...

This benchmark exhibits non-deterministic performance.
lukego added a commit to lukego/LuaJIT-test-cleanup that referenced this pull request Oct 8, 2016
The test harness shuold do this in some suitable way e.g. with values
that are reproducible but still contain entropy for the JIT.
@lukego lukego mentioned this pull request Oct 9, 2016
@lukego
Copy link
Author

lukego commented Oct 9, 2016

I pushed an update so that the random seed is not initialized in the benchmark script. This should be done separately by the test harness (if it is able to deal with randomness). For #10 the test harness is running the test N times and using a different but predictable random seed (1..N) each time. On the one hand this means that a full test run should be exposed to entropy, on the other hand it means that each individual test should be readily reproducible e.g. to reproduce the result of run 42 call math.randomseed(42) before running the script.

@lukego lukego mentioned this pull request Oct 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant