PHPLIB-1187: Run benchmark on Evergreen #1185
Conversation
| @@ -6,6 +6,5 @@ | |||
| "runner.file_pattern": "*Bench.php", | |||
| "runner.path": "src", | |||
| "runner.php_config": { "memory_limit": "1G" }, | |||
| "runner.iterations": 3, | |||
| "runner.revs": 10 | |||
There was a problem hiding this comment.
Removed this in favour of increasing revs only for benchmarks that run in the microsecond range. For anything that takes milliseconds, the precision is usually good enough with a single rev.
| @@ -74,15 +72,15 @@ public function afterIteration(): void | |||
| * Using a single thread to export multiple files. | |||
| * By executing a single Find command for multiple files, we can reduce the number of roundtrips to the server. | |||
| * | |||
| * @param array{chunk:int} $params | |||
| * @param array{chunkSize:int} $params | |||
There was a problem hiding this comment.
Decided to rename this since I got confused as to what chunk meant: I thought chunk: 1 meant a single chunk of files while it actually meant 100 chunks of 1 file each.
| * @param array{chunk:int} $params | ||
| */ | ||
| #[ParamProviders(['provideChunkParams'])] | ||
| public function benchBulkWrite(array $params): void |
There was a problem hiding this comment.
This subject essentially tests the same as benchInsertMany, so I decided to skip it in order to shave ~6 minutes off the run time.
There was a problem hiding this comment.
We could keep this and remove the other that is a little less efficient
There was a problem hiding this comment.
The reason I removed bulkWrite and kept insertMany is that benchmarking the latter also exposes performance regressions introduced in the library, while the other only tests the extension.
| 'created_at' => date(DATE_ATOM), | ||
| 'completed_at' => date(DATE_ATOM), |
There was a problem hiding this comment.
I don't think this information is relevant to us, so I didn't bother trying to extract the exact dates. The documentation also didn't mark this as optional, so I decided to just include the current date instead.
There was a problem hiding this comment.
Well, created_at could take $_SERVER['REQUEST_TIME'].
alcaeus
force-pushed
the
phplib-1187-evg-benchmark
branch
from
40f013a to
bbcc0f9
Compare
| @@ -1 +1,2 @@ | |||
| extension=mongodb.so | |||
| memory_limit=-1 | |||
There was a problem hiding this comment.
You should increase the limit. Otherwise we don't know how much memory is used if it's too much. What is the memory limit of the job runner?
There was a problem hiding this comment.
For some reason it aborted when it hit a previous 128M limit, apparently ignoring the 1G limit definer in the runner config. We can also report memory usage from the benchmarks using perf.send, which would be a better indicator than failing the build when it hits some limit. Want me to add those numbers?
There was a problem hiding this comment.
perf.send is too late if the process crashes, no?
There was a problem hiding this comment.
Correct, perf.send would not be executed in that case. With -1 the memory would be unlimited, so the only case in which it would crash is if it used up all memory including the page file, which I'd consider highly unlikely.
| 'created_at' => date(DATE_ATOM), | ||
| 'completed_at' => date(DATE_ATOM), |
There was a problem hiding this comment.
Well, created_at could take $_SERVER['REQUEST_TIME'].
PHPLIB-1187
This pull request adds a task to run benchmarks on Evergreen. It also uses
perf.sendto store benchmark metrics in CI so we can analyse the performance.I've also removed a couple of benchmarks in order to bring the benchmark time down. Most recent tests show a 33 minute runtime on Evergreen. I also had to skip the AMP subjects as I ran into issues with socket creation. Since AMP does not perform significantly different from forking, I think it's safe to exclude them from the metrics we collect on Evergreen.