test: add efps to CI #7556

ricokahler · 2024-09-29T15:12:07Z

Description

The following PR:

Revamps the eFPS performance suite to compare the results of the benchmark to a released tag on npm. This is set to latest by default.
Adds a efps.yml workflow file that runs the eFPS suite and includes the results as a comment in PRs.

In addition to 1, the eFPS suite has also be updated to rerun the tests 3 times in CI and then the best performing value is taken. This is to mitigate against noisy neighbors. I've found that comparing the median performance between two different runs and re-running the tests 3 times yield relatively stable results for the relatively unstable environment. Hopefully this provides a good signal on performance without requiring the complex infrastructure.

What to review

Take a look at the output here and see if everything makes sense or if you have any suggestions
Take a look at the changes to the eFPS suite. Do they all make sense?
Take a look at the workflow file as well.

Testing

Tested many times in a different branch to see how it would look in CI.

Notes for release

N/A - internal only

vercel · 2024-09-29T15:12:11Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
page-building-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 1, 2024 8:20pm
performance-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 1, 2024 8:20pm
test-compiled-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 1, 2024 8:20pm
test-next-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 1, 2024 8:20pm
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 1, 2024 8:20pm

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Oct 1, 2024 8:20pm

github-actions · 2024-09-29T15:13:08Z

No changes to documentation

github-actions · 2024-09-29T15:25:17Z

Component Testing Report Updated Oct 1, 2024 8:30 PM (UTC)

✅ All Tests Passed -- expand for details

File	Status	Duration	Passed	Skipped
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	46s	15	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	8s	3	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	✅ Passed (Inspect)	30s	6	0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	37s	11	7
formBuilder/inputs/PortableText/copyPaste/CopyPasteFields.spec.tsx	✅ Passed (Inspect)	0s	0	12
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	17s	6	0
formBuilder/inputs/PortableText/DisableFocusAndUnset.spec.tsx	✅ Passed (Inspect)	11s	3	0
formBuilder/inputs/PortableText/DragAndDrop.spec.tsx	✅ Passed (Inspect)	3m 0s	0	0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	44s	15	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	1m 42s	21	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	1m 14s	18	0
formBuilder/inputs/PortableText/PresenceCursors.spec.tsx	✅ Passed (Inspect)	9s	3	9
formBuilder/inputs/PortableText/RangeDecoration.spec.tsx	✅ Passed (Inspect)	26s	9	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	18s	6	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	✅ Passed (Inspect)	35s	12	0
formBuilder/tree-editing/TreeEditing.spec.tsx	✅ Passed (Inspect)	0s	0	3
formBuilder/tree-editing/TreeEditingNestedObjects.spec.tsx	✅ Passed (Inspect)	0s	0	3

github-actions · 2024-09-29T15:29:30Z

⚡️ Editor Performance Report

Updated Tue, 01 Oct 2024 20:36:20 GMT

Benchmark	reference ^{latency of sanity@latest}	experiment ^{latency of this branch}	Δ (%) ^{latency difference}
article (title)	17.2 efps (58ms)	17.5 efps (57ms)	-1ms (-1.7%)	✅
article (body)	60.2 efps (17ms)	62.9 efps (16ms)	-1ms (-4.2%)	✅
article (string inside object)	19.2 efps (52ms)	18.5 efps (54ms)	+2ms (+3.8%)	✅
article (string inside array)	14.9 efps (67ms)	14.9 efps (67ms)	+0ms (-/-%)	✅
recipe (name)	30.3 efps (33ms)	30.3 efps (33ms)	+0ms (-/-%)	✅
recipe (description)	33.9 efps (30ms)	33.9 efps (30ms)	+0ms (-/-%)	✅
recipe (instructions)	99.9+ efps (6ms)	99.9+ efps (6ms)	+0ms (-/-%)	✅
synthetic (title)	15.2 efps (66ms)	14.6 efps (69ms)	+3ms (+3.8%)	✅
synthetic (string inside object)	15.2 efps (66ms)	15.9 efps (63ms)	-3ms (-4.5%)	✅

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	58ms	61ms	71ms	191ms	1089ms	13.6s
article (body)	17ms	18ms	20ms	91ms	109ms	5.0s
article (string inside object)	52ms	58ms	68ms	154ms	819ms	8.8s
article (string inside array)	67ms	74ms	86ms	127ms	1619ms	9.8s
recipe (name)	33ms	35ms	41ms	60ms	53ms	9.2s
recipe (description)	30ms	31ms	35ms	80ms	34ms	6.3s
recipe (instructions)	6ms	8ms	9ms	11ms	0ms	3.2s
synthetic (title)	66ms	71ms	91ms	298ms	1997ms	16.5s
synthetic (string inside object)	66ms	69ms	75ms	229ms	1776ms	10.3s

🧪 Experiment result

The performance result of this branch

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	57ms	63ms	69ms	178ms	1184ms	13.6s
article (body)	16ms	19ms	22ms	118ms	180ms	5.2s
article (string inside object)	54ms	56ms	72ms	186ms	1090ms	9.2s
article (string inside array)	67ms	72ms	84ms	127ms	1799ms	10.1s
recipe (name)	33ms	37ms	39ms	62ms	47ms	9.1s
recipe (description)	30ms	32ms	36ms	60ms	32ms	6.3s
recipe (instructions)	6ms	8ms	9ms	23ms	0ms	3.3s
synthetic (title)	69ms	72ms	80ms	167ms	2101ms	16.7s
synthetic (string inside object)	63ms	67ms	78ms	288ms	1672ms	9.7s

📚 Glossary

column definitions

benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".

latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.

p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.

p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.

p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.

blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.

test duration — how long the test run took to complete.

rexxars

Great work. See my comment on requestIdleCallback.

perf/efps/helpers/measureBlockingTime.ts

* test: revamp eFPS suite * chore: add efps github actions * chore: use different token for efps tests * test: add comment with perf report result * test: keep playwright install script * feat: add workflow dispatch inputs * fix: use `requestAnimationFrame` instead --------- Co-authored-by: Binoy Patel <[email protected]>

vercel bot deployed to Preview – test-compiled-studio September 29, 2024 15:13 View deployment

vercel bot deployed to Preview – test-next-studio September 29, 2024 15:14 View deployment

ricokahler mentioned this pull request Sep 29, 2024

test: compare local benchmark to latest and exit 1 #7529

Closed

ricokahler marked this pull request as ready for review September 29, 2024 15:25

ricokahler requested a review from a team as a code owner September 29, 2024 15:25

ricokahler requested review from rexxars and removed request for a team September 29, 2024 15:25

rexxars requested changes Sep 30, 2024

View reviewed changes

perf/efps/helpers/measureBlockingTime.ts Outdated Show resolved Hide resolved

vercel bot temporarily deployed to Preview – page-building-studio September 30, 2024 21:01 Inactive

vercel bot temporarily deployed to Preview – test-next-studio September 30, 2024 21:01 Inactive

vercel bot temporarily deployed to Preview – test-compiled-studio September 30, 2024 21:01 Inactive

vercel bot deployed to Preview – performance-studio September 30, 2024 21:01 View deployment

vercel bot deployed to Preview – test-studio September 30, 2024 21:01 View deployment

ricokahler force-pushed the test/opertionalize-efps branch from e507c76 to 1727b56 Compare September 30, 2024 21:01

vercel bot deployed to Preview – performance-studio September 30, 2024 21:02 View deployment

vercel bot deployed to Preview – page-building-studio September 30, 2024 21:02 View deployment

vercel bot deployed to Preview – test-studio September 30, 2024 21:02 View deployment

vercel bot deployed to Preview – test-next-studio September 30, 2024 21:07 View deployment

vercel bot deployed to Preview – test-compiled-studio September 30, 2024 21:11 View deployment

ricokahler and others added 7 commits October 1, 2024 15:16

test: revamp eFPS suite

753a3ca

chore: add efps github actions

04eb6ac

chore: use different token for efps tests

8d6badb

test: add comment with perf report result

4c05bf0

test: keep playwright install script

979aae8

feat: add workflow dispatch inputs

aad808c

fix: use requestAnimationFrame instead

350fdd7

ricokahler force-pushed the test/opertionalize-efps branch from 1727b56 to 350fdd7 Compare October 1, 2024 20:16

vercel bot deployed to Preview – page-building-studio October 1, 2024 20:17 View deployment

vercel bot deployed to Preview – performance-studio October 1, 2024 20:17 View deployment

vercel bot deployed to Preview – test-studio October 1, 2024 20:17 View deployment

rexxars approved these changes Oct 1, 2024

View reviewed changes

vercel bot deployed to Preview – test-next-studio October 1, 2024 20:19 View deployment

vercel bot deployed to Preview – test-compiled-studio October 1, 2024 20:20 View deployment

ricokahler added this pull request to the merge queue Oct 1, 2024

Merged via the queue into next with commit aeb9e71 Oct 1, 2024
51 checks passed

ricokahler deleted the test/opertionalize-efps branch October 1, 2024 21:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add efps to CI #7556

test: add efps to CI #7556

ricokahler commented Sep 29, 2024 •

edited

Loading

vercel bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024

github-actions bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024 •

edited

Loading

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

rexxars left a comment

test: add efps to CI #7556

test: add efps to CI #7556

Conversation

ricokahler commented Sep 29, 2024 • edited Loading

Description

What to review

Testing

Notes for release

vercel bot commented Sep 29, 2024 • edited Loading

github-actions bot commented Sep 29, 2024

github-actions bot commented Sep 29, 2024 • edited Loading

github-actions bot commented Sep 29, 2024 • edited Loading

⚡️ Editor Performance Report

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

rexxars left a comment

Choose a reason for hiding this comment

ricokahler commented Sep 29, 2024 •

edited

Loading

vercel bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024 •

edited

Loading