Add `egui_kittest`, a test harness for egui #5166

lucasmerlin · 2024-09-25T10:05:58Z

closes Integration testing/event injection #3491
closes Improved unit/regression test support #3926

This adds a testing library to egui based on kittest. Kittest is a new AccessKit-based testing library. The api is inspired by the js testing-library where the idea is also to query the dom based on accessibility attributes.
We made kittest with egui in mind but it should work with any rust gui framework with AccessKit support.

It currently has support for:

running the egui app, frame by frame
building the AccessKit tree
ergonomic queries via kittest
- via e.g. get_by_name, get_by_role
simulating events based on the accesskit node id
creating arbitrary events based on Harness::input_mut
rendering screenshots via wgpu
snapshot tests with these screenshots

A simple test looks like this:

fn main() {
    let mut checked = false;
    let app = |ctx: &Context| {
        CentralPanel::default().show(ctx, |ui| {
            ui.checkbox(&mut checked, "Check me!");
        });
    };

    let mut harness = Harness::builder().with_size(egui::Vec2::new(200.0, 100.0)).build(app);
    
    let checkbox = harness.get_by_name("Check me!");
    assert_eq!(checkbox.toggled(), Some(Toggled::False));
    checkbox.click();
    
    harness.run();

    let checkbox = harness.get_by_name("Check me!");
    assert_eq!(checkbox.toggled(), Some(Toggled::True));

    // You can even render the ui and do image snapshot tests
    #[cfg(all(feature = "wgpu", feature = "snapshot"))]
    egui_kittest::image_snapshot(&egui_kittest::wgpu::TestRenderer::new().render(&harness), "readme_example");
}

~~Since getting wgpu to run in ci is a hassle, I'm taking another shot at creating a software renderer for egui (ideally without a huge dependency like skia)~~ (this didn't work as well as I hoped and it turns out in CI you can just run tests on a mac runner which comes with a real GPU)

Here is a example of a failed snapshot test in ci, it will say which snapshot failed and upload an artifact with the before / after and diff images:
https://github.com/emilk/egui/actions/runs/11183049487/job/31090724606?pr=5166

github-actions · 2024-09-25T10:09:38Z

Preview available at https://egui-pr-preview.github.io/pr/5166-lucas/testing_library
Note that it might take a couple seconds for the update to show up after the preview_build workflow has completed.

emilk · 2024-09-25T15:47:39Z

After some discussion we came to these conclusions:

Investigate using git lfs for storing the .png snapshot files so we don't bloat the git repository. Try to set it up in such a way that git clone + cargo test still works for contributors without lfs.
Try to set up a proper software wgpu backend that works on CI, same as the wgpu repository uses. This will be the most powerful and accurate, and allow us to test more than just egui (i.e. the Rerun Viewer)
Move the accesskit-specific part of the integration to its own repository and eventually invite collaborators from other GUI toolkits to collaborate on it

.github/workflows/rust.yml

emilk

This is so awesome!

The PR description is a bit outdated though

emilk · 2024-10-11T12:50:11Z

.gitattributes

@@ -1,2 +1,3 @@
 * text=auto eol=lf
 Cargo.lock linguist-generated=false
+**/tests/snapshots/**/*.png filter=lfs diff=lfs merge=lfs -text


Should we just apply lfs to all png:s everywhere?

I think so, but I think we should do that in a separate PR. And it should also include gifs, which seem to be the biggest files in the repo. And maybe also the entirety of the media folder.

.gitignore

.github/workflows/rust.yml

crates/egui_demo_lib/src/demo/widget_gallery.rs

emilk · 2024-10-11T13:00:43Z

crates/egui_demo_lib/tests/snapshots/demos/Tooltips.png

There's something wrong with the framing on this one!

Ah, that's because this one has constrain(false). No idea how I could fix that though without changing constrain to true 🤔

crates/egui_kittest/src/snapshot.rs

emilk · 2024-10-11T13:16:49Z

crates/egui_kittest/src/snapshot.rs

+        });
+    }
+
+    let result = dify::diff::get_results(previous, current.clone(), 0.1, true, None, &None, &None);


And separately - have you tested tolerance=0?

I've changed it to 0, but I think we should at least have someone test this on linux and windows. I can test on windows but someone else will need to test it for linux. Preferably once with the vulkan and once with the gl backend.

crates/egui_kittest/src/snapshot.rs

crates/egui_kittest/src/wgpu.rs

crates/egui_kittest/src/texture_to_bytes.rs

…disable dithering

lucasmerlin · 2024-10-12T13:32:24Z

I've changed everything you mentioned and also added Harness::wgpu_snapshot as a convenicence.

Also, I've disabled dithering since it caused minor differences between frames. This made the pngs much smaller!

The check.sh script was broken in #5166, this fixes it * Related to #5297 * [x] I have followed the instructions in the PR template --------- Co-authored-by: Emil Ernerfeldt <[email protected]>

- closes emilk#3491 - closes emilk#3926 This adds a testing library to egui based on [kittest](https://github.com/rerun-io/kittest). Kittest is a new [AccessKit](https://github.com/AccessKit/accesskit/)-based testing library. The api is inspired by the js [testing-library](https://testing-library.com/) where the idea is also to query the dom based on accessibility attributes. We made kittest with egui in mind but it should work with any rust gui framework with AccessKit support. It currently has support for: - running the egui app, frame by frame - building the AccessKit tree - ergonomic queries via kittest - via e.g. get_by_name, get_by_role - simulating events based on the accesskit node id - creating arbitrary events based on Harness::input_mut - rendering screenshots via wgpu - snapshot tests with these screenshots A simple test looks like this: ```rust fn main() { let mut checked = false; let app = |ctx: &Context| { CentralPanel::default().show(ctx, |ui| { ui.checkbox(&mut checked, "Check me!"); }); }; let mut harness = Harness::builder().with_size(egui::Vec2::new(200.0, 100.0)).build(app); let checkbox = harness.get_by_name("Check me!"); assert_eq!(checkbox.toggled(), Some(Toggled::False)); checkbox.click(); harness.run(); let checkbox = harness.get_by_name("Check me!"); assert_eq!(checkbox.toggled(), Some(Toggled::True)); // You can even render the ui and do image snapshot tests #[cfg(all(feature = "wgpu", feature = "snapshot"))] egui_kittest::image_snapshot(&egui_kittest::wgpu::TestRenderer::new().render(&harness), "readme_example"); } ``` ~Since getting wgpu to run in ci is a hassle, I'm taking another shot at creating a software renderer for egui (ideally without a huge dependency like skia)~ (this didn't work as well as I hoped and it turns out in CI you can just run tests on a mac runner which comes with a real GPU) Here is a example of a failed snapshot test in ci, it will say which snapshot failed and upload an artifact with the before / after and diff images: https://github.com/emilk/egui/actions/runs/11183049487/job/31090724606?pr=5166

The check.sh script was broken in emilk#5166, this fixes it * Related to emilk#5297 * [x] I have followed the instructions in the PR template --------- Co-authored-by: Emil Ernerfeldt <[email protected]>

lucasmerlin added tests Unit tests, regression tests, manual tests, … dev-experience labels Sep 25, 2024

lucasmerlin force-pushed the lucas/testing_library branch from 9b76b0b to a0a9671 Compare September 25, 2024 21:05

lucasmerlin force-pushed the lucas/testing_library branch from 4ac8fea to d167452 Compare October 5, 2024 14:24

lucasmerlin added 24 commits October 5, 2024 16:26

Add egui testing library

c50394a

Improved usability

d5dedba

Rename and move kittest to external crate

608e4df

Enable git lfs for snapshots files

6f2691f

Add widget_gallery.png

e791a74

Fix Cargo.toml formatting

0853a52

Add snapshot tests for all demos

661bc3e

Move demo snapshots

87cdb17

Update kittest

ee3e766

Add text_edit.rs test and fix ime input

e63ffa3

Typo

bba834b

Run tests with gpu

b2db13d

Improve snapshot error

94d0b84

Test if snapshots fail in ci

f26e98d

Update dify

02f734c

Fix lints

80381ab

Try if tests fail on style change

7cbf557

Upload snapshots as artifacts

d478162

Fail tests on missing snapshot

10c0767

Checkout with lfs

b4f7129

Always upload artifacts

b00315b

Update dify

2ac2d85

"un-break" tests

d7b874e

Fix lint

66a6c00

lucasmerlin added 2 commits October 10, 2024 14:52

Refactor events and impl debug for Harness

311d447

Use kittest from main

ddf79ce

lucasmerlin commented Oct 11, 2024

View reviewed changes

.github/workflows/rust.yml Show resolved Hide resolved

lucasmerlin requested a review from emilk October 11, 2024 10:12

emilk approved these changes Oct 11, 2024

View reviewed changes

lucasmerlin added 3 commits October 12, 2024 13:23

Review changes

148e79a

Remember mouse position

24f29d6

Add comment about threshold and for the SnapshotError variants

27414ea

lucasmerlin force-pushed the lucas/testing_library branch from 88324d1 to 27414ea Compare October 12, 2024 11:53

lucasmerlin added 7 commits October 12, 2024 14:05

Improved snapshot error handling

b2fe731

Rename texture_to_bytes

554fc57

Round screen size

97ecb25

Add step function

c9d1bea

Add more convenient snapshot api, allow reusing the TestRenderer and …

2b1fafa

…disable dithering

lints

df24f74

Update example snapshot

e824e01

lucasmerlin requested a review from emilk October 12, 2024 13:29

This was referenced Oct 21, 2024

GUI testing on CI rerun-io/rerun#3736

Closed

Run e2e GUI tests with full viewer using software rasterizer on CI rerun-io/rerun#2255

Closed

lucasmerlin merged commit 70a0113 into master Oct 22, 2024
44 checks passed

lucasmerlin deleted the lucas/testing_library branch October 22, 2024 10:39

lucasmerlin changed the title ~~Add egui testing library~~ Add egui_kittest, a test harness for egui Oct 22, 2024

This was referenced Oct 22, 2024

Snapshot tests fail withunity feature #5297

Open

Fix cargo test in check.sh #5299

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `egui_kittest`, a test harness for egui #5166

Add `egui_kittest`, a test harness for egui #5166

lucasmerlin commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024

emilk commented Sep 25, 2024

emilk left a comment

emilk Oct 11, 2024

lucasmerlin Oct 12, 2024

emilk Oct 11, 2024

lucasmerlin Oct 12, 2024

emilk Oct 11, 2024

lucasmerlin Oct 12, 2024

lucasmerlin commented Oct 12, 2024

Add egui_kittest, a test harness for egui #5166

Add egui_kittest, a test harness for egui #5166

Conversation

lucasmerlin commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 25, 2024

emilk commented Sep 25, 2024

emilk left a comment

Choose a reason for hiding this comment

emilk Oct 11, 2024

Choose a reason for hiding this comment

lucasmerlin Oct 12, 2024

Choose a reason for hiding this comment

emilk Oct 11, 2024

Choose a reason for hiding this comment

lucasmerlin Oct 12, 2024

Choose a reason for hiding this comment

emilk Oct 11, 2024

Choose a reason for hiding this comment

lucasmerlin Oct 12, 2024

Choose a reason for hiding this comment

lucasmerlin commented Oct 12, 2024

Add `egui_kittest`, a test harness for egui #5166

Add `egui_kittest`, a test harness for egui #5166

lucasmerlin commented Sep 25, 2024 •

edited

Loading