e2e browser crashes

[martinpitt]

We have a lot of random test crashes like this or this where chromium goes bananas and gets stuck, and the test eventually times out. I've never seen this with firefox nor on RHOS. Sometimes it's so bad that ¾ of the tests fail that way.

I suspect RAM over-commitment, so possibly we need to dial down the parallelism.

example Feb 20

example Feb 26

example Feb 27

---

[mvo] I can have a look, to learn more about our testing infrastructure.

Reducing TEST_JOBS seems like a good idea, it's at 8 currently.

---

example Feb 26

This has happened on rhos, but doesn't have the browser OOM message. Other tests include this:

[0227/081645.186300:ERROR:v8_initializer.cc(798)] V8 javascript OOM (CALL_AND_RETRY_LAST).

correct theory: SELinux denials

At the machines where this happens, there are thousands of SElinux rejections:

AVC avc:  denied  { execheap } for  pid=3725873 comm="ThreadPoolForeg" scontext=system_u:system_r:container_t:s0:c439,c758 tcontext=system_u:system_r:container_t:s0:c439,c758 tclass=process permissive=0

and some

ANOM_ABEND auid=4294967295 uid=1111 gid=1111 ses=4294967295 subj=system_u:system_r:container_t:s0:c439,c758 pid=3725873 comm="headless_shell" exe="/usr/lib64/chromium-browser/headless_shell" sig=5 res=1
systemd-coredump[3725903]: [🡕] Process 3725873 (headless_shell) of user 1111 terminated abnormally without generating a coredump.

Checking PSI cluster (rhos01):

ansible -i inventory -m shell -a 'journalctl | grep ThreadPoolForeg' openstack_tasks

→ not a single SELinux denial. Since January, there are a 11 instances of

ThreadPoolForeg invoked oom-killer: gfp_mask=0x8c40(GFP_NOFS|__GFP_NOFAIL), order=0, oom_score_adj=0
Jan 14 23:12:49 rhos-01-19 kernel: CPU: 6 PID: 728617 Comm: ThreadPoolForeg Not tainted 6.5.6-300.fc39.x86_64 #1

and that's a sign that we stress the rhos01 instances quite a bit. But it happens rarely enough that it's not an immediate concern IMHO.

On Feb 27 15:45 CET, @martinpitt ran ansible -f20 -i inventory -m shell -a 'setenforce 0' e2e. Let's watch out for failures in the next days, and see if it still happens. If not, and that's it, then we can see if we can fix this in a finer-grained way.

A big difference between rhos and e2e is that the former runs "classic" Fedora 39 server, while the latter runs Fedora IoT (also based on 39). But that shouldn't matter much for the on-disk runtime bits. However, all our machines are quite a bit behind in terms of upgrades (they don't matter much, these aren't internet-facing machines):

On rhos (classic rpm):

selinux-policy-targeted-38.28-1.fc39.noarch
kernel-core-6.5.6-300.fc39.x86_64
podman-4.8.2-1.fc39.x86_64

on e2e (IoT):

selinux-policy-targeted-39.3-1.fc39.noarch
kernel-core-6.6.9-200.fc39.x86_64
podman-4.8.2-1.fc39.x86_64

On Feb 28 08:15 CET, all our machines were dnf/ostree upgraded to latest package versions and rebooted, so that they are now very similar. They both have selinux-policy-targeted-39.4-1.fc39.noarch and podman-4.9.3-1.fc39.x86_64. e2e has kernel-core-6.7.5-200.fc39.x86_64, while rhos has kernel-core-6.7.6-200.fc39.x86_64.

Also, I setenforce 0 everywhere right away: We can still see the rejections with permissive=1 without having to actually fail our tests. If we still get them, let's investigate/report and allow execheap in a more targeted fashion.

So tomorrow, let's run

ansible -f20 -i inventory -m shell -a 'journalctl -b | grep AVC.*denied' e2e

and on openstack_tasks too.

Theory: /dev/shm

Current theory: Our --shm-size=1024M is not enough for TEST_JOBS=8, and recent chromium versions just need moar of it. Let's try to provoke that error in a local podman container with smaller SHM and/or --memory, and see if it's shm or heap.

Scaled down to ¼, the parameters as in production are:

podman run -it --rm --device=/dev/kvm --memory=6g --pids-limit=4096 --shm-size=256m -v ~/.cache/cockpit-images:/cache/images -e TEST_JOBS=2 quay.io/cockpit/tasks sh -exc 'git clone https://github.com/cockpit-project/cockpit-podman; cd cockpit-podman/; test/run'

which works fine locally. However, with --shm-size=80m the test fails very quickly with

DP: {"source":"network","level":"error","text":"Failed to load resource: net::ERR_INSUFFICIENT_RESOURCES","timestamp":1709040209423.008,"url":"http://127.0.0.2:9091/cockpit/@localhost/podman/index.js","networkRequestId":"928.355"}
RuntimeError: ReferenceError: cockpit is not defined

[0227/132330.730287:ERROR:validation_errors.cc(117)] Invalid message: VALIDATION_ERROR_DESERIALIZATION_FAILED
[0227/132330.730345:ERROR:interface_endpoint_client.cc(702)] Message 1033874247 rejected by interface viz.mojom.CopyOutputResultSender
[0227/132330.730396:ERROR:browser_child_process_host_impl.cc(754)] Terminating child process for bad message: Received bad user message: Validation failed for viz.mojom.CopyOutputResultSender.0  [VALIDATION_ERROR_DESERIALIZATION_FAILED]

and some more noise. podman exec -itl watch df -h /dev/shm also shows at the start of the test (when it didn't fail yet) that it's quickly eating away space:

Filesystem      Size  Used Avail Use% Mounted on
shm              80M   53M   28M  66% /dev/shm

This already happens during image-customize, so I take it that QEMU itself also started to use /dev/shm. With --shm-size=180m it works well enough, with Use going up to ~ 72%. But it changes quickly, I may have missed a few peaks.

A better command is

podman exec -itl bash -c 'while true; do df -h /dev/shm | grep ^shm; sleep 0.5; done'

Similar with --shm-size 140m -- tests start to fail with "cockpit is not defined" (like above), but this time /dev/shm actually hit the root (97%).

With --shm-size=200m the error message looks different:

RuntimeError: TypeError: Cannot read properties of undefined (reading 'includes')

.. which is interesting as I think I've seen this somewhere. It could be a bug in our page of course, and totally unrelated. And it's not the OOM hang.

I just thought about a counter-argument: rhos-01 seems much less affected (if at all), but it uses the same SHM config. It only runs one container per host, as opposed to 4 on e2e, so RAM/heap over-commitment matters, but not /dev/shm.

Theory: general RAM/heap

I ran the test with --memory=3g which should be too little for two parallel tests (it's half the size of production) -- our nond machines alone already take > 2 GiB. Interestingly, nothing inside of the container actually seems to care -- free, top etc. all show 16 GiB (my laptop's RAM size). Looking at the container cgroup:

❱❱❱ cat /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/user.slice/libpod-dd7deee8be573bedbb870c342ec3f25fd649d257118a775087690607153431a6.scope/memory.{max,peak}
3221225472
3221237760

i.e. it does hit the roof, but nothing consequential -- the tests run happily. Then again, with --memory=1g the rpmbuild inside of the VM fails, so this option does something. With 2g it successfully builds, but both qemu and browser (headless_shell) get hit over the head by the OOM killer. So, conclusions: (1) that memory limit option does work, (2) even half the allocated size is enough, and (3) it doesn't fail in the observed mode.

[martinpitt]

One day later after the upgrade, we now see the same violation on all our machines, including rhos:

Feb 28 13:24:05 rhos-01-1 audit[205724]: AVC avc:  denied  { execheap } for  pid=205724 comm="ThreadPoolForeg" scontext=system_u:system_r:container_t:s0:c114,c423 tcontext=system_u:system_r:container_t:s0:c114,c423 tclass=process permissive=1

This seems to be a regression between selinux-policy-targeted-38.28-1.fc39.noarch and selinux-policy-targeted-39.3-1.fc39.noarch then.

There are some similar reports , and the general tone seems to be that this is intended -- heap shouldn't be writable and executable at the same time. So Chromium is doing something dubious here.

The other dubious thing is this "c114,c423" type range, but I don't know enough about SELinux to know if that's intended.

Combining ausearch which is installed on classic Fedora, with audit2allow which is installed on IoT 🤯 , I get such a my-ThreadPoolForeg.te policy drop-in:

module my-ThreadPoolForeg 1.0;
require {
	type container_t;
	class process execheap;
}

#============= container_t ==============
allow container_t self:process execheap;

This can be built and activated with

checkmodule -M -m -o my-ThreadPoolForeg.mod my-ThreadPoolForeg.te
semodule_package -o my-ThreadPoolForeg.pp -m my-ThreadPoolForeg.mod
semodule -i my-ThreadPoolForeg.pp

I sent PR #580 to ansiblify this and rolled it out. Let's check tomorrow if it worked.

[martinpitt]

We changed another variable: cockpit-project/cockpit#20107 (comment)

The tasks container now has chromium 122 which fixed at least one "NOTREACHED" assertion. If there are no violations tomorrow, try to undo the SELinux policy and see if it happens again.

[martinpitt]

I ran ansible -f20 -i inventory -m shell -a 'journalctl --since "18 hours ago" | grep AVC.*denied' e2e and also on openstack_tasks, and confirm that there are no violations at all.

So I ran ansible -f20 -i inventory -m shell -a 'semodule -r cockpituous' e2e (and on openstack_tasks) to see if this is still necessary with Chromium 122. I'll check tomorrow/Monday.

[mvollmer]

So Chromium is doing something dubious here.

The JavaScript JIT generates machine code at run time and thus needs executable memory. But the devel is probably in the details.

[martinpitt]

After I've done that, there are plenty of rejections again:

Mar 01 12:41:20 rhos-01-11 audit[429316]: AVC avc:  denied  { execheap } for  pid=429316 comm="ThreadPoolForeg" scontext=system_u:system_r:container_t:s0:c132,c559 tcontext=system_u:system_r:container_t:s0:c132,c559 tclass=process permissive=1

So, (1) this was unrelated to the Chromium update, and (2) let's land #580 to fix this.

[martinpitt]

References:

• https://bugzilla.redhat.com/show_bug.cgi?id=2254434
• https://lore.kernel.org/all/[email protected]/