CI fails on macOS M1

[zaneenders]

Hi, Getting the following error when running one of the CI commands locally on main right now. This error does not seem to show up on gh-actions.

With the following setup.
macOS 14.5 (23F79) M1
Welcome to Racket v8.13 [cs].
rustc 1.79.0 (129f3b996 2024-06-10) (Homebrew)

➜  herbie git:(main) racket -y infra/ci.rkt --precision binary64 --seed 0 bench/hamming/
Running Herbie on 28 tests, seed: #(1062452524 2961816100 342112271 2854655039 3321940838 3542344109)
  1/28  [   5.954s]   30→ 0     expax (section 3.5)
  2/28  [   4.576s]   40→ 0     expq2 (section 3.11)
Warning: could not determine a ground truth
  a = -3.123184171133734e-285
  b = 6.897881534926379e+187
  c = -3.8172071442738415e-36
See <https://herbie.uwplse.org/doc/2.2/faq.html#ground-truth> for more.
  3/28  [   6.596s]   30→ 9     quadp (p42, positive)
Warning: could not determine a ground truth
  a = -1.3101822340751238e-253
  b = -1.7603332822549246e+137
  c = -1.0698332799219484e-122
See <https://herbie.uwplse.org/doc/2.2/faq.html#ground-truth> for more.
  4/28  [   5.826s]   31→ 9     quadm (p42, negative)
  5/28  [   5.036s]   31→ 9     quad2m (problem 3.2.1, negative)
  6/28  [   5.060s]   31→ 9     quad2p (problem 3.2.1, positive)
  7/28  [   4.977s]   60→ 0     2sqrt (example 3.1)
  8/28  [   5.604s]   40→ 1     2isqrt (example 3.6)
  9/28  [   4.810s]   15→ 0     2frac (problem 3.3.1)
 10/28  [   5.342s]   19→ 0     3frac (problem 3.3.3)
 11/28  [   7.333s]   59→ 2     2cbrt (problem 3.3.4)

Input (59.38752497735161 bits):
(-.f64 (cbrt.f64 (+.f64 x #s(literal 1 binary64))) (cbrt.f64 x))

Output (1.7246714426775525 bits):
(/.f64 (cbrt.f64 x) (*.f64 x #s(literal 3 binary64)))

Target (0.5984404203645186 bits):
((!
  :herbie-platform
  default
  (/
   1
   (+
    (* (cbrt (+ x 1)) (cbrt (+ x 1)))
    (* (cbrt x) (cbrt (+ x 1)))
    (* (cbrt x) (cbrt x)))))
 .
 #t)

[pavpanchekha]

You say this passes in CI. Does the seed match what's in CI? Do you have a clean checkout? Can you delete the "compiled" folders everywhere and recheck?

[zaneenders]

Ok so I
rm -rf herbie
git clone herbie ..
make install
racket -y infra/ci.rkt --precision binary64 --seed 0 bench/hamming/
Still seeing the failure on my MacBook Air.
By passes Ci the gh-actions pass but not on my two Macs(Both ARM).
The GH action always seem to run with --seed 0 so I think I'm testing correctly.

[pavpanchekha]

Confirmed locally.

[pavpanchekha]

Update: I can reproduce a difference between my laptop (macOS) and the nightly server (Ubuntu 22.04). I can also confirm that the same points are sampled, but the errors differ slightly (math library quality?). I don't know yet if that causes the underlying divergence (I hope not).

[pavpanchekha]

Looks like the divergence is in simplification, not the actual sampling (thank goodness). I'm seeing a batch simplify two different ways. Working with @bksaiki to fix.

[pavpanchekha]

Here are two reports, for the same benchmark (2cbrt from Hamming) on the same seed. Pay attention to the first main simplify (search the page for "48 → 46"). Here the same input batch is simplified in two different ways. The macos/out version is using Racket 8.13 on M1 macOS, while the linux/out2 version is using Racket 8.11 on Ubuntu Linux.

linux.tgz
macos.tgz

[bksaiki]

The egg library produces different results on different architectures for the same expressions and rules. An issue has been filed with egg: egraphs-good/egg#328

[pavpanchekha]

I suppose we should close it, since we've filed the related bug in egg. Closing as "wontfix" which I want you to think of as "not our fault"