Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add macos 14 and only build macos 11 in mpi release #712

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

MFraters
Copy link
Member

@MFraters MFraters commented May 9, 2024

Macos 14 is the latest release, and also a arm version. So it would be good to see how well the tester works for this.

@MFraters MFraters added the testing enhancement Requests for improvements to the testing of the GWB label May 9, 2024
@MFraters MFraters force-pushed the add_macos14_tester branch from de1ca9b to 0477611 Compare May 9, 2024 14:01
@MFraters MFraters force-pushed the add_macos14_tester branch from 0477611 to 6ae0556 Compare May 9, 2024 14:05
@MFraters
Copy link
Member Author

MFraters commented May 9, 2024

16 out of the 132 tests have issues on mac 14 arm. It seems to me mostly either small difference in the output numbers, or some small difference in the computed position caused by round off error causing it to fall on the other side of a line. It may be possible to make some the tests less sensitive to these round off errors, but I am not sure it is the case for all of them. Another option is to disable these specific tests on arm testers.

@gassmoeller @tjhei opinions?

Copy link

github-actions bot commented May 9, 2024

Benchmark Main Feature Difference (99.9% CI)
Slab interpolation simple none 1.017 ± 0.004 (s=446) 1.019 ± 0.003 (s=440) +0.1% .. +0.3%
Slab interpolation curved simple none 1.018 ± 0.004 (s=464) 1.026 ± 0.011 (s=419) +0.6% .. +1.0%
Spherical slab interpolation simple none 1.176 ± 0.007 (s=395) 1.176 ± 0.006 (s=373) -0.1% .. +0.1%
Slab interpolation simple curved CMS 1.063 ± 0.006 (s=418) 1.062 ± 0.005 (s=431) -0.2% .. -0.0%
Spherical slab interpolation simple CMS 1.558 ± 0.009 (s=275) 1.557 ± 0.011 (s=305) -0.3% .. +0.1%
Spherical fault interpolation simple none 1.174 ± 0.007 (s=380) 1.175 ± 0.009 (s=389) -0.0% .. +0.3%
Cartesian min max surface 2.302 ± 0.016 (s=189) 2.306 ± 0.019 (s=204) -0.1% .. +0.4%
Spherical min max surface 7.240 ± 0.032 (s=60) 7.237 ± 0.030 (s=67) -0.3% .. +0.2%

@gassmoeller
Copy link
Contributor

Disabling tests for ARM seems a bit problematic, what if they are actual bugs that only surface on arm architecture? Making the tests less sensitive to roundoff or understanding why they fail seems like the safer path (and may also have benefits for future x86 tests on new operating systems). It is more work though, and maybe not the highest priority?

@tjhei
Copy link
Contributor

tjhei commented May 17, 2024

Do you use numdiff for comparing test output right now like we do for ASPECT?

@tjhei
Copy link
Contributor

tjhei commented May 17, 2024

I would also separate Linux and osx testing.

@MFraters
Copy link
Member Author

Thanks for the feedback. Yes, numdiff is being used.

I will take the longer route than in this case, although it may take a bit before I have time to address this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing enhancement Requests for improvements to the testing of the GWB
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants