add MacOS testing

[tomeichlersmith]

Currently, I have templated out MacOS testing within the testing workflow.

denv/.github/workflows/test.yml

Lines 63 to 77 in 80efb6d

	# test-macos:
	# name: Test on MacOS
	# runs-on: macos-latest
	# steps:
	# - uses: douglascamata/setup-docker-macos-action@v1-alpha
	# - uses: actions/checkout@v4
	# - name: enable debug mode if requested
	# if: ${{runner.debug == '1'}}
	# run: echo "DENV_DEBUG=1" >> "$GITHUB_ENV"
	# - name: Install
	# run: sudo ./install
	# - name: deduce runner name
	# run: echo "DENV_RUNNER=docker" >> "$GITHUB_ENV"
	# - name: Test
	# run: ./ci/test

It is commented out due to a few complexities.

1. GitHub macos runners do not currently provide an installation of docker and/or podman. This means we either have to (a) somehow test denv without a container runner or (b) install a container runner during the action.
2. I tried using https://github.com/douglascamata/setup-docker-macos-action to initialize docker since that is probably the most popular runner on macos, which appears to work but it left me with some odd issues within denv that are difficult to debug since I don't have MacOS locally.
   • In the past when I tried running with this, I attempted mounting the entrypoint ro at a special (root) location. This got around the issue of being unable to launch the container, but the tests still failed.

[tomeichlersmith]

#102 encouraged me to push a few more ideas into the MacOS CI testing, which has illuminated a few issues.

---

Switching from --user UID:GID to --userns=host breaks the Linux-host, docker-runner tests, so I tried including both. Those tests are what I'm reporting on here. I've copied the full log output below for future reference since the logs get deleted within 30ish days.

Copy of Full bats Test Printout

 INFO: Testing denv with 'docker'
1..31
ok 1 print version of denv
ok 2 print check help
ok 3 basic check run
ok 4 quiet check run
ok 5 check fails when using unsupported runner
ok 6 check that we are in a workspace
ok 7 check that we are not in a workspace
not ok 8 print config
# tags: norunner
# (from function `assert_line' in file test/test_helper/bats-assert/src/assert_line.bash, line 202,
#  in test file test/config.bats, line 56)
#   `assert_line --index 0 "denv_workspace=\"${PWD}\""' failed
#
# -- line differs --
# index    : 0
# expected : denv_workspace="/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/config.bats-8-oCmHnG"
# actual   : denv_workspace="/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/config.bats-8-oCmHnG"
# --
#
ok 9 change image being used
ok 10 add a new mount
ok 11 change shell program
ok 12 disable network connection
not ok 13 denv name available in denv
# (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
#  in test file test/env-var.bats, line 17)
#   `assert_success' failed
#
# -- command failed --
# status : 1
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/env-var.bats-1-1tHISE/.bashrc': Permission denied
# --
#
not ok 14 we can share host environment variables
# (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
#  in test file test/env-var.bats, line 24)
#   `assert_success' failed
#
# -- command failed --
# status : 1
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/env-var.bats-2-xiz8aN/.bashrc': Permission denied
# --
#
not ok 15 we can prevent sharing of host env vars
# (from function `refute_output' in file test/test_helper/bats-assert/src/refute_output.bash, line 173,
#  in test file test/env-var.bats, line 33)
#   `refute_output' failed
#
# -- output non-empty, but expected no output --
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/env-var.bats-3-wm3EIc/.bashrc': Permission denied
# --
#
not ok 16 we can copy specific env vars
# (from function `assert_output' in file test/test_helper/bats-assert/src/assert_output.bash, line 194,
#  from function `assert_container_output' in file test/test_helper/denv.bash, line 36,
#  in test file test/env-var.bats, line 43)
#   `assert_container_output "buz"' failed
#
# -- output differs --
# expected : buz
# actual   : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/env-var.bats-4-KqALVx/.bashrc': Permission denied
# --
#
not ok 17 we can set (and override) env vars
# (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
#  in test file test/env-var.bats, line 51)
#   `assert_success' failed
#
# -- command failed --
# status : 1
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/env-var.bats-5-1zRHL4/.bashrc': Permission denied
# --
#
not ok 18 pass exit code onto caller
# (in test file test/exit-codes.bats, line 16)
#   `run -42 denv exit 42' failed, expected exit code 42, got 1
# cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/exit-codes.bats-1-Qg5gdy/.bashrc': Permission denied
not ok 19 call a simple executable non-interactively
# (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
#  in test file test/exit-codes.bats, line 24)
#   `assert_success' failed
#
# -- command failed --
# status : 1
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/exit-codes.bats-2-gCKjL5/.bashrc': Permission denied
# --
#
not ok 20 by-pass broken .bashrc if non-interactive
# (in test file test/exit-codes.bats, line 30)
#   `run -6 denv' failed, expected exit code 6, got 1
# cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/exit-codes.bats-3-3yOoLb/.bashrc': Permission denied
ok 21 no init denv should fail
ok 22 can print denv help without a denv
ok 23 can print denv init help without a denv
ok 24 can print denv config help without a denv
ok 25 can print denv config env help without a denv
ok 26 denv can init
ok 27 denv should not init twice
not ok 28 setup_file failed
# (from function `setup_file' in test file test/network.bats, line 9)
#   `cp -t . ${OLDPWD}/test/internet-access.py' failed with status 64
# Using default tag: latest
# latest: Pulling from library/python
# 71215d55680c: Pulling fs layer
# 3cb8f9c23302: Pulling fs layer
# 5f899db30843: Pulling fs layer
# 567db630df8d: Pulling fs layer
# d68cd2123173: Pulling fs layer
# 63941d09e532: Pulling fs layer
# 097431623722: Pulling fs layer
# 09527fa4de8d: Pulling fs layer
# 567db630df8d: Waiting
# d68cd2123173: Waiting
# 63941d09e532: Waiting
# 097431623722: Waiting
# 09527fa4de8d: Waiting
# 3cb8f9c23302: Verifying Checksum
# 3cb8f9c23302: Download complete
# 71215d55680c: Verifying Checksum
# 71215d55680c: Download complete
# d68cd2123173: Verifying Checksum
# d68cd2123173: Download complete
# 63941d09e532: Verifying Checksum
# 63941d09e532: Download complete
# 097431623722: Verifying Checksum
# 097431623722: Download complete
# 5f899db30843: Verifying Checksum
# 5f899db30843: Download complete
# 09527fa4de8d: Verifying Checksum
# 09527fa4de8d: Download complete
# 567db630df8d: Verifying Checksum
# 567db630df8d: Download complete
# 71215d55680c: Pull complete
# 3cb8f9c23302: Pull complete
# 5f899db30843: Pull complete
# 567db630df8d: Pull complete
# d68cd2123173: Pull complete
# 63941d09e532: Pull complete
# 097431623722: Pull complete
# 09527fa4de8d: Pull complete
# Digest: sha256:19973e1796237522ed1fcc1357c766770b47dc15854eafdda055b65953fe5ec1
# Status: Downloaded newer image for python:latest
# docker.io/library/python:latest
# cp: illegal option -- t
# usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file target_file
#        cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file ... target_directory
not ok 30 whoami matches inside and outside denv
# (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
#  in test file test/ownership.bats, line 17)
#   `assert_success' failed
#
# -- command failed --
# status : 1
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/ownership.bats-1-FjFIvy/.bashrc': Permission denied
# --
#
not ok 31 file created inside denv has id match
# (from function `assert_equal' in file test/test_helper/bats-assert/src/assert_equal.bash, line 40,
#  in test file test/ownership.bats, line 23)
#   `assert_equal "$(stat -c %u file-from-denv)" "$(id -u ${USER})"' failed
# stat: illegal option -- c
# usage: stat [-FLnq] [-f format | -l | -r | -s | -x] [-t timefmt] [file ...]
#
# -- values do not equal --
# expected : 501
# actual   :
# --
#
# bats warning: Executed 30 instead of expected 31 tests

---

The directory seen by denv and the one "seen" by the host are different. denv has an extra /private prefix (the actual) compared to what the host sees (the expected which is just the value of ${PWD} in this test). This breaks several of the tests, but most obviously the test attempting to print and check the configuration.

 not ok 8 print config
# tags: norunner
# (from function `assert_line' in file test/test_helper/bats-assert/src/assert_line.bash, line 202,
#  in test file test/config.bats, line 56)
#   `assert_line --index 0 "denv_workspace=\"${PWD}\""' failed
#
# -- line differs --
# index    : 0
# expected : denv_workspace="/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/config.bats-8-oCmHnG"
# actual   : denv_workspace="/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/config.bats-8-oCmHnG"
# --

---

_denv_entrypoint via docker does not have permission to write into the workspace directory. This also breaks many tests, for example

 not ok 13 denv name available in denv
# (from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
#  in test file test/env-var.bats, line 17)
#   `assert_success' failed
#
# -- command failed --
# status : 1
# output : cp: cannot create regular file '/private/var/folders/h1/8hndypj13nsbj5pn4xsnv1tm0000gn/T/env-var.bats-1-1tHISE/.bashrc': Permission denied
# --

---

The tests are not written with MacOS's non-POSIX utils in mind. Test 28 fails to get setup since I use the -t flag of cp which isn't available on MacOS.

not ok 28 setup_file failed
# (from function `setup_file' in test file test/network.bats, line 9)
#   `cp -t . ${OLDPWD}/test/internet-access.py' failed with status 64
... image being downloaded output ...
# cp: illegal option -- t
# usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file target_file
#        cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file ... target_directory

Test 31 also fails due to a difference in how stat works.

not ok 31 file created inside denv has id match
# (from function `assert_equal' in file test/test_helper/bats-assert/src/assert_equal.bash, line 40,
#  in test file test/ownership.bats, line 23)
#   `assert_equal "$(stat -c %u file-from-denv)" "$(id -u ${USER})"' failed
# stat: illegal option -- c
# usage: stat [-FLnq] [-f format | -l | -r | -s | -x] [-t timefmt] [file ...]
#
# -- values do not equal --
# expected : 501
# actual   :
# --

[tomeichlersmith]

I get the same errors shown above using colima and docker instead of the setup-docker-macos action.

brew install colima docker
colima start
./ci/test docker

Edit: which makes sense since the action is just a wrapper around setting up colima with more well tuned settings for the runner environment.