feat: Build bunq2ynab image with nix and Bazel

[mvgijssel]

ref #451

History trying to get nix to work inside BuildBuddy

• ❌ Install nix using https://github.com/DeterminateSystems/nix-installer, but this will make nix only available to the root user and not the buildbuddy user
• ❌ Use determinate installer and create binstubs which sudo the binary but get error

error: getting status of '/home/buildbuddy/workspace/output-base/external/raw_python38_base_image/bazel-support/nix-out-link': No such file or directory

• ❌ Use nix-portable which fails due to Fails to run with vfs mount DavHau/nix-portable#74
• ❌ Install nix directly as described here https://zameermanji.com/blog/2023/3/26/using-nix-without-root/#fn1, but this gives the nix binary not the nix-build binary which is required.
• ❌ Create a nix-build file and forward to docker, but the docker daemon is not running inside the BuildBuddy workflow executors
• ✅ Use determinate installer and chown the /nix directory to buildbuddy user

TODO:

• Make nix work inside BuildBuddy (chown -R buildbuddy:buildbuddy /nix)
• Setup rules_oci
• Create image with the nix archive as a starting point
• Figure out how to switch architecture inside nix
• Setup Python rule for images https://github.com/aspect-build/bazel-examples/tree/main/oci_python_image
• Setup alternative platform and toolchain for py_binary, can we use hermetic interpreter? (for example https://github.com/vgijssel/setup/pull/495/files)
• Fix runfiles location
• Fix AssertionError: Cannot find .runfiles directory for /opt/hello_world in BuildBuddy CI
• turn rules_task into py_binary (Convert rules_task binary into regular py_binary #588)
• create py_image macro
• wire up with bunq2ynab
• fix not copying hermetic python interpreter into base image
• Make sure the bunq2ynab image actually works (with 1password)
• Make sure locally the image works on macOS, currently the .so from darwin are copied into the image?
• Use requirement helper everywhere
• Try pycross with pdm
• Use hermetic interpreter inside pycross
• Fix mtree specification pointing to pycross dependencies as a file instead of directory ([Bug]: tar mtree should expand treeartifacts automatically bazel-contrib/bazel-lib#625 (comment))
• symlink bsdtar to tar into path in devbox
• Cleanup unnecessary packages from base images (like vim etc)
• Cleanup platform targets
• Cleanup pycross targets
• Setup lock diff test pdm bazel lock file
• Remove requirements + rules from setup
• Extract rules_task changes (refactor: Remove unused platforms attribute #598)

How the end result macro should look:

py_image(
  name = "bunq2ynab_image",
  binary = ":bunq2ynab",
  base = "@raw_python38_base_image//:image",
)

The py_image rule will wrap the binary with a task which sets the target_platforms flag to containerized to make sure we are not copying in the hermetic interpreter.

[mvgijssel]

Steps to setup on macOS. Following the documentation from https://nixos.org/manual/nixpkgs/stable/#chap-special

1. Run linux builder machine using nix run --extra-experimental-features nix-command --extra-experimental-features flakes nixpkgs#darwin.linux-builder

2. Setup /etc/nix/nix.conf with the following

build-users-group = nixbld
builders = ssh://builder@localhost aarch64-linux /etc/nix/builder_ed25519 4

3. Update /var/root/.ssh/config with

Host localhost
  User builder
  HostName 127.0.0.1
  Port 31022
  IdentityFile /etc/nix/builder_ed25519

4. Restart the nix-daemon

sudo launchctl kickstart -k system/org.nixos.nix-daemon

[mvgijssel]

Copied an example from https://github.com/jvolkman/bazel-nix-example/tree/134e5c9d66b7e3baa0f29ce049bf27d9794f352b. It kinda works, if I modify the app.binary inside the docker image to point to the right python executable (instead of using the embedded one).

Let's try if we can make this work with rules_oci

[mvgijssel]

Asking for help in the BuildBuddy community https://buildbuddy.slack.com/archives/CUY16GNK1/p1695741392487519?thread_ts=1695126450.569509&cid=CUY16GNK1

[mvgijssel]

Now running into an issue with wheels

bash-5.2# /opt/tools/bunq2ynab/bunq2ynab
/config.json
Traceback (most recent call last):
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/bunq2ynab/bunq2ynab.py", line 1, in <module>
    from lib.sync import Sync
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/bunq2ynab/lib/sync.py", line 5, in <module>
    from lib import bunq_api
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/bunq2ynab/lib/bunq_api.py", line 1, in <module>
    from lib import bunq
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/bunq2ynab/lib/bunq.py", line 2, in <module>
    from OpenSSL import crypto
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/rules_python~0.25.0~pip~pip-setup_310_pyopenssl/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import SSL, crypto
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/rules_python~0.25.0~pip~pip-setup_310_pyopenssl/site-packages/OpenSSL/SSL.py", line 9, in <module>
    from OpenSSL._util import (
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/rules_python~0.25.0~pip~pip-setup_310_pyopenssl/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/opt/tools/bunq2ynab/bunq2ynab.runfiles/rules_python~0.25.0~pip~pip-setup_310_cryptography/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module>
    from cryptography.hazmat.bindings._rust import _openssl, openssl
ImportError: /opt/tools/bunq2ynab/bunq2ynab.runfiles/rules_python~0.25.0~pip~pip-setup_310_cryptography/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: invalid ELF header

As mentioned here pyca/cryptography#6378 (comment) this might be because the wrong wheel files are downloaded by Bazel or that the nix base image is missing necessary files. The latter would also explain why the hermetic interpreter wouldn't run in the first place 🤔