Skip to content

Commit

Permalink
Feature/nix support (#131)
Browse files Browse the repository at this point in the history
# Description

This is a WIP PR to add nix support to the repo. The aim is to combat
the packaging requirements around cuda and tket.

It is not ready for merging yet. For effective use there are three
things remaining:
- Documentation
- Remote Caching with cachix. At current, cachix does not skip uploading
packages that are already available in the upstream
cuda-maintainers.cachix.org cache, so a push would be quite large and
fill up precious space. I have reached out to Domen and he has told me
that an update to help us on this will be out next week.
- Usage with non-nixos systems. I have successfully run this flake on an
ubuntu docker image with cuda drivers loaded, but it is awkward. We need
to make use of a third-party NixGL flake and wrap binaries in it, in
order to have the system's cuda drivers available for nix to use. I want
to improve this somewhat, otherwise it will be confusing for users.

When this is ready and merged, CI should build any changes and upload
them to cachix. I will pre-populate cachix with the heavy dependencies
to make this process quick and easy. When this is all in place, we can
think about use the nix flake for testing in CI using a gpu-enabled
machine without being concerned about the awkward dependency setup
process.

# Related issues

N/A

# Checklist

- [x] I have run the tests on a device with GPUs.
- [ ] I have performed a self-review of my code.
- [ ] I have commented hard-to-understand parts of my code.
- [ ] I have made corresponding changes to the public API documentation.
- [x] I have added tests that prove my fix is effective or that my
feature works.
- [ ] I have updated the changelog with any user-facing changes.
  • Loading branch information
jake-arkinstall authored Aug 1, 2024
2 parents c210d6b + 8aac308 commit 1c5d293
Show file tree
Hide file tree
Showing 14 changed files with 720 additions and 1 deletion.
41 changes: 41 additions & 0 deletions .github/workflows/build-with-nix.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: build with nix

on:
pull_request:
branches:
- main
workflow_dispatch: {}

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
strategy:
matrix:
os: ['ubuntu-24.04']
runs-on: ${{matrix.os}}
steps:
- uses: actions/checkout@v4
- uses: cachix/install-nix-action@V27
- uses: cachix/cachix-action@v15
with:
name: tket
authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
- name: Build pytket-cutensornet
run: nix build --accept-flake-config
#
# need GPU runners for this
#
#test:
# needs: build
# runs-on: nixos-gpu
# steps:
# - uses: actions/checkout@v4
# - uses: cachix/install-nix-action@V27
# - uses: cachix/cachix-action@v15
# with:
# name: tket
# authToken: '${{ secrets.CACHIX_AUTH_TOKEN }}'
# - name: Test pytket-cutensornet
# run: nix run .#tests --accept-flake-config
6 changes: 5 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,8 @@ dist
obj
docs/extensions
.ipynb_checkpoints
pytket/extensions/cutensornet/_metadata.py
pytket/extensions/cutensornet/_metadata.py
# nix artifacts
result
.direnv
.envrc
117 changes: 117 additions & 0 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

52 changes: 52 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
{
description = "Pytket Cutensornet Extension";
nixConfig.extra-substituters = "https://tket.cachix.org https://cache.nixos.org https://cuda-maintainers.cachix.org https://nix-community.cachix.org";
nixConfig.trusted-public-keys = ''
tket.cachix.org-1:ACdm5Zg19qPL0PpvUwTPPiIx8SEUy+D/uqa9vKJFwh0=
cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
cuda-maintainers.cachix.org-1:0dq3bujKpuEPMCX6U4WylrUDZ9JyUG0VpVZa7CNfq5E=
nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
'';
inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.tket.url = "github:CQCL/tket";
inputs.nixpkgs.follows = "tket/nixpkgs";
outputs = { self, nixpkgs, flake-utils, tket }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
config = {
allowUnfree = true;
cudaSupport = true;
};
overlays = [
(self: super: {
inherit (tket.packages."${system}") tket pytket;
})
(self: super: {
cuda-bundle = pkgs.callPackage ./nix-support/cuda-bundle.nix {};
})
(self: super: {
cupy' = pkgs.python3Packages.callPackage ./nix-support/cupy.nix {};
})
(self: super: {
pycuquantum = pkgs.python3Packages.callPackage ./nix-support/pycuquantum.nix {};
})
(import ./nix-support/pytket-cutensornet.nix)
];
};
in {
packages = {
default = pkgs.pytket-cutensornet;
cupy = pkgs.cupy';
pytket-cutensornet = pkgs.pytket-cutensornet;
tests = pkgs.run-pytket-cutensornet-tests;
};
devShells = {
default = pkgs.mkShell { buildInputs = [ pkgs.pytket-cutensornet ]; };
};
checks = {
# no GPU support in checks at the time of writing
};
});
}
4 changes: 4 additions & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,7 @@ ignore_errors = True
[mypy-lark.*]
ignore_missing_imports = True
ignore_errors = True

[mypy-sympy.*]
ignore_missing_imports = True
ignore_errors = True
37 changes: 37 additions & 0 deletions nix-support/cuda-bundle.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
symlinkJoin,
cudaPackages,
}:
let
cutensor' = cudaPackages.callPackage ./cutensor.nix {};
cuquantum' = cudaPackages.callPackage ./cuquantum.nix {};
cusparselt' = cudaPackages.callPackage ./cusparselt.nix {};
in
symlinkJoin {
name = "cuda-bundle-${cudaPackages.cudaVersion}";
paths = with cudaPackages; [
cuda_cccl # <nv/target>
cuda_cccl.dev
cuda_cudart
cuda_nvcc.dev # <crt/host_defines.h>
cuda_nvcc
cuda_nvprof
cuda_nvrtc
cuda_nvtx
cuda_profiler_api
libcublas
libcufft
libcurand
libcusolver
libcusparse
cusparselt'
cusparselt'.dev
cutensor'
cutensor'.dev
cuquantum'
];
postBuild = ''
ln -sfn lib $out/lib64
'';
}

Loading

0 comments on commit 1c5d293

Please sign in to comment.