Add Nix support for more reproductive build #344

SnO2WMaN · 2022-08-18T22:35:39Z

Description

Added some Nix files for more reproducible (or more no (global) environment configuration) builds. Primary goal is making it easier to handle png supported upstream SATySFi with Nix.

I forgot to check before working this, a similar attempt was made several month ago. #322

Current Status

I tested on Linux (x86_64-linux) but may be able to in x86_64-darwin. You need to enable Nix Flake.

nix build ".#satysfi" for build satysfi executable binary and lib-satysfi with fonts.
nix build ".#doc-demo" for build ./demo/demo.saty
nix build ".#doc-lang" for build ./doc/doc-lang.saty
nix build ".#doc-primitives" for build ./doc/doc-primitives.saty
nix run ".#satysfi" -- ./tests/images/test.saty or else to build document with latest version satysfi.

All results to ./result

SnO2WMaN · 2022-08-19T00:50:51Z

Currently can build by nix build ".#satysfi'" (Note that satysfi')

Copy src from https://github.com/NixOS/nixpkgs/blob/762b003329510ea855b4097a37511eb19c7077f0/pkgs/tools/typesetting/satysfi/default.nix

However, it is completely dependent on the version of nixpkgs.ocamlPackage, in other words, it does not consider satysfi.opam at all.

I found that tweag/opam-nix may have some problems.

because tweag/opam-nix#17

SnO2WMaN · 2022-08-19T21:45:28Z

Currently can't build document with png images ( supported in #329 ) and I don't know why

Happens error
https://github.com/SnO2WMaN/SATySFi/blob/1bc985722d86c9fe5d2a52145fdc221b4d225581/src/frontend/main.ml#L623-L628

Occurence (maybe)
https://github.com/SnO2WMaN/SATySFi/blob/1bc985722d86c9fe5d2a52145fdc221b4d225581/src/backend/loadRawImage.ml#L11-L14

camlimage not successfully resolved as a dependency?

gfngfn · 2022-08-20T16:39:14Z

Currently can't build document with png images ( supported in #329 ) and I don't know why

Happens error
https://github.com/SnO2WMaN/SATySFi/blob/1bc985722d86c9fe5d2a52145fdc221b4d225581/src/frontend/main.ml#L623-L628

Occurence (maybe)
https://github.com/SnO2WMaN/SATySFi/blob/1bc985722d86c9fe5d2a52145fdc221b4d225581/src/backend/loadRawImage.ml#L11-L14

Could you show us the displayed error message?

camlimage not successfully resolved as a dependency?

I think that that is unlikely. Succeeding in building SATySFi implies having succeeded in resolving dependency.

leque · 2022-08-20T17:37:22Z

I tried this branch and got an error like below:

$ nix run ".#satysfi" -- ./tests/images/test.saty
...
 ---- ---- ---- ----
  embedding fonts ...
! [Error] cannot load image file 'basn0g02.png';
      unsupported

This error seems to be raised by camlimages. I am afraid that camlimages is not built with png support. (It was not able to find libpng at configure time?)

SnO2WMaN · 2022-08-20T20:37:15Z

@leque
As you pointed out, I found that there is some problem between camlimages that Nix is (supposedly) building internally and libpng

Different output (more-detailed) when I switched to libpng12 (unpatched libpng 1.2.59)

 ---- ---- ---- ----
  embedding fonts ...
libpng warning: Invalid color type/bit depth combination in IHDR
libpng error: Invalid IHDR data
! [Error] cannot load image file 'basn0g02.png';
      png read error

SnO2WMaN · 2022-08-20T22:46:02Z

I have tested with overriding the official definition file for satysfi, and I found that the problem may be cause of either nixpkgs.ocamlPackages.camlimages or nixpkgs.libpng. Unfortunately, I'm not familiar with OCaml, so this pr will be postponed until some effective way of dealing with this problem.

balsoft · 2022-08-22T10:42:25Z

The aforementioned test passes if I run nix run . -- ./tests/images/test.saty with the following flake.nix and latest version of opam-nix (I have included some new overlays in there that fix camlimages and camlpdf here, so make sure to run nix flake lock --update-input opam-nix). I also took a liberty to fix your nix-filter invocation and remove some unneeded overlays.

{
  description = "Nix Flake for SATySFi";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    opam-nix.url = "github:tweag/opam-nix";
    satysfi-external-repo = {
      url = "github:gfngfn/satysfi-external-repo";
      flake = false;
    };

    # develop
    nix-filter.url =
      "github:numtide/nix-filter/3e1fff9ec0112fe5ec61ea7cc6d37c1720d865f8";
    devshell.url = "github:numtide/devshell";
    flake-utils.url = "github:numtide/flake-utils";
    flake-compat = {
      url = "github:edolstra/flake-compat";
      flake = false;
    };
  };
  outputs = { self, nixpkgs, nix-filter, opam-nix, satysfi-external-repo
    , devshell, flake-utils, ... }:
    flake-utils.lib.eachSystem [ "x86_64-linux" "x86_64-darwin" ] (system:
      let
        inherit (pkgs) lib stdenv;
        inherit (opam-nix.lib.${system}) buildDuneProject makeOpamRepoRec;
        pkgs = nixpkgs.legacyPackages.${system}.extend devshell.overlay;

        duneProject = buildDuneProject {
          repos = [
            opam-nix.inputs.opam-repository
            (makeOpamRepoRec satysfi-external-repo)
          ];
        } "satysfi" (with nix-filter.lib;
          filter {
            root = ./.;
            include = [
              "dune-project"
              "satysfi.opam"
              "Makefile"
              (inDirectory "bin")
              (inDirectory "obsolete")
              (inDirectory "src")
              (inDirectory "tools")
              (inDirectory "lib-satysfi")
            ];
            exclude = [ ".merlin" ];
          }) { satysfi = null; };
        duneProjectOverlay = self: super: {
          satysfi = super.satysfi.overrideAttrs (oa: {
            doNixSupport = false;
            removeOcamlReferences = true;
            preConfigure = ''
              substituteInPlace src/frontend/main.ml \
              --replace \
                '/usr/local/share/satysfi"; "/usr/share/satysfi' \
                $out/share/satysfi
            '';
            installPhase = with pkgs; ''
              cp -r ${lmodern}/share/fonts/opentype/public/lm/* lib-satysfi/dist/fonts/
              cp -r ${lmmath}/share/fonts/opentype/latinmodern-math.otf lib-satysfi/dist/fonts/
              cp -r ${ipaexfont}/share/fonts/opentype/* lib-satysfi/dist/fonts/
              cp -r ${junicode}/share/fonts/junicode-ttf/* lib-satysfi/dist/fonts/
              make install PREFIX=$out LIBDIR=$out/share/satysfi
              mkdir -p $out/share/satysfi/
              cp -r lib-satysfi/dist/ $out/share/satysfi/
            '';
          });
        };
        ocamlPkgs = duneProject.overrideScope' duneProjectOverlay;

        mkDoc = { name, root, entrypoint, output, }:
          stdenv.mkDerivation {
            name = "doc-${name}";
            buildInputs = with self.packages.${system}; [ satysfi ];
            src = with (nix-filter.lib);
              filter {
                root = root;
                exclude = [ (matchExt "pdf") (matchExt "satysfi-aux") ];
              };
            buildPhase = ''
              satysfi ${entrypoint} --output ${output}
            '';
            installPhase = ''
              mkdir -p $out
              cp ${output} $out
            '';
          };
      in {
        packages.default = self.packages.${system}.satysfi;
        packages.satysfi = ocamlPkgs.satysfi;

        packages.doc-demo = mkDoc {
          name = "demo";
          root = ./demo;
          entrypoint = "demo.saty";
          output = "demo.pdf";
        };
        packages.doc-lang = mkDoc {
          name = "lang";
          root = ./doc;
          entrypoint = "doc-lang.saty";
          output = "doc-lang.pdf";
        };
        packages.doc-primitives = mkDoc {
          name = "lang";
          root = ./doc;
          entrypoint = "doc-primitives.saty";
          output = "doc-primitives.pdf";
        };
        packages.tests-images = mkDoc {
          name = "tests-image";
          root = ./tests/images;
          entrypoint = "test.saty";
          output = "test.pdf";
        };

        apps.satysfi =
          flake-utils.lib.mkApp { drv = self.packages.${system}.satysfi; };

        apps.default = self.apps.${system}.satysfi;

        devShells.default = pkgs.devshell.mkShell {
          packages = with pkgs; [
            # develop
            treefmt # wrapper cli to run all formatters
            alejandra # Nix formatter
          ];
        };

        checks = {
          inherit (self.packages.${system})
            satysfi doc-demo doc-lang doc-primitives;
        };
      });
}

SnO2WMaN · 2022-08-22T19:26:23Z

@balsoft

I have tested that I can build documents with pngs here as well, thank you very much!. This PR is now undrafted.

nsfisis · 2024-06-20T18:00:14Z

I tried to build SATySFi by Nix with this branch and the latest master (879664e), but failed to build with the following error:

error: builder for '/nix/store/pngqw8wyxlrqw9223zzibk4ndm1gdgqb-resolve.drv' failed with exit code 20;¬                              last 5 log lines:
> [ERROR] No solution including optional dependencies for satysfi:   * Missing dependency:
>             - batteries >= 3.6.0
>             no matching version

I updated the fixed flake version by nix flake update and slightly modified flake.nix and satysfi.opam:

diff --git a/flake.nix b/flake.nix
index 6dec7065..0cd943cf 100644
--- a/flake.nix
+++ b/flake.nix
@@ -38,7 +38,7 @@
         pkgs = import nixpkgs {
           inherit system;
           overlays = [
-            devshell.overlay
+            devshell.overlays.default
           ];
         };

@@ -62,7 +62,7 @@
                 (inDirectory "lib-satysfi")
               ];
               exclude = [".merlin"];
-            }) {satysfi = null;};
+            }) {satysfi = "*";};
         duneProjectOverlay = self: super: {
           satysfi = super.satysfi.overrideAttrs (oa: {
             doNixSupport = false;
diff --git a/satysfi.opam b/satysfi.opam
index 4a2ff9b6..282855e3 100644
--- a/satysfi.opam
+++ b/satysfi.opam
@@ -20,7 +20,7 @@ remove: [
 ]
 # Packages whose version suffix is "+satysfi" are distributed on satysfi-external-repo.
 depends: [
-  "ocaml" {>= "4.12.0" & < "5.0.0"}
+  "ocaml" {>= "4.13.0" & < "5.0.0"}
   "batteries" {>= "3.6.0" & < "4.0.0"}
   "camlimages" {>= "5.0.1" & < "5.0.5"}
   "camlpdf" {= "2.3.1+satysfi"}

These two changes are for the new versions of devshell and opam-nix.

-            devshell.overlay
+            devshell.overlays.default
-            }) {satysfi = null;};
+            }) {satysfi = "*";};

This change is for addressing issue where OCaml 4.12.0 was unable to be built by Nix for some reason.

-  "ocaml" {>= "4.12.0" & < "5.0.0"}
+  "ocaml" {>= "4.13.0" & < "5.0.0"}

EDIT: This is unnecessary. OCaml 4.12.0 works with the changes that I commented below.

nsfisis · 2024-08-14T11:57:50Z

Because the version of dune was upgraded recently, Nix build got broken again. I needed to apply these changes to fix it:

apply Add Nix support for more reproductive build #344 (comment)
override nixpkgs and opam-repository of opam-nix to use the latest dune.
fix installPhase to execute the original installPhase.

--- a/flake.nix
+++ b/flake.nix
@@ -3,7 +3,15 @@

   inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
-    opam-nix.url = "github:tweag/opam-nix";
+    opam-repository = {
+      url = "github:ocaml/opam-repository";
+      flake = false;
+    };
+    opam-nix = {
+      url = "github:tweag/opam-nix";
+      inputs.nixpkgs.follows = "nixpkgs";
+      inputs.opam-repository.follows = "opam-repository";
+    };
     satysfi-external-repo = {
       url = "github:gfngfn/satysfi-external-repo";
       flake = false;

--- a/flake.nix
+++ b/flake.nix
@@ -81,7 +81,7 @@
                 '/usr/local/share/satysfi"; "/usr/share/satysfi' \
                 $out/share/satysfi
             '';
-            installPhase = with pkgs; ''
+            installPhase = with pkgs; oa.installPhase + ''
               cp -r ${lmodern}/share/fonts/opentype/public/lm/* lib-satysfi/dist/fonts/
               cp -r ${lmmath}/share/fonts/opentype/latinmodern-math.otf lib-satysfi/dist/fonts/
               cp -r ${ipaexfont}/share/fonts/opentype/* lib-satysfi/dist/fonts/

nsfisis · 2024-08-15T12:45:08Z

My fixed version:

4d27f13...nsfisis:SATySFi:7fcd6f95452189790958124df44c21be77d1503f

I merged the branch of this pull request into the latest master and added some fixes desribed above.

SnO2WMaN added 2 commits August 19, 2022 03:56

Add Nix Flake

303da9b

Add build bin wip

e0146c2

SnO2WMaN changed the title ~~Add Nix Flake for more reproductive build~~ [WIP] Add Nix Flake for more reproductive build Aug 18, 2022

SnO2WMaN changed the title ~~[WIP] Add Nix Flake for more reproductive build~~ Add Nix Flake for more reproductive build Aug 18, 2022

can build ".#satysfi'"

70fd4db

SnO2WMaN added 3 commits August 19, 2022 10:49

can build .#satysfi

55afd23

all in one ".#satysfi"

e7e7af5

Add document build

650dd81

SnO2WMaN mentioned this pull request Aug 19, 2022

Filtering project without --impure option for build*Project tweag/opam-nix#17

Closed

SnO2WMaN added 3 commits August 20, 2022 04:16

Ignore .vscode/settings.json

497b5c2

src filtering (can't)

f5308a3

because tweag/opam-nix#17

addd x86_64-darwin

1bc9857

SnO2WMaN marked this pull request as ready for review August 19, 2022 19:37

SnO2WMaN changed the title ~~Add Nix Flake for more reproductive build~~ Add Nix support for more reproductive build Aug 19, 2022

SnO2WMaN marked this pull request as draft August 19, 2022 20:05

Add build ./tests/images/test.saty

feb5201

SnO2WMaN added 2 commits August 21, 2022 05:40

duneProject outside for camlimages

4962bf2

overriding existing satysfi definition version

155de26

fix nix-filter version

192dc60

Update opam-nix and can build w/ png

2dbe7be

SnO2WMaN marked this pull request as ready for review August 22, 2022 19:26

fix typo

40bfbae

gfngfn added this to the v0.0.12 milestone Apr 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Nix support for more reproductive build #344

Add Nix support for more reproductive build #344

SnO2WMaN commented Aug 18, 2022 •

edited

Loading

SnO2WMaN commented Aug 19, 2022 •

edited

Loading

SnO2WMaN commented Aug 19, 2022 •

edited

Loading

gfngfn commented Aug 20, 2022 •

edited

Loading

leque commented Aug 20, 2022

SnO2WMaN commented Aug 20, 2022 •

edited

Loading

SnO2WMaN commented Aug 20, 2022

balsoft commented Aug 22, 2022 •

edited

Loading

SnO2WMaN commented Aug 22, 2022 •

edited

Loading

nsfisis commented Jun 20, 2024 •

edited

Loading

nsfisis commented Aug 14, 2024 •

edited

Loading

nsfisis commented Aug 15, 2024

Add Nix support for more reproductive build #344

Are you sure you want to change the base?

Add Nix support for more reproductive build #344

Conversation

SnO2WMaN commented Aug 18, 2022 • edited Loading

Description

Current Status

SnO2WMaN commented Aug 19, 2022 • edited Loading

SnO2WMaN commented Aug 19, 2022 • edited Loading

gfngfn commented Aug 20, 2022 • edited Loading

leque commented Aug 20, 2022

SnO2WMaN commented Aug 20, 2022 • edited Loading

SnO2WMaN commented Aug 20, 2022

balsoft commented Aug 22, 2022 • edited Loading

SnO2WMaN commented Aug 22, 2022 • edited Loading

nsfisis commented Jun 20, 2024 • edited Loading

nsfisis commented Aug 14, 2024 • edited Loading

nsfisis commented Aug 15, 2024

SnO2WMaN commented Aug 18, 2022 •

edited

Loading

SnO2WMaN commented Aug 19, 2022 •

edited

Loading

SnO2WMaN commented Aug 19, 2022 •

edited

Loading

gfngfn commented Aug 20, 2022 •

edited

Loading

SnO2WMaN commented Aug 20, 2022 •

edited

Loading

balsoft commented Aug 22, 2022 •

edited

Loading

SnO2WMaN commented Aug 22, 2022 •

edited

Loading

nsfisis commented Jun 20, 2024 •

edited

Loading

nsfisis commented Aug 14, 2024 •

edited

Loading