Create separate build derivations for each dependency

[djahandarie]

Currently import-cargo just builds a derivation full of vendored source code.

It'd be quite amazing if it could build a set of derivations that compile each dependency separately (in order to avoid recompiling things unnecessarily).

Currently cargo (nightly) provides --build-plan, which outputs a plan full of the relevant rustc calls/args/env variables and the dependency tree. It seems like it should be possible to use this to create the set of nix derivations (with a little bit of rewriting of paths and -L inputs etc.)

A snip of a bit of the --build-plan output for a single dependency in the middle of my tree:

    {
      "package_name": "syn",
      "package_version": "0.11.11",
      "target_kind": [
        "lib"
      ],
      "kind": "Host",
      "compile_mode": "build",
      "deps": [
        1,
        3,
        2
      ],
      "outputs": [
        "/home/darius/whist-rust/target/release/deps/libsyn-45a26153f0636545.rlib",
        "/home/darius/whist-rust/target/release/deps/libsyn-45a26153f0636545.rmeta"
      ],
      "links": {},
      "program": "rustc",
      "args": [
        "--crate-name",
        "syn",
        "/home/darius/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-0.11.11/src/lib.rs",
        "--error-format=json",
        "--json=diagnostic-rendered-ansi,artifacts",
        "--crate-type",
        "lib",
        "--emit=dep-info,metadata,link",
        "-C",
        "opt-level=3",
        "-C",
        "debuginfo=2",
        "--cfg",
        "feature=\"default\"",
        "--cfg",
        "feature=\"parsing\"",
        "--cfg",
        "feature=\"printing\"",
        "--cfg",
        "feature=\"quote\"",
        "--cfg",
        "feature=\"synom\"",
        "--cfg",
        "feature=\"unicode-xid\"",
        "--cfg",
        "feature=\"visit\"",
        "-C",
        "metadata=45a26153f0636545",
        "-C",
        "extra-filename=-45a26153f0636545",
        "--out-dir",
        "/home/darius/whist-rust/target/release/deps",
        "-L",
        "dependency=/home/darius/whist-rust/target/release/deps",
        "--extern",
        "quote=/home/darius/whist-rust/target/release/deps/libquote-103bc26133e54dfe.rmeta",
        "--extern",
        "synom=/home/darius/whist-rust/target/release/deps/libsynom-5d0d6e9fb7bedce0.rmeta",
        "--extern",
        "unicode_xid=/home/darius/whist-rust/target/release/deps/libunicode_xid-528627904f457e26.rmeta",
        "--cap-lints",
        "allow"
      ],
      "env": {
        "CARGO": "/nix/store/akbsvnqdzj1rspmicv1j8qs8l7rhwbjv-rust/bin/cargo",
        "CARGO_MANIFEST_DIR": "/home/darius/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-0.11.11",
        "CARGO_PKG_AUTHORS": "David Tolnay <[email protected]>",
        "CARGO_PKG_DESCRIPTION": "Nom parser for Rust source code",
        "CARGO_PKG_HOMEPAGE": "",
        "CARGO_PKG_NAME": "syn",
        "CARGO_PKG_REPOSITORY": "https://github.com/dtolnay/syn",
        "CARGO_PKG_VERSION": "0.11.11",
        "CARGO_PKG_VERSION_MAJOR": "0",
        "CARGO_PKG_VERSION_MINOR": "11",
        "CARGO_PKG_VERSION_PATCH": "11",
        "CARGO_PKG_VERSION_PRE": "",
        "LD_LIBRARY_PATH": "/home/darius/whist-rust/target/release/deps:/nix/store/pk26s5n6myxda66splfnvbbg2kw0g40n-rust-1.42.0-nightly-2020-01-01-1ed41b072/lib:/run/opengl-driver/lib"
      },
      "cwd": "/home/darius/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-0.11.11"
    },

cc @mstone @copumpkin @P-E-Meunier

[P-E-Meunier]

Sorry for the late answer. The main issue with that solution (apart from working on Nightly), is features: currently, buildRustCrate translates crate features into Nix, which means that one can override features from Nix. Also, some features depend on the platform, and Nix can detect that automatically, which the build-plan cannot do.

[djahandarie]

@P-E-Meunier Thanks for the response; fair points. The reason I switched off of buildRustCrate some months ago was actually because of the nix expression evaluation time (on a very complex Rust project with lots of deps) was unbearable, possibly due to the overridability, which I didn't need. I still like the idea of using a flake here to build the relevant set of derivations and I'm curious how to make it flexible enough for people.