Use crates.io over git for dependencies

[djacu]

The presence of git dependencies can be problematic for build systems like Nix. progenitor as a git dependencies and progenitor-client as a registry causes the build to fail because it tries to vendor progenitor-client from two places: from crates.io as a direct dependency, and from GitHub as a transitive dependency of progenitor.

This commit changes the Cargo.toml and lock file to pull progenitor from the registry instead of GitHub.

Fixes #950

Tagging @sarcasticadmin as they testing the nix-packaged version of the CLI.

[djacu]

Seems like a few commits have caused a merge conflict since I forked earlier. I'll rebase and push.

[ahl]

Often the work in progenitor is in service of new features in the SDK and CLI; requiring us to publish a new version every time would be cumbersome for us.

Would another solution be for progenitor to have its progenitor-client dependency behind a feature flag? For our use case here (i.e. generation to files) the progenitor-macro and progenitor-client dependencies seem unnecessary. I'd happily accept a PR to progenitor. Indeed, we did something similar in typify: oxidecomputer/typify#213

[djacu]

Hey @ahl

Thanks for offering an alternative solution as was done for typify.

Unfortunately, that would not work in the case for nixpkgs. There are two ways nixpkgs can vendor the dependencies of a crate at build time. Both of these are methods that are reproducible which is a primary goal for nixpkgs.

1. If all dependencies are in a registry (e.g. crates.io) we specify a cargoHash which is a hash of all the dependencies. See this package definition of wl-clipboard-rs as an example.
2. If any git dependencies are present, the Cargo.lock file must be vendored in nixpkgs with the package definition (we call them derivations in Nix jargon). During the build process, Nix uses a specialized fetcher to fetch and vendor all the dependencies in the lock file before build time. During that process it creates symlinks in a sandboxed build environment of all the dependencies.

Because of the existence of git dependencies, nixpkgs must use the 2nd option. At the moment, I do not believe nix inspects feature flags when vendoring dependencies; all dependencies are vendored. The issue comes when symlinks are created for progenitor-client. Here is the entry for progenitor-client as a dependency of the workspace. Here it is as a transitive dependency of progenitor. Because they have the same name and semantic version, they will try to be symlinked to the same location, which ultimately causes the build to fail.

There is a mechanism which involves some Nix trickery and patching the source while vendoring the lock file. It might work but I have not tried this method before so I am not sure if it will succeed. I'll give it a try and let you know if it works out.

As an alternative or completely separate offer for improved automation, would you be interested in some tooling to help automate publishing in CI? I've been working on some tooling at my job that automates the process of publishing crates. I was going to use something similar for a personal rust project of my own and deploy it in the next couple weeks. It's a fairly simple in that it:

• searches the repository for all Cargo.toml files
• if it finds a name and version entry, it checks crates.io to see if that particular version is published
• if it doesn't find it that version on crates.io it publishes it
  I'd happily share it, if you are interested.

[ahl]

I decided to make the dependency optional regardless (#952). Does having a single progenitor-client in Cargo.lock help at all?

[djacu]

Possibly! I'd have to double check but my initial gut feeling is that it should work now.

I was able to get things building using the trickery I mentioned previously. See NixOS/nixpkgs#366077

If the changes you made in #952 do pan out, it should make packaging future versions simpler so thanks a bunch!

I'm happy to close out this PR and the related issue if you are.

[ahl]

Yes please!