Add support for the Nix package manager #172

coolGi69 · 2024-06-11T07:35:42Z

Similar to #150 except updated to the latest version of libmacchina. (closes #120)
Unlike that commit which attempts to estimate the total amount of nix pkgs by counting the files in /nix/store, this uses nix's sqlite database (/nix/var/nix/db/db.sqlite).
The sqlite library has also been updated (0.31.1 -> 0.36.0) due to it having the ability to read the path as a uri (so that I can tell sqlite that the database is immutable).

This queries the database using SELECT COUNT(path) FROM ValidPaths WHERE ultimate=1 (which is equivalent to nix path-info --all --sigs | grep ultimate | wc -l) which gives a time of around 10ms, while the equivalent nix command gives around 40ms.

This lookup is grabbing a slightly different metric which gives 739 on my system (nix store giving 1306). So I think nix store is including extra things that don't count as packages (as the /nix/store folder stores everything from fetches, sources, man pages, binaries, etc).

(all packages built with cargo build --release, and tested with hyperfine -w 4 -m 500 -N)

Type	Speed
Original Macchina	5.9 ms ± 0.2 ms
SQLite Lookup	14.2 ms ± 1.7 ms

Gobidev · 2024-06-11T10:54:15Z

This looks very interesting. From my experience, invoking nix-store was significantly slower than every other readout of libmacchina. I am currently using pretty much the same method as in this PR in pfetch-rs, but added an extra option to skip it because it is comparatively very slow. Maybe a recent Nix update made this method faster, I will do some testing myself later.

Just out of curiosity, what is your macchina runtime if you skip Nix package count?

P.S. What are the runtime differences when you compile macchina with optimizations (cargo b --release)?

Gobidev · 2024-06-11T11:37:19Z

I have done a bit of testing in a NixOS Live ISO environment with about 1100 packages installed, and this method is still very slow, with the package count taking about 200ms. pfetch-rs currently has a total runtime below 10ms in the same environment when skipping the package count, so this is still a big hit to the runtime. This also only gets worse with more packages installed and on older hardware.

So as much as I would like Nix support to be added, this method just is not fast enough for libmacchina in my opinion.

If you are interested, you could try to get the package count in other ways, as I tried in #150, but this probably requires deeper knowledge of the nix source code.

I also had a look at what fastfetch does, they seem to use the command invocation and cache the result for future queries, which is something we could also think about for libmacchina.

coolGi69 · 2024-06-11T12:24:56Z

I think that command is the standard for getting the amount of packages. It's also whats used in fastfetch (which is which is what I compared it to), as well as most stack overflow answers.

(all packages built with cargo build --release, and tested with hyperfine -w 4 -m 500 -N)

Type	Speed
Default	2.2 ms ± 0.4 ms
This pull	116.5 ms ± 22.1 ms
This pull, but removing nix-user (as pr 150 only gives one value)	50.3 ms ± 9.6 ms
This pull, but replacing the method with the one in pr 150	28.9 ms ± 4.8 ms

I think my previous test were wrong (I'm not even sure how I got the original results). These numbers seem to be more realistic.

In reply to your latest comment, as well as the more realistic results, it is probably best to cache the result somewhere. I'll look into it later tomorrow to see if that's the best way to go.

coolGi69 · 2024-06-11T12:31:41Z

(I just changed it to a draft until I find a faster, but still reliable method)

grtcdr · 2024-06-11T20:47:57Z

Hi, thanks for the contribution. I personally want to see more progress in #150 due to the performance penalty of dishing out to a command-line program.

To respond to @Gobidev on the caching proposal, I believe it shouldn't be libmacchina's responsibility to cache things. I'd rather keep the library as stateless as possible (despite the obvious oxymoron), downstream programs should implement this if they so wish.

coolGi69 · 2024-06-14T14:36:57Z

Quick update:
Nix stores all its data for stuff in /nix/var/nix/db/db.sqlite (defined in the libstore/local-store)
In it, it contains several tables, but the one we are interested in is the DerivationOutputs table, and its id column. The quickest way is to just count the amount of items with the id of dist giving an extremely close approximation of the value gotten by the original nix-store command (nix-store giving 1306, and the amount of dists 1033) dist is not the correct thing to use.
I think I'll just need to look into the code a bit more to see where the extra 300 packages come from so it can be more accurate.

Hopefully this method works and can give good speeds! (well, as good as you'll get with an sqlite database)

Gobidev · 2024-06-15T13:34:59Z

Nice, that sounds really promising, thank you for putting in the work! I will close #150 in favor of this PR.

coolGi69 · 2024-06-16T08:07:36Z

I think the best sqlite query is SELECT COUNT(path) FROM ValidPaths WHERE ultimate=1 (which is equivalent to nix path-info --all --sigs | grep ultimate | wc -l) which gives a time of around 10ms, while that nix command gives around 40ms.

Here are the results of hyperfine:

Type	Speed
Original Macchina	5.9 ms ± 0.2 ms
SQLite Lookup	14.2 ms ± 1.7 ms

This lookup is grabbing a slightly different metric which gives 739 on my system (nix store giving 1306). So I think nix store is including extra things that don't count as packages (as the /nix/store folder stores everything from fetches, sources, man pages, binaries, etc).

Also, I've updated sqlite from 0.31.1 to 0.36.0.

src/linux/mod.rs

grtcdr · 2024-06-18T19:31:31Z

Big thanks for the contribution (TIL about breaking out of loops with labels, thanks for that too!).

Gobidev · 2024-06-18T20:35:47Z

Thanks again for your work. I didn't have much time to test this, yet, but at least in a NixOS live CD environment, it doesn't really seem to work right now. Your method returns 3 installed packages, while nix-store -q --requisites /run/current-system/sw | wc -l returns 1098 and nix-store -q --requisites ~/.nix-profile | wc -l returns 15.

This lookup is grabbing a slightly different metric which gives 739 on my system (nix store giving 1306). So I think nix store is including extra things that don't count as packages (as the /nix/store folder stores everything from fetches, sources, man pages, binaries, etc).

If the lower count is more accurate than the one returned by nix-store, this is of course fine, though I expect that there will be issues reported because of inconsistent package counts across different fetch tools.

I will check your method on a system with NixOS actually installed and on a different distro with Nix installed to see if that gives different results, maybe there is just something weird about the NixOS live CD.

coolGi69 · 2024-06-19T06:12:08Z

It only lists the main installed packages. I've uploaded a video of me adding cargo (to a Mint system that I freshly installed Nix onto), and despite it having several dependencies, it only recognizes it as a single one.
The result is the same on NixOS, but I didn't have my system with it installed on hand to record.

example_of_method_text.mp4

(I haven't tested it on the NixOS live env, but my guess is that it only shows 3 as it is registering basically everything under a single meta package)

coolGi69 · 2024-06-19T07:29:51Z

The other option for an sqlite search is doing SELECT COUNT(DISTINCT path) FROM DerivationOutputs WHERE id="out", but that lists everything and I just prefer the current lookup format.

Gobidev · 2024-06-19T11:52:11Z

The other option for an sqlite search is doing SELECT COUNT(DISTINCT path) FROM DerivationOutputs WHERE id="out", but that lists everything

How does that count compare to the other methods?

coolGi69 · 2024-06-19T13:40:22Z

Wrote my thought process, as well as some notes. Hopefully you'd agree that the current method in the pr is the best one (well, at least that I could find):

Type	Speed	Packages	Note
Original	2.9 ms ± 0.1 ms	N/A
`SELECT COUNT(path) FROM ValidPaths WHERE ultimate=1` (this pr)	5.3 ms ± 0.9 ms	832	Remember, this only includes the packages in the available generations. So stuff from `nix-shell -p` wont count towards this number (older generations that haven't been deleted are included here, remember this as it'll come up at the end)
`SELECT COUNT(DISTINCT path) FROM DerivationOutputs WHERE id="out"`	15.1 ms ± 3.0 ms	8521	This includes every output (so that includes generations, patches, and different entries for the tar.gz vs extracted stuff, so on and so on. Check it yourself if you wanna see just how many other things are included)

If I remove the DISTINCT to make the other lookup go faster gets quicker times, but the amount of packages also go way up.

Type	Speed	Packages	Note
`SELECT COUNT(path) FROM DerivationOutputs WHERE id="out"`	6.4 ms ± 0.3 ms	16333	Despite this being nearly on-par with the current method (maybe the same as its within a small enough error), this also includes duplicates making the number nearly double!

nix-store -q --requisites /run/current-system/sw | wc -l: 1306
nix-store -q --requisites ~/.nix-profile | wc -l: 1171
(note, the packages in current system and nix profile will have overlap)

Getting the overlap by doing the following:

nix-store -q --requisites /run/current-system/sw > /tmp/pkgs_system.txt
nix-store -q --requisites ~/.nix-profile > /tmp/pkgs_profile.txt
awk 'FNR==NR {a[$0]++; next} !($0 in a)' /tmp/pkgs_profile.txt /tmp/pkgs_system.txt | wc -l # Gets the count of different lines between the 2 files

(on my system being 651)
Then subtracting that from the sum gives 1826 on my system. (again, remember that this number includes more than just packages, and only counts it for the current running generation (so nix-shell -p will still be disregarded, as well as older generations))

Gobidev · 2024-06-19T18:58:42Z

Thanks a lot for the extensive comparison :)

Based on your numbers, your current method seems to be the one to go with. If there was a way to include dependencies in the count, that would be preferable, but that might not be possible with this approach.

I tested your method on a stock NixOS system, and it returns 313 installed packages, which is a believable number. I will test some more scenarios in which Nix might be used tomorrow.

Thanks again for the amount of work you put into this, it is nice to finally see Nix support being added to libmacchina^^

Gobidev · 2024-06-20T09:13:55Z

I tested this PR on a single- and multi-user Nix installation on Arch Linux and on both it returns the package count without dependencies as expected.

@grtcdr what is your opinion on the dependency count not being included? AFAIK it is included for all other package managers, but the count without it is not inherently wrong.

grtcdr · 2024-06-20T21:58:00Z

I'd include the dependencies to avoid the inevitable, unwanted reports we'd undoubtedly receive from users. I'm familiar with the basic concepts of Nix, but the intricacies of it escape me, so if the majority agrees that including just the base packages is a better index then I'm all for it.

coolGi69 · 2024-06-20T22:51:03Z

How about something like SELECT COUNT(path) FROM ValidPaths WHERE sigs IS NOT NULL
It seems to give the correct results with dependacies.

On a fresh install of nix on another distro, it outputs 0. Then once I install cargo (which has 63 dependacies, that it says at the top of the output when its downloading), it outputs 64 as expected.
I think this is what we've been looking for!

Sorry, not sure how I didnt notice that before. I guess doing my tests on a non-nix system is easier to read as there isn't soo many packages installed.

coolGi69 · 2024-06-20T23:03:13Z

Yeah, this is wayy better!
Doing the same thing to get the amount of packages on both the current generations (printing the output of the nix-store thingie and getting the amount of non-duplicates) gives 1826. Then with this new method it gives 1870.

This slightly larger number is understandable as the new sqlite lookup includes every generation (which won't add any on my system due to me cleaning old generations recently), as well as packages on your system but not on the current generation (ie from nix-shell)

Gobidev · 2024-06-21T07:36:47Z

I have tested this new method and found the following:

On an Arch installation with Nix installed in multi-user mode, I get 67 installed packages after installing cargo and pfetch-rs with nix, which looks good.

On an Arch installation with Nix installed in single-user mode, I get 28 installed packages after installing wget and cargo, but it stays at 28 after uninstalling wget.

On a NixOS installation, i get 331 installed packages (it was 314 with the previous method), while nix-store -q --requisites /run/current-system/sw | wc -l alone returns 929.

This method is still not perfect but imo the best we've had so far.

coolGi69 · 2024-06-21T08:27:17Z

Remember to clear your cache ((sudo) nix-collect-garbage -d). Nix never removes stuff unless you tell it to, hence why it is still counting it even if you "uninstall" it.

grtcdr · 2024-06-21T20:19:58Z

I'm happy with what we've got - thank you so much for the quality work - shall we get this merged?

coolGi69 · 2024-06-21T22:45:28Z

Yeah, sounds good!

* Fix Armv7 build (#158) * Add armv7 build target * Fix mismatched types for pointer width 32 * Use `sh` instead of `python` for `which` assertion * Replace mach with mach2 (#157) * feat: add support for RPM `ndb` databases (#159) * feat: add support for RPM `ndb` databases * chore: always try sqlite before librpm * chore: remove `rpm` feature * fix: librpm call not lazily evaluated * Add `librpm` dependency note to README (#160) * Update the changelog * Bump version to 7.1.0 * Fix i686 build (#162) closes #161 * Add Sonoma to macos_version_to_name (#163) * BREAKING CHANGE: Change BatteryReadout::health return value to u8 There's no particular reason why one should allocate 64 bits for a value that can only be <= 100. As a bonus, ceil() the return value before finally casting to u8. * Update BatteryReadout::health function signature for other operating systems besides Linux. * Added general detection for wayland compositors (#164) * Upgrade dependencies to their latest versions * Update changelog Make some formatting changes as well. * Add missing generic argument sqlite's changed a bit between releases. * Bump version to 7.2.0 * Add missing second generic argument to sqlite read() call * Remove unneeded argument to unistd::gethostname function call * Remove unused variable * Refactor obsolete find_ifa function * Bump version to 7.2.1 * Bump vergen version * Use gitcl feature of vergen This depends on the git binary, more ubiquituous than the libgit2 bindings, so it should technically work on every platform we support. * Refactor the old vergen interface * Add new entry to the changelog * Update version to 7.2.2 * Replace flatten() calls with map_while(Result::ok) * Fix Readouts struct's network field type Closes: #168 * Improve CI workflow (#169) * Replace discontinued actions-rs * Split cargo fmt and clippy into their own CI job * Faster package count on Alpine Linux (#170) * Bump version to 7.2.3 * added macos 15 version name (#171) https://www.apple.com/newsroom/2024/06/macos-sequoia-takes-productivity-and-intelligence-on-mac-to-new-heights/ * Removed panic if local gpu db is not able to be read (#173) * Add support for the Nix package manager (#172) Added support for the Nix package manager using Nix' SQLite database. * Bump version and update changelog * linux: Safely exit when homebrew is not installed * Improve linuxbrew keepme safeguard * Remove unused import * Bump version to 7.3.1 * Allow disk_space function to accept a path argument (#156) BREAKING CHANGE: allow disk_space function to accept a path argument - Bump version and update changelog - Change disk_space path argument to be of type &Path and check path exists in shared::disk_space - Add missing import for openwrt --------- Co-authored-by: Adrian Groh <[email protected]> Co-authored-by: Silas Groh <[email protected]> Co-authored-by: grtcdr <[email protected]> Co-authored-by: Rex Ng <[email protected]> Co-authored-by: Absolpega <[email protected]> Co-authored-by: grtcdr <[email protected]> Co-authored-by: Matthias Baer <[email protected]> Co-authored-by: Rex Ng <[email protected]> Co-authored-by: coolGi <[email protected]>

Added support for the Nix package manager

ce9a7d7

coolGi69 changed the title ~~Added support for the Nix package manager~~ Add support for the Nix package manager Jun 11, 2024

coolGi69 marked this pull request as draft June 11, 2024 12:31

Gobidev mentioned this pull request Jun 15, 2024

Add nix package count #150

Closed

Using nix's sqlite for getting amount of packages

0f61333

coolGi69 marked this pull request as ready for review June 16, 2024 08:07

Slight fixup to the comments

4e37bd4

grtcdr reviewed Jun 17, 2024

View reviewed changes

src/linux/mod.rs Outdated Show resolved Hide resolved

coolGi69 added 3 commits June 18, 2024 06:16

Removed redundant comment

b3ea2d4

Removed panic if gpu db is not able to be read

3f2c16d

--ammend

e635da4

Added missing comma (causing ci lint to fail)

cd228e6

coolGi69 and others added 2 commits June 19, 2024 15:43

Merge branch 'Macchina-CLI:main' into main

3f61812

Hopefully fixed linting?

c4ae567

Gobidev approved these changes Jun 20, 2024

View reviewed changes

Changed sqlite lookup on nix to a better one

e6cdaf2

coolGi69 added 2 commits June 21, 2024 08:49

The linter wanted this to happen, so here you go

0f77bb7

Removed a now useless comment

1a56a0a

grtcdr approved these changes Jun 22, 2024

View reviewed changes

grtcdr merged commit 3a5bec0 into Macchina-CLI:main Jun 22, 2024
9 checks passed

grtcdr mentioned this pull request Jul 13, 2024

NixOS Packages not listed #176

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for the Nix package manager #172

Add support for the Nix package manager #172

coolGi69 commented Jun 11, 2024 •

edited

Loading

Gobidev commented Jun 11, 2024 •

edited

Loading

Gobidev commented Jun 11, 2024

coolGi69 commented Jun 11, 2024 •

edited

Loading

coolGi69 commented Jun 11, 2024

grtcdr commented Jun 11, 2024

coolGi69 commented Jun 14, 2024 •

edited

Loading

Gobidev commented Jun 15, 2024

coolGi69 commented Jun 16, 2024 •

edited

Loading

grtcdr commented Jun 18, 2024

Gobidev commented Jun 18, 2024

coolGi69 commented Jun 19, 2024

coolGi69 commented Jun 19, 2024

Gobidev commented Jun 19, 2024

coolGi69 commented Jun 19, 2024 •

edited

Loading

Gobidev commented Jun 19, 2024

Gobidev commented Jun 20, 2024 •

edited

Loading

grtcdr commented Jun 20, 2024

coolGi69 commented Jun 20, 2024 •

edited

Loading

coolGi69 commented Jun 20, 2024

Gobidev commented Jun 21, 2024

coolGi69 commented Jun 21, 2024 •

edited

Loading

grtcdr commented Jun 21, 2024

coolGi69 commented Jun 21, 2024

Add support for the Nix package manager #172

Add support for the Nix package manager #172

Conversation

coolGi69 commented Jun 11, 2024 • edited Loading

Gobidev commented Jun 11, 2024 • edited Loading

Gobidev commented Jun 11, 2024

coolGi69 commented Jun 11, 2024 • edited Loading

coolGi69 commented Jun 11, 2024

grtcdr commented Jun 11, 2024

coolGi69 commented Jun 14, 2024 • edited Loading

Gobidev commented Jun 15, 2024

coolGi69 commented Jun 16, 2024 • edited Loading

grtcdr commented Jun 18, 2024

Gobidev commented Jun 18, 2024

coolGi69 commented Jun 19, 2024

coolGi69 commented Jun 19, 2024

Gobidev commented Jun 19, 2024

coolGi69 commented Jun 19, 2024 • edited Loading

Gobidev commented Jun 19, 2024

Gobidev commented Jun 20, 2024 • edited Loading

grtcdr commented Jun 20, 2024

coolGi69 commented Jun 20, 2024 • edited Loading

coolGi69 commented Jun 20, 2024

Gobidev commented Jun 21, 2024

coolGi69 commented Jun 21, 2024 • edited Loading

grtcdr commented Jun 21, 2024

coolGi69 commented Jun 21, 2024

coolGi69 commented Jun 11, 2024 •

edited

Loading

Gobidev commented Jun 11, 2024 •

edited

Loading

coolGi69 commented Jun 11, 2024 •

edited

Loading

coolGi69 commented Jun 14, 2024 •

edited

Loading

coolGi69 commented Jun 16, 2024 •

edited

Loading

coolGi69 commented Jun 19, 2024 •

edited

Loading

Gobidev commented Jun 20, 2024 •

edited

Loading

coolGi69 commented Jun 20, 2024 •

edited

Loading

coolGi69 commented Jun 21, 2024 •

edited

Loading