feat: Add support for using watchman(1) instead of walking filesystem #263

KAction · 2023-12-17T18:43:36Z

Anecodically, for project having 4k files and 7 different formatters, this change can reduce time it takes for treefmt(1) to figure out that it has nothing to do from ~350ms down to ~55ms, out of which getting response from watchman daemon takes ~35ms, so there is room for further improvements.

If "WATCHMAN_SOCK" environment variable is not set and "watchman(1)" command are not available, "treefmt" transparently falls back on walking the file system. Setting the environment variable is highly recommended, since is avoids subprocess invocation.

Ref: #261

Regarding further optimizations, this is what I think can be improved::

[DEBUG] load cache: 8.36ms (Δ 6.85ms)
...
[DEBUG] watchman query: 43.73ms (Δ 35.02ms)
[DEBUG] post-process watchman list: 50.55ms (Δ 6.82ms)
...

* Build docs in nix * Build docs in CI and update gh-pages * Format nix

Avoid the big fat string literal

Instead of calling CLOG directly, use the "log" crate on the producer side, and then plug CLOG as a consumer. This allows for some nice decoupling and the log macros are also nicer on the eye. --log-level has disappeared for now

* nix: simplify the code a bit * docs: fix the website Make the paths relative so they work on GitHub Pages. Fix the landing page overriding the docs. Put the docs in the right sub-folder.

* customlog: convert emoji to unicode escape * engine: print reformatted files on failure

* implement --stdin A first pass at making --stdin work * use the tempfile crate

In cases where the formatter path is relative, the "which" crate doesn't remove the "./" in between.

* treat exit status code > 0 as an error * adding success status check before matching status code

) Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from v12 to v13. - [Release notes](https://github.com/cachix/install-nix-action/releases) - [Commits](cachix/install-nix-action@v12...8d6d5e9) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [cachix/cachix-action](https://github.com/cachix/cachix-action) from v8 to v9. - [Release notes](https://github.com/cachix/cachix-action/releases) - [Commits](cachix/cachix-action@v8...2689c27) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Handling output formatter

Broken by numtide#101

Flake lock file updates: • Updated input 'flake-parts': 'github:hercules-ci/flake-parts/8d0e2444ab05f79df93b70e5e497f8c708eb6b9b' (2022-12-07) → 'github:hercules-ci/flake-parts/c9afaba3dfa4085dbd2ccb38dfade5141e33d9d4' (2023-10-03) • Updated input 'mkdocs-numtide': 'github:numtide/mkdocs-numtide/af6c4a5f7c0a59da3b557795f57dcae5707523ac' (2023-04-01) → 'github:numtide/mkdocs-numtide/b3008171c75083f2bf2f1dc4e6781d4737dfaa49' (2023-06-26) • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/4172cdda7e56a48065475fb98c57b03b83c1fde4' (2022-12-16) → 'github:NixOS/nixpkgs/01441e14af5e29c9d27ace398e6dd0b293e25a54' (2023-10-11) • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/7da2f6b3a0c32f661cb2864d7fbd1d7e6f0c7543' (2022-12-16) → 'github:oxalica/rust-overlay/dce60ca7fca201014868c08a612edb73a998310f' (2023-10-14) • Updated input 'rust-overlay/flake-utils': 'github:numtide/flake-utils/c0e246b9b83f637f4681389ecabcb2681b4f3af0' (2022-08-07) → 'github:numtide/flake-utils/cfacdce06f30d2b68473a46042957675eebb3401' (2023-04-11) • Added input 'rust-overlay/flake-utils/systems': 'github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e' (2023-04-09) • Updated input 'systems': 'github:nix-systems/default/4e9a51a15ceb27e5141819142a7d2ee827943fc8' (2023-04-08) → 'github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e' (2023-04-09)

Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]>

Add the `hidden` and `no-hidden` flag that allows configuring the traversal of hidden files. Examples: ``` treefmt --hidden ``` will traverse hidden files. ``` treefmt --hidden --no-hidden ``` will not traverse hidden files, same as the default (`treefmt`). ``` treefmt --hidden --no-hidden --hidden ``` will traverse hidden files.

Flake lock file updates: • Updated input 'rust-overlay': 'github:oxalica/rust-overlay/dce60ca7fca201014868c08a612edb73a998310f' (2023-10-14) → 'github:oxalica/rust-overlay/e494404d36a41247987eeb1bfc2f1ca903e97764' (2023-10-15) Co-authored-by: mic92-buildbot <[email protected]>

Bumps [cachix/install-nix-action](https://github.com/cachix/install-nix-action) from 23 to 24. - [Release notes](https://github.com/cachix/install-nix-action/releases) - [Commits](cachix/install-nix-action@v23...v24) --- updated-dependencies: - dependency-name: cachix/install-nix-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [rustix](https://github.com/bytecodealliance/rustix) from 0.37.23 to 0.37.27. - [Release notes](https://github.com/bytecodealliance/rustix/releases) - [Commits](bytecodealliance/rustix@v0.37.23...v0.37.27) --- updated-dependencies: - dependency-name: rustix dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>

Bumps [cachix/cachix-action](https://github.com/cachix/cachix-action) from 12 to 13. - [Release notes](https://github.com/cachix/cachix-action/releases) - [Commits](cachix/cachix-action@v12...v13) --- updated-dependencies: - dependency-name: cachix/cachix-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

KAction · 2023-12-19T14:06:25Z

src/engine.rs

+// first place to discover a changed file that matches "includes" pattern.
+//
+// FIXME: There should be a cleaner solution.
+fn path_excluded(gs: &GlobSet, path: &PathBuf) -> bool {


This one is really inefficient. For nixpkgs, I get following timings:

[DEBUG] watchman query: 404.46ms (Δ 120.62ms) [DEBUG] post-process watchman list: 601.97ms (Δ 197.52ms)

KAction · 2023-12-19T14:14:21Z

Also, for huge repositories, writing and loading cache is slow. For nixpkgs,

load cache: 290.48ms (Δ 290.28ms)
write cache: 702.86ms (Δ 29.27ms)

I think I should make separate cache for watchman-enabled scans that keeps only since parameter.

KAction · 2023-12-19T14:15:53Z

As for excludes, I think I can at least optimize happy path and change excludes: Option<GlobSet>. Or mangle excludes pattern to add trailing /*.

KAction · 2023-12-22T20:17:12Z

Updated the pull request. Previous version misused watchman, causing it to return way too much results.
With this version, whole nixpkgs is down to 12ms for no-op, my $dayjob repository is down to 8.5ms.

KAction · 2023-12-30T01:50:52Z

error (ignored): error: SQLite database '/home/garnix/.cache/nix/eval-cache-v5/6de50559647aa37e20a3fd649274afe85c9def927c4d55696d8a84953e4ccaa3.sqlite' is busy

Should I be concerned?

zimbatm

hey, I finally had the time to take a closer look and actually run it, and it's great. The only thing missing is to fix the default user experience if watchman is not installed, and a bit of documentation on how to use that feature.

zimbatm · 2024-03-13T11:17:49Z

src/command/format.rs

+            warn!(
+                "watchman is not available (err = {:?}), falling back on stat(2)",
+                e
+            );


Most users should be able to use treefmt without watchman, so having this as a warning isn't the best user experience for them.

I propose three modes: "auto" (default) tries watchman and silently falls back on "stat," "stat" is stat, and "watchman" errors if watchman is not available.

Then, introduce a CLI flag and/or project-level global config to allow switching between modes.

zimbatm · 2024-03-13T11:18:22Z

src/command/format.rs

+        }
+        Ok(c) => {
+            // This is important. Subprocess wastes ~20ms.
+            if !std::env::var("WATCHMAN_SOCK").is_ok() {


this needs to be documented, to explain how to set it

zimbatm · 2024-03-13T11:27:20Z

src/formatter.rs

@@ -191,7 +201,10 @@ impl Formatter {
            options: cfg.options.clone(),
            work_dir,
            includes,
+            // TODO: Avoid clone().


the list is generally super small, I don't think the TODO needs to stay.

Anecodically, for project having 4k files and 7 different formatters, this change can reduce time it takes for treefmt(1) to figure out that it has nothing to do from ~350ms down to ~8m. If "WATCHMAN_SOCK" environment variable is not set and "watchman(1)" command are not available, "treefmt" transparently falls back on walking the file system. Setting the environment variable is highly recommended, since is avoids subprocess invocation. Ref: numtide#261

Writing list of all files takes non-trivial amount of time

KAction · 2024-04-05T13:52:18Z

I think I addressed all suggestions it latest revision.

zimbatm and others added 30 commits March 2, 2021 17:36

ci: test on macOS (numtide#85)

b202af5

docs: add more compatibility notes

c787831

fix clippy::pedantic warnings

b7ba437

Publish the docbook to GitHub Pages (numtide#89)

0ecbb2c

* Build docs in nix * Build docs in CI and update gh-pages * Format nix

Add clang-format and stylish-haskell configs to the docs (numtide#88)

405d83a

init: use std::include_bytes (numtide#91)

6b2d6fc

Avoid the big fat string literal

customlog: use the log facade (numtide#90)

e7ccb2d

Instead of calling CLOG directly, use the "log" crate on the producer side, and then plug CLOG as a consumer. This allows for some nice decoupling and the log macros are also nicer on the eye. --log-level has disappeared for now

remove dead code

5a4516f

update the eval cache by mutating it rather than cloning it (numtide#87)

d439941

fix docs (numtide#94)

3dc5943

* nix: simplify the code a bit * docs: fix the website Make the paths relative so they work on GitHub Pages. Fix the landing page overriding the docs. Put the docs in the right sub-folder.

print files on --fail-on-change (numtide#96)

babc297

* customlog: convert emoji to unicode escape * engine: print reformatted files on failure

README: fix link to formatter spec

a6b43be

implement --stdin (numtide#95)

8de1add

* implement --stdin A first pass at making --stdin work * use the tempfile crate

formatter: clean the formatter path

cfa4024

In cases where the formatter path is relative, the "which" crate doesn't remove the "./" in between.

README: fix typo

8353f13

README: add super-linter to the related projects

751e598

treat exit status code > 0 as an error

ffba2a7

adding success status check before matching status code

73f5f25

treat exit status code > 0 as an error (numtide#98)

0df89fb

* treat exit status code > 0 as an error * adding success status check before matching status code

add Result when collecting formatter result

c3512d1

create match on formatter output

d3ade34

move error handling to fmt function

ae7dc6a

Merge branch 'master' into output-formatter

fb8330c

Merge pull request numtide#101 from numtide/output-formatter

b113d53

Handling output formatter

engine: fix elapsed reporting

26da756

Broken by numtide#101

Release v0.1.0

05f612b

Report formatter output on error (numtide#104)

63c58e4

fix clippy warnings

4edd9dc

Mic92 and others added 14 commits October 14, 2023 12:07

restore default loglevel to INFO

bf5ae09

flake.nix: add checks for packages and devShells

5a4b535

upgrade to new flake-parts syntax

42676ea

work around terraform beeing marked as unfree

de751ab

fix formatting of index.html

90e98e0

Release v0.6.1

216e78f

fixup! Release v0.6.1

5943bab

KAction commented Dec 19, 2023

View reviewed changes

KAction force-pushed the contrib/0/watchman/out branch from c6b74b5 to b5050cf Compare December 22, 2023 20:12

zimbatm force-pushed the contrib/0/watchman/out branch from b5050cf to e6b747d Compare March 13, 2024 10:48

zimbatm reviewed Mar 13, 2024

View reviewed changes

Dmitry Bogatov added 2 commits March 22, 2024 15:20

Use separate cache for watchman-enabled scans

34ddc73

Writing list of all files takes non-trivial amount of time

KAction force-pushed the contrib/0/watchman/out branch from e6b747d to b9555e9 Compare March 22, 2024 19:21

feat: Add command line option to pick filesystem scan method

c5f8a2a

KAction force-pushed the contrib/0/watchman/out branch from b9555e9 to c5f8a2a Compare March 22, 2024 19:54

brianmcgee closed this May 13, 2024

brianmcgee force-pushed the main branch from 0a285d3 to 4a86924 Compare May 13, 2024 08:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add support for using watchman(1) instead of walking filesystem #263

feat: Add support for using watchman(1) instead of walking filesystem #263

KAction commented Dec 17, 2023

KAction Dec 19, 2023

KAction commented Dec 19, 2023

KAction commented Dec 19, 2023 •

edited

Loading

KAction commented Dec 22, 2023 •

edited

Loading

KAction commented Dec 30, 2023 •

edited

Loading

zimbatm left a comment

zimbatm Mar 13, 2024

zimbatm Mar 13, 2024

zimbatm Mar 13, 2024

KAction commented Apr 5, 2024

feat: Add support for using watchman(1) instead of walking filesystem #263

feat: Add support for using watchman(1) instead of walking filesystem #263

Conversation

KAction commented Dec 17, 2023

KAction Dec 19, 2023

Choose a reason for hiding this comment

KAction commented Dec 19, 2023

KAction commented Dec 19, 2023 • edited Loading

KAction commented Dec 22, 2023 • edited Loading

KAction commented Dec 30, 2023 • edited Loading

zimbatm left a comment

Choose a reason for hiding this comment

zimbatm Mar 13, 2024

Choose a reason for hiding this comment

zimbatm Mar 13, 2024

Choose a reason for hiding this comment

zimbatm Mar 13, 2024

Choose a reason for hiding this comment

KAction commented Apr 5, 2024

KAction commented Dec 19, 2023 •

edited

Loading

KAction commented Dec 22, 2023 •

edited

Loading

KAction commented Dec 30, 2023 •

edited

Loading