Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

watch-deps watches (and hashes) all files in project #7

Open
casselc opened this issue Jul 7, 2022 · 1 comment
Open

watch-deps watches (and hashes) all files in project #7

casselc opened this issue Jul 7, 2022 · 1 comment
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@casselc
Copy link

casselc commented Jul 7, 2022

The default behaviors of https://github.com/nextjournal/beholder and https://github.com/gmethvin/directory-watcher cause watched folders to register watches on all subfolders recursively, and 'watching' a folder reads and hashes every file within it at startup and rehashes any created/modified files on every modification. Due to this, the DirectoryWatcher created by watch-deps is reading and hashing everything under the project folder recursively (including .git, node_modules, etc) and rehashing all modified files even though the callback registered by watch-deps ultimately ignores everything but modifications to deps.edn. This can cause a long delay when watch-deps is invoked for projects with large numbers of files or just large files (e.g. data files under resources/) due the hashing on startup, and unnecessary resource usage caused by rehashing every git commit, asset file update, etc.

DirectoryWatcher can be configured not to hash or to use the file modified timestamp instead of hashing, but these options are not exposed by beholder. It doesn't appear to be possible to watch a single folder without its children using DirectoryWatcher

Would a PR that uses the native java.nio.file.WatchService directly instead of beholder/DirectoryWatcher be welcome? It appears the downside would be that WatchService uses a polling implementation on MacOS, but since it would be possible to watch the project root without any children, I would think it would still be a net win for performance.

@alysbrooks alysbrooks added enhancement New feature or request help wanted Extra attention is needed labels Jul 8, 2023
@alysbrooks
Copy link
Member

I'm inclined to say we should first try to keep hashing and just try to avoid hashing what we don't need to. It looks like DirectoryWatcher allows for setting a custom FileTreeVisitor, which I believe could do that filtering. But if that doesn't end up being very straightforward, dropping hashing is something to consider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants