Remove wildcard searching in redis scheduler #1408
Conversation
There was a problem hiding this comment.
Reviewable status: 0 of 1 LGTMs obtained, and 0 of 10 files reviewed, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Coverage, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, Web Platform Deployment / macos-14, Web Platform Deployment / ubuntu-24.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable (waiting on @adam-singer)
allada
force-pushed
the
redis-scheduler-remove-wild-card-search
branch
from
d0f5001 to
9352217
Compare
allada
force-pushed
the
redis-scheduler-remove-wild-card-search
branch
from
9352217 to
7609aec
Compare
There was a problem hiding this comment.
Reviewed 9 of 9 files at r2, all commit messages.
Reviewable status: 0 of 1 LGTMs obtained, and all files reviewed, and pending CI: Bazel Dev / macos-13, Bazel Dev / macos-14, Bazel Dev / ubuntu-24.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Coverage, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, NativeLink.com Cloud / Remote Cache / macos-14, NativeLink.com Cloud / Remote Cache / ubuntu-24.04, Publish image, Publish nativelink-worker-init, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (22.04), macos-13, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, windows-2022 / stable (waiting on @adam-singer)
Redis does not use a b-tree under the hood, instead it aggregates the key space, then does a union on all documents in the database. Thus, redis has a limit on how many items are unioned (MAXEXPANSIONS); by default redis has this set to 200. This limitation causes issues if there ever becomes more than 200 scheduled tasks, so we get around this by making the sort and filter index separate. Now we search explicitly for exact match state, then sort on a different key.
allada
force-pushed
the
redis-scheduler-remove-wild-card-search
branch
from
7609aec to
456e054
Compare
There was a problem hiding this comment.
Reviewed 1 of 2 files at r3.
Reviewable status: 0 of 1 LGTMs obtained, and 9 of 10 files reviewed, and pending CI: pre-commit-checks (waiting on @adam-singer)
There was a problem hiding this comment.
Reviewable status: 1 of 1 LGTMs obtained, and 9 of 10 files reviewed
There was a problem hiding this comment.
Reviewed 1 of 2 files at r3.
Reviewable status:complete! 1 of 1 LGTMs obtained, and all files reviewed
Redis does not use a b-tree under the hood, instead it aggregates the
key space, then does a union on all documents in the database. Thus,
redis has a limit on how many items are unioned (MAXEXPANSIONS); by
default redis has this set to 200. This limitation causes issues if
there ever becomes more than 200 scheduled tasks, so we get around this
by making the sort and filter index separate. Now we search explicitly
for exact match state, then sort on a different key.
This change is