Tracking: faster cluster troubleshooting and recovery

[kwannoel]

• Add a fast docker build buildkite workflow #18097
• Easily identify stream graph bottleneck #13481. This is also slow, because we have to manually read the await-tree and find the bottleneck.

[wjf3121]

For 1 (Speed up docker build), we may consider moving to a hermetic build system with remove cache support like Bazel in CI to speed up the build (AFAIK cargo itself doesn't have good support for this). Usually for a patch build, only a small portion of the code is changed so the majority of the build can make use of cache. A rough guess is we can reduce the build time to within 5 mins with remote cache for most patch build.

There are tools to auto generate bazel build files from cargo: https://bazelbuild.github.io/rules_rust/crate_universe.html

This could also improve developer productivity as PR checks & developer image building can be sped up.

cc @lmatz @huangjw806 @cyliu0

[kwannoel]

For 1 (Speed up docker build), we may consider moving to a hermetic build system with remove cache support like Bazel in CI to speed up the build (AFAIK cargo itself doesn't have good support for this). Usually for a patch build, only a small portion of the code is changed so the majority of the build can make use of cache. A rough guess is we can reduce the build time to within 5 mins with remote cache for most patch build.

There are tools to auto generate bazel build files from cargo: https://bazelbuild.github.io/rules_rust/crate_universe.html

This could also improve developer productivity as PR checks & developer image building can be sped up.

cc @lmatz @huangjw806 @cyliu0

May require a non-trivial amount of work. We use some non-standard things like crate patching for madsim.
Hermetic build systems are also more work to maintain. Will require some investigation.

Another idea shared is to simply add a new buildkite workflow which will use larger machine sizes. @huangjw806 can comment on that, do you know if this work is already in progress?

[BugenZhao]

Usually for a patch build, only a small portion of the code is changed so the majority of the build can make use of cache. A rough guess is we can reduce the build time to within 5 mins with remote cache for most patch build.

IIRC, a significant contributory factor of the compile time (up to ~8 min) is the final step of linking (specifically LTO), which can barely be speeded up with caching. Though we currently have no idea whether LTO improves performance much in production.

Also, I believe @xxchan is highly experienced in this area. Kindly request for insights. 🙏

[xxchan]

previous attempt to speed up docker build: #12193

But

risingwave/docker/Dockerfile

Line 37 in f54c7fe

# TODO: cargo-chef doesn't work well now, because we update Cargo.lock very often.

[huangjw806]

Speed up docker build.

The docker build pipeline needs to build x86 and arm images. Our pipeline jobs run concurrently, so the image upload time depends on the slowest job. Previously, the arm job was much slower than the x86 job, but now it is basically the same after I upgraded the arm instance type.

In addition, since the cloud environment now runs on arm machines, if we need an urgent patch, maybe we only need to build the arm image.

[kwannoel]

IIRC, a significant contributory factor of the compile time (up to ~8 min) is the final step of linking (specifically LTO), which can barely be speeded up with caching. Though we currently have no idea whether LTO improves performance much in production.

Sounds good to me to remove lto for patch builds.

[xxchan]

use larger machine sizes

This also sounds worth trying to me. We can try to benchmark building images on different instance specs.

[huangjw806]

use larger machine sizes

Maybe we can use a larger arm instance type to do arm docker build, but it is difficult to upgrade the instance type of x86, so we need to innovate and create an aws cloudfromation stack. This is because buildkite can only select machines by stack, not by instance type. The x86 stack is not only used for docker build, but also for other ci pipelines. If the instance type is upgraded, it will cost more.

[lmatz]

The x86 stack is not only used for docker build, but also for other ci pipelines. If the instance type is upgraded, it will cost more.

what is the cost of having another dedicated x86 stack for docker build only? Is it even possible

[huangjw806]

what is the cost of having another dedicated x86 stack for docker build only? Is it even possible

A new cloudfromation stack needs to be created, and the maintenance cost will increase a bit, but the money won't increase much. In fact, for urgent patches, is it enough to just build the arm image?

[kwannoel]

what is the cost of having another dedicated x86 stack for docker build only? Is it even possible

A new cloudfromation stack needs to be created, and the maintenance cost will increase a bit, but the money won't increase much. In fact, for urgent patches, is it enough to just build the arm image?

Larger stack just for arm image lgtm.

[kwannoel]

Summarize:

• @huangjw806 provision a dedicated stack for arm image.
• @kwannoel add a dedicated pipeline which does not use LTO.

[BugenZhao]

but
TODO: cargo-chef doesn't work well now, because we update Cargo.lock very often.

I suppose this won't be a problem regarding patching on the release branch and we can still benefit from it. 🤔

[xxchan]

Yes, I think it's still a worth exploring direction (to see how technologies incl cargo-chef can work best), but I don't have the bandwidth to do the trial and error currently. I'd be glad to help if someone else want to try it. 😄