rust is killed because it uses exceedingly high memory

[yurivict]

What type of bug is this?

Other

What subsystems are affected?

Other

Minimal reproduce step

Compilation runs out of memory.

The final link step keeps growing to 28GB+ and is being killed.

How to reduce the memory?

What did you expect to see?

n/a

What did you see instead?

see above

What operating system did you use?

FreeBSD 14.0

What version of GreptimeDB did you use?

0.6.0

Relevant log output and stack trace

n/a

[WenyXu]

try this https://github.com/rui314/mold

[tisonkun]

@yurivict what is your compilation command and could you share with the compilation logs with the error report?

[yurivict]

Here is the log.

[yurivict]

I was able to link the application with this patch:

--- Cargo.toml.orig     2024-01-12 09:12:11 UTC
+++ Cargo.toml
@@ -195,7 +195,8 @@ rev = "abbd357c1e193cd270ea65ee7652334a150b628f"
 rev = "abbd357c1e193cd270ea65ee7652334a150b628f"

 [profile.release]
-debug = 1
+debug = 0
+lto = "off"

 [profile.nightly]
 inherits = "release"

[tisonkun]

Great. I'll move this ticket to open-ended discussion for its visibility and this is not a certain issue to resolve, but a continuous effort to reduce the compilation memory consumption - it's good for developer experience, although we might prioritize the runtime efficiency.

It's somehow a Rust Philosophical issue, see - https://pingcap.medium.com/the-rust-compilation-model-calamity-1a8ce781cf6c

[sunng87]

By default, rust spawn N jobs for compile and linking, where N is the number of your cpu cores. You can change this behaviour by using -j <N> option for cargo build. You may also customize this in your .cargo/config.toml

[tisonkun]

However, the final link step can only leverage one core (handled in one task) - is this correct?

[tisonkun]

And also the reported issue is about memory. Increasing parallelism doesn't seem help.

[sunng87]

However, the final link step can only leverage one core (handled in one task) - is this correct?

No. At the moment, it follows -j parameter as well. This is the reason why you might experience OOM even if you have large memory setup.