Hard to debug in standalone mode due to logs of different components being in the same file

[lmatz]

          > use standalone mode.

We will recommend users to use standalone mode by default once it is integrated into the test pipelines and we found no problem

@lmatz After change the integration pipeline to use standalone mode, all logs printed by different components (cn, compactor, etc) are interleaved. And we cannot easily grep by component name, since there is no component name prefix in the log, only crate prefix. That makes it hard to analyze the failed test case.

So what about change the docker-compose.yml of integration test to inherit from docker-compose-distributed.yml instead of docker-compose.yml.

Originally posted by @StrikeW in #13235 (comment)

[kwannoel]

Hmm isn't crate prefix mostly sufficient (risingwave_meta, risingwave_stream, etc...)? Could you share some examples @StrikeW ?

[StrikeW]

Some crates are shared by multiple components. IMO the extent of interleave depends on the workload. In the integration test scenario, workload is usually light, so it is still acceptable. But if we run complex workload e.g. ch-benchmark, I think it would be hard to analyze issue specific to a component.

[kwannoel]

Some crates are shared by multiple components. IMO the extent of interleave depends on the workload. In the integration test scenario, workload is usually light, so it is still acceptable. But if we run complex workload e.g. ch-benchmark, I think it would be hard to analyze issue specific to a component.

What I'm thinking is if the solution can be as simple as providing a script or tool to partition the logs by crate.

[BugenZhao]

IIRC, most of our workloads run under their specific runtimes, where the custom thread names are set.

risingwave/src/stream/src/task/stream_manager.rs

Lines 424 to 434 in 9dcad74

	let runtime = {
	let mut builder = tokio::runtime::Builder::new_multi_thread();
	if let Some(worker_threads_num) = config.actor_runtime_worker_threads_num {
	builder.worker_threads(worker_threads_num);
	}
	builder
	.thread_name("risingwave-streaming-actor")
	.enable_all()
	.build()
	.unwrap()
	};

Do you think including the thread name in the logs will help?

https://docs.rs/tracing-subscriber/latest/tracing_subscriber/fmt/format/struct.Format.html#method.with_thread_names

[StrikeW]

Do you think including the thread name in the logs will help?

This one looks better to me. But I think it may not cover the case of the embedded jvm, those log printed in java side is running in a dedicated thread. Or we may need to assign a name to it.

Do we will use standalone on the cloud? I am not sure whether the plugin can collect the logs belonging to different components with the above solutions.

[lmatz]

Do we will use standalone on the cloud?

Yes, for dev tier

[fuyufjh]

Closed by #14114