storage: add log lines about removed data

[andrwng]

It's very difficult to retroactively debug why records are removed by compaction. This adds some logging about removed data.

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

Improvements

• Adds logging to mention data removed by compaction.

[vbotbuildovich]

CI test results

test results on build#60138

test_id	test_kind	job_url	test_status	passed
rptest.tests.datalake.partition_movement_test.PartitionMovementTest.test_cross_core_movements.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/60138#0193fbc0-165c-46d3-8d19-3316953f53ef	FLAKY	3/6

test results on build#60371

test_id	test_kind	job_url	test_status	passed
rm_stm_tests_rpunit.rm_stm_tests_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/60371#0194426f-9a77-4b0c-9ac5-2cef1d02af04	FLAKY	1/2
rm_stm_tests_rpunit.rm_stm_tests_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/60371#0194426f-9a78-4c03-889c-5960f45a0b1f	FLAKY	1/2
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade	ducktape	https://buildkite.com/redpanda/redpanda/builds/60371#019442b2-6b37-4c14-809f-4fa476fe7637	FLAKY	5/6
rptest.tests.partition_reassignments_test.PartitionReassignmentsTest.test_reassignments_kafka_cli	ducktape	https://buildkite.com/redpanda/redpanda/builds/60371#019442cc-edc4-47e7-bea7-3f1dd5aa955b	FLAKY	3/6

[WillemKauf]

I don't have strong feelings that this shouldn't be at the info level, but I can imagine that it would make compaction output for a log with a high number of segments much noisier.

What do you think?

[dotnwat]

Definitely more noisy, but we already put up with an info level message when we create a segment, so maybe not too bad. are there situations where we re-compact a segment many times, or is it usually <= 2x?

[andrwng]

Yea I'm trying to be cognizant about noise in making this conditionally a debug log.

It's certainly possible we'll compact a segment multiple times (it's not hard to imagine with high cardinality segments). But generally I'd argue that despite with the noise, if we're removing data, the log line is worth it. There was one recent instance where tracing back a record's removal was impossible because of a lack of observability in this area.

[andrwng]

/ci-repeat

[vbotbuildovich]

/backport v24.3.x

[vbotbuildovich]

/backport v24.2.x

[vbotbuildovich]

Failed to create a backport PR to v24.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-24659-v24.2.x-347 remotes/upstream/v24.2.x
git cherry-pick -x 33c3aa492e 8c84ffdabf

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v24.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-24659-v24.3.x-438 remotes/upstream/v24.3.x
git cherry-pick -x 33c3aa492e 8c84ffdabf

Workflow run logs.