rptest: add log_compaction_test.py

[WillemKauf]

Serves as a stress test for compaction and tombstone removal.

Depends on changes made in redpanda-data/kgo-verifier#60, and so the commit SHA for ducktape-deps/kgo-verifier has been updated (CI will be red if this isn't merged yet)

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.3.x
• v24.2.x
• v24.1.x

Release Notes

Bug Fixes

• Fixes a bug in which segments which may have tombstones in them were not considered eligible for self-compaction

Improvements

• Adds a number of new metrics to improve observability of the compaction subsystem:
  • Added the _segment_cleanly_compacted metric to segment::probe.
  • Added the _segments_marked_tombstone_free metric to segment::probe.
  • Added the _num_rounds_window_compaction metric to segment::probe.

[vbotbuildovich]

the below tests from https://buildkite.com/redpanda/redpanda/builds/58820#019369b8-b74d-4c4f-9b4a-3da1ca75260f have failed and will be retried

storage_e2e_single_thread_rpunit

the below tests from https://buildkite.com/redpanda/redpanda/builds/59112#01938800-b170-476c-89c6-a08ca0f89f11 have failed and will be retried

kafka_server_rpfixture

the below tests from https://buildkite.com/redpanda/redpanda/builds/59290#019397a1-424a-44d4-8ee9-8d90c6545947 have failed and will be retried

gtest_raft_rpunit

the below tests from https://buildkite.com/redpanda/redpanda/builds/59376#01939c49-56cd-4534-8cca-6e8c29cafbad have failed and will be retried

partition_balancer_simulator_test_rpunit

[dotnwat]

   assert consumer.consumer_status.validator.tombstones_consumed == 0
AssertionError

eek!

[WillemKauf]

DEBUG 2024-12-06 00:12:13,392 [shard 0:lc ] storage-gc - disk_log_impl.cc:1312 - Compaction of {kafka/tapioca/0} stopped because of shutdown

DEBUG 2024-12-06 00:12:49,334 [shard 0:lc ] storage-gc - segment_utils.cc:749 - detected /var/lib/redpanda/data/kafka/tapioca/0_22/9228-1-v1.compaction_index is already compacted

TRACE 2024-12-06 00:12:53,410 [shard 0:main] storage - segment.cc:92 - closing segment: {reader={/var/lib/redpanda/data/kafka/tapioca/0_22/9228-1-v1.log, (470102 bytes)}, clean_compact_timestamp:{{timestamp: 1733443928831}}, may_have_tombstone_records:1}

This is a segment closed with a clean_compact_timestamp marked but with may_have_tombstone_records == 1.

Eek indeed. I believe this is a bug.

This if statement here should check if the segment may_have_tombstone_records and proceed with self compacting it, even if the compacted_index::recovery_state is already_compacted

[WillemKauf]

1500 CI runs pays off - added commit to deal with the above.

470fdc3

[dotnwat]

This if statement here should check if the segment may_have_tombstone_records and proceed with self compacting it, even if the compacted_index::recovery_state is already_compacted

Interesting. Is this a case where we would have been better off not falling back to old style compaction after sliding window (is that's whats happening there with self compact)?

[WillemKauf]

/ci-repeat 5
release
skip-units
dt-repeat=100
tests/rptest/tests/log_compaction_test.py

[WillemKauf]

(oops. only half of my commit got added last night. that's what I get for working late)

Interesting. Is this a case where we would have been better off not falling back to old style compaction after sliding window (is that's whats happening there with self compact)?

No, there's only two places where tombstones are removed, during self compaction de-duplication and sliding window deduplication.

Therefore the tombstones for this case would have been removed if more sliding window rounds took place in the future, but since we produce and then stop with this test, it's expected that self compaction is going to remove the rest of the tombstones before the consumer sees them (which wouldn't happen before this fix)

[vbotbuildovich]

Retry command for Build#59376

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/availability_test.py::AvailabilityTests.test_recovery_after_catastrophic_failure
tests/rptest/tests/topic_creation_test.py::CreateTopicReplicaDistributionTest.test_topic_aware_distribution

[WillemKauf]

/ci-repeat 5
release
skip-units
dt-repeat=100
tests/rptest/tests/log_compaction_test.py

[dotnwat]

Retry command for Build#59376

Where all the failures for other tests?

[WillemKauf]

/ci-repeat 5
release
skip-units
skip-redpanda-build
dt-repeat=100
tests/rptest/tests/log_compaction_test.py

[WillemKauf]

Looking good

[vbotbuildovich]

/backport v24.3.x

[vbotbuildovich]

Failed to create a backport PR to v24.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-24187-v24.3.x-979 remotes/upstream/v24.3.x
git cherry-pick -x 1f5f3633ee 5b50f25e9b 8d66388ecf 87d442bf2a 85bea2b00b 15ab6a8557 ac273cc321 985beffa20 d238b1c9b6

Workflow run logs.