How to completely remove overshadowed segments under var/druid/segments/ ?

[nibinqtl]

Affected Version

I'm running the currently latest 31.0.0

Description

• Cluster size: single-server setup
• Configurations in use: small

I repeatedly use the "local input source" to ingest records in json format at about 10,000 rows every 20 seconds.
Each ingest end up as a segment.
Then I use auto-compaction to combine them into larger segments of about 3M rows. It worked really well as shown on the console. In the segments list, I can only see the compacted large segments and the recent small segments not yet compacted.

However, I noticed that the old (overshadowed) small segments are still in the storage taking disk space. They are at:
var/druid/segments/
none of them are actually removed.
The directory var/druid/segments-cache/ seems to contain only the active segments and its size matches the size of the data source shown on the console.

How can I configure druid to actually remove those old unused overshadowed segments from disk?

[nibinqtl]

I found this API endpoint achieved exactly what I want:
https://druid.apache.org/docs/31.0.0/api-reference/data-management-api#permanently-delete-segments
It reduced the size of var/druid/segments/ down to 5% and var/druid/segments-cache/ is untouched.
According to the document, this API should only delete "unused segments". I noticed that most compacted segments var/druid/segments/ also got deleted. Maybe it is because there is another copy under var/druid/segments-cache/?

[kfaraz]

@nibinqtl , typically, you shouldn't need to call the API explicitly.
You could enable kill of unused segments by setting druid.coordinator.kill.on=true in the runtime.properties of the coordinator service.

[nibinqtl]

@kfaraz Thanks, but I already have

druid.coordinator.kill.on=true
druid.coordinator.kill.period=PT12H
druid.coordinator.kill.durationToRetain=P2D
druid.coordinator.kill.bufferPeriod=P1D
druid.coordinator.kill.maxSegments=10000

in the configuration, which does not seems to do anything.
I waited a few days and nothing happened to var/druid/segments/
Any suggetions?

[nibinqtl]

I just checked the content under var/druid/segments/, all the ingested small segments since 2024-11-18 (the day after I ran the DELETE API) are still there. Are they supposed to be killed 3 days after being compacted according to my configuration? Is there any other factors that can prevent the killing from happening?
@kfaraz

[nibinqtl]

And those segments are not in the "sys.segments" table. Maybe this is why the killing is not working?
It seems to be a bug to me.

[kfaraz]

And those segments are not in the "sys.segments" table. Maybe this is why the killing is not working?

sys.segments only shows "used" segments. So it is expected that unused segments will not show up there.

---

Are they supposed to be killed 3 days after being compacted according to my configuration? Is there any other factors that can prevent the killing from happening?

Based on Druid configuration docs,

druid.coordinator.kill.bufferPeriod=P1D

the bufferPeriod ensures that once segments have been marked as unused, we should wait for a day before completely deleting them.

druid.coordinator.kill.durationToRetain=P2D

the durationToRetain denotes that only the unused segments whose interval end is older than 2 days from now are eligible for deletion.

To debug your particular problem, you can check if any kill task was launched. If it was, you can try to look at the logs of that task to identify the segments that were actually killed.