There are too many small files in the delta-log directory in the collection

[AndrewOYLK]

There are too many small files in the delta-log directory in the collection. Currently, we have encountered a collection with 10 million small files. Is there any way to reduce these small files

[yhmo]

I guess you called collection.flush() after each delete(). Each flush() will generate a tiny file.

[AndrewOYLK]

I have a collection containing more than 10,000 segments, with a total entity of more than 600 million, and each segment has more than 1,500 to 2,400 incremental log files. This causes my collection to be very slow every time I load it. Is there any way to reduce the number of segments and delta_log?

mivlus version is v2.4.5

[xiaofan-luan]

There are too many small files in the delta-log directory in the collection. Currently, we have encountered a collection with 10 million small files. Is there any way to reduce these small files

There are too many small files in the delta-log directory in the collection. Currently, we have encountered a collection with 10 million small files. Is there any way to reduce these small files

Are you doing frequent deletes? if so then you need to check why compaction is not triggered?
usually, when you have many delta log in one segment then compaction should automatically triggered.

if you can offer logs for datacoord and datanode we can help on investigate on why

[AndrewOYLK]

this is datacoord log：

birdwatcher info：

[AndrewOYLK]

segememts.txt

[AndrewOYLK]

How to adjust the GC parameters to accelerate the collection of these gems? Do you have any suggested parameters：

gc:
interval: 3600 # gc interval in seconds
missingTolerance: 86400 # file meta missing tolerance duration in seconds, default to 24hr(1d)
dropTolerance: 10800 # file belongs to dropped entity tolerance duration in seconds. 3600
removeConcurrent: 32 # number of concurrent goroutines to remove dropped s3 objects
scanInterval: 168 # garbage collection scan residue interval in hours

[xiaofan-luan]

Under you case, you have many small segments with 20000-30000 entities. and mix compaction seems to be slow to catch up.

A suggest is to create more shards when create collection and assign more datanodes to do the compaction.

if you have larger datanode maybe you can try to increase compaction concurrency.

What is your schema? how many partitions do you have? On each segments, how many deltalog do you have? what is the avg size of each delta log?

[xiaofan-luan]

There are 21 segments from different partitions.

[xiaofan-luan]

@tedxu

please also have an eye on this

[AndrewOYLK]

What compression parameters of datanode can be adjusted?

Collection ID：450679418892396320
entities: 642,296,196
partitions: 1
segments num：11002
total detal log num：1100w+
the avg size of each delta log：5KB - 20KB

[AndrewOYLK]

Which parameter should I adjust to speed up the reduction of Dropped segments?

[tedxu]

The initial suggestion I have is to minimize the creation of excessive segments. It's likely that the frequent execution of #flush() calls or improperly configured import tasks have led to the generation of many small segments. You might consider modifying these operations to prevent the creation of additional small segments.

As for optimizing compaction, here are a couple of adjustments you could consider:

1. Starting from version 2.4.14, setting datacoord.compaction.taskPrioritizer=mix will prioritize mix compactions over L0 compactions. Please be aware that this is an advanced configuration option, and it will block L0 compactions until the mix compactions are completed.
2. Increasing the number of datanodes can lift compaction parallelism. If you're working with large datanodes, you might also consider increasing the value of datanode.slot.slotCap, which is set to 16 by default for a setup with 4 CPU cores and 8 GB of memory.

[xiaofan-luan]

That's what I've been see
there is many small segments that can not be compacted.

by default, if each segment has more than 200 delta log, it will trigger a compaction. But right now it seems to be too many compactions.

Unfortunately this maxParallelTaskNum is a fixed number and we need to issue a fix on it

[xiaofan-luan]

The initial suggestion I have is to minimize the creation of excessive segments. It's likely that the frequent execution of #flush() calls or improperly configured import tasks have led to the generation of many small segments. You might consider modifying these operations to prevent the creation of additional small segments.

As for optimizing compaction, here are a couple of adjustments you could consider:

1. Starting from version 2.4.14, setting datacoord.compaction.taskPrioritizer=mix will prioritize mix compactions over L0 compactions. Please be aware that this is an advanced configuration option, and it will block L0 compactions until the mix compactions are completed.
2. Increasing the number of datanodes can lift compaction parallelism. If you're working with large datanodes, you might also consider increasing the value of datanode.slot.slotCap, which is set to 16 by default for a setup with 4 CPU cores and 8 GB of memory.

1. The priority setup of compaction is gonna to be a complicate problem, we need to evaluate by compaction size, segment numbers reducted, data cleaned and many other aspected. Let's discuss about it
2. I will introduce a new compaction selection policy try try to compact multiple small segments into one large segments. This could potentialy reduce the number of compaction and index building ncessary.
3. L0 compaction might be too slow. under that case concurrent L0 compaction might help?