Concurrent inserts for the same key

[rere950303]

I am using milvus 2.3.12 version.

When there are 2 or more datanode replicas, can flush to object storage with the same key occur at the same time? If so, I am asking because I think there may be an issue with the object storage we are using.

Looking at the key format, it is ${tenant}/insert_log/${collection_id}/${partition_id}/${segment_id}/${field_id}/_${log_idx}, so log_idx is only the log file existing in each datanode, so wouldn't that happen?

[yhmo]

${tenant}/insert_log/${collection_id}/${partition_id}/${segment_id}/${field_id}/_${log_idx} is a path of a binlog file. Binlog file is a column-based format data file of milvus to store multiple rows data of one field.
Binlog files are persisted and uploaded by data node, different data nodes will process different segments with different "segment_id".

[yhmo]

Most of the files are uploaded at different time, by different nodes(data node, index node).

[rere950303]

@yhmo
What I'm wondering is, is it possible to flush objects simultaneously with the same key?

[yhmo]

Not sure what you want to do.
Milvus manages the data by its logic, automatically. The internal segment id//log_id are generated by milvus, and the ids are unique. So, why milvus needs to "flush objects with the same key"?

[xiaofan-luan]

@yhmo What I'm wondering is, is it possible to flush objects simultaneously with the same key?

LogID is globally unique and assigned by Rootcoord. that's why there is never gonna to be a same key here

[rere950303]

@yhmo @xiaofan-luan thanks for replying!