What is the relationship and differences of two segments in DataNode & QueryNode

[Reidddddd]

As far as I known, DataNode will handle writing data into object store by writeNode in a channel pipeline where there's a syncSegment call.

But I'm a bit confused when reading the source codes of QueryNode when I found there's a insertNode also will new segment and insert data into it. (seems to be local segment?)

Here we back to the question as QA title.

Hoping can get some insight answers from community. Thx.

[yhmo]

A "growing segment" exists both in query node and data node.
Query node subscribes data from message queue, the newly insert data is accumulated in the "growing segment".
Data node also subscribes data from message queue, the newly insert data also is accumulated inthe "growing segment".
The "growing segment" in query node and data node are pointing to the same logic segment.

Once the size of the growing segment exceeds a threshold value, the data node will "seal" it and it becomes a "sealed segment". At the same moment, the query node will know it become a sealed segment.

[Reidddddd]

are pointing to the same logic segment.

Is that mean, physically they are different?

[Reidddddd]

If physically different, then where is the segment inside Querynode stored?

[yhmo]

You create a collection, call collection.load() previously, and then insert 10 rows of data.
There is only one logic segment in the collection, the segment has 10 rows. The segment is "growing" since it only has 10 rows.
The query node has one copy of the segment in memory.
The data node also has one copy of the segment in memory.

You continue to insert 5 rows of data.
The query node receives the 5 rows from the message queue, now the querynode's growing segment has 15 rows.
The data node also receives the 5 rows from the message queue, now the datanode growing segment has 15 rows.

Query node doesn't persist the segment, it only keeps it in memory for search/query.
Data node doesn't do any search/query, it just keeps the segment in memory until its size exceeds the threshold. One exceed the threshold, the data node will persist the segment into storage as a "seal segment", and notify the query node "this segment is now converted to be sealed, you can sync its data to this timepoint xxxx". Then the query node knows "ok, sync the data to this timepoint xxxx and mark this segment as sealed".

[Reidddddd]

Screenshots are querynode's parts

[Reidddddd]

Screenshots are querynode's parts

@yhmo
from you sayings, these parts, in fact, is in memory.

[yhmo]

I believe this function is the query node receive data from the message queue and insert the data into the in-memory growing segment.
Yes, it is in memory, for query node.

func (sd *shardDelegator) ProcessInsert(insertRecords map[int64]*InsertData) {
      growing, err = segments.NewSegment()
      .......
      err := growing.Insert(context.Background(), insertData.RowIDs, insertData.Timestamps, insertData.InsertRecord)
}

[Reidddddd]

I believe this function is the query node receive data from the message queue and insert the data into the in-memory growing segment. Yes, it is in memory, for query node.

func (sd *shardDelegator) ProcessInsert(insertRecords map[int64]*InsertData) {
      growing, err = segments.NewSegment()
      .......
      err := growing.Insert(context.Background(), insertData.RowIDs, insertData.Timestamps, insertData.InsertRecord)
}

yes it is, thx!

May I ask why this part of memory needs to be written in C++, any advantage concern?

[liliu-z]

I believe this function is the query node receive data from the message queue and insert the data into the in-memory growing segment. Yes, it is in memory, for query node.

func (sd *shardDelegator) ProcessInsert(insertRecords map[int64]*InsertData) {
      growing, err = segments.NewSegment()
      .......
      err := growing.Insert(context.Background(), insertData.RowIDs, insertData.Timestamps, insertData.InsertRecord)
}

yes it is, thx!

May I ask why this part of memory needs to be written in C++, any advantage concern?

In the design, C++ is the data plane and Go is the control plane. All index building and searching work of vector/scalar data are in C++ side to make it more efficient, so to avoid complicated data transfer between Go and C++, we decide to put all data processing work in the C++ side

[Reidddddd]

Thanks @yhmo @liliu-z 🙏