Load collection and insert data

[ventaubain]

During an "insert campaign", i would like verify (for each new data) if the data is already indexed in the database by checking its ID (which it is defined by me) else ignore it. For each step, I must be sure there is not duplication so the database must be up-to-date at each insertion step. To check if the ID is already indexed, I have to realize a query and, to do that, load the collection in memory before. The load operation is time-consuming and not very efficient for massive insertion. I use Milvus 2.0.
What's the good approach to do that in an efficient way ? The query/load step is necessary or there is an another method ?

Thanks for your help !

Edit: Resolved: Misunderstanding with the API - Dideplucation/Search by id in the Roadmap

[xiaofan-luan]

You only need to load once and then you query by id would work even if insertion happens after load.
Primary key deduplication is already in our roadmap, once the functionality is ready, you no longer need to do check by yourself~

[xiaofan-luan]

Why do you even need to load a collection? Won't it work without .load() command?

Can I insert new values after .load() command?

Yes, you can insert after load.
Since milvus put all data in memory, and some users don't have enough memory.
So you always need to load once before you search, but you can insert without load

[LMERose]

hello, if I want to use this :create_collection --> insert（） --> create_index（） --> load（） --> search（）,what should I do?Which part of the program should I modify? I'm using the langchain interface

[barzan-hayati]

Does this resolved in the newest version of milvus?

Primary key deduplication is already in our roadmap

[liliu-z]

Does this resolved in the newest version of milvus?

Primary key deduplication is already in our roadmap

Upsert can help on this

[liliu-z]

hello, if I want to use this :create_collection --> insert（） --> create_index（） --> load（） --> search（）,what should I do?Which part of the program should I modify? I'm using the langchain interface

Did meet any issue with this order?