Magic Search #37927

AriesJin · 2024-11-22T04:59:17Z

AriesJin
Nov 22, 2024

i want to test the search function, but i have get magic search result

store data

res = client.query(
    collection_name="test_ez2",
    filter="",  # 如果需要，可以在这里添加过滤条件
    output_fields=["*"],  # 返回所有字段
    partition_names=["_default","partA"],  # 指定要查询的分区名称
    limit=1000  # 调整这个值以适应您的需求
)
data: ["{'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'}", "{'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'}"]

search 1

only return "file_name" and "color", it's work fine.

res = client.search(
    collection_name="test_ez2", # Replace with the actual name of your collection
    data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
    limit=10, # Max. number of search results to return
    search_params={"metric_type": "COSINE", "params": {"nprobe": 200, # Radius of the search circle
        # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                                  }}, # Search parameters
    output_fields=["file_name", "color"] # Output fields to return
)
data: ["[{'id': '222', 'distance': 0.9995954036712646, 'entity': {'file_name': '222', 'color': 'red_7025'}}, {'id': '111', 'distance': 0.9989381432533264, 'entity': {'file_name': '111', 'color': 'pink_8682'}}]"]

search 2

return "file_name" ,"color" and " vector", it's work error.

res = client.search(
    collection_name="test_ez2", # Replace with the actual name of your collection
    data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
    limit=10, # Max. number of search results to return
    search_params={"metric_type": "COSINE", "params": {"nprobe": 200, # Radius of the search circle
        # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                                  }}, # Search parameters
    output_fields=["file_name", "color", "vector"] # Output fields to return
)
RPC error: [search], <MilvusException: (code=65535, message=incomplete query result, missing id 222, len(searchIDs) = 2, len(queryIDs) = 1, collection=453236263229647026)>, <Time:{'RPC start': '2024-11-22 04:57:51.694358', 'RPC error': '2024-11-22 04:57:51.708931'}>
Failed to search collection: test_ez2

search 2

i change the limit=5
return "file_name" ,"color" and " vector", and it's work fine. but only return ont embedding

res = client.search(
    collection_name="test_ez2", # Replace with the actual name of your collection
    data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
    limit=5, # Max. number of search results to return
    search_params={"metric_type": "COSINE", "params": {"nprobe": 200, # Radius of the search circle
        # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                                  }}, # Search parameters
    output_fields=["file_name", "color", "vector"] # Output fields to return
)
data: ["[{'id': '222', 'distance': 0.9995954036712646, 'entity': {'vector': [3.200000047683716, 0.20000000298023224, 0.20000000298023224, 0.2199999988079071, 0.33000001311302185, 9.329999923706055], 'file_name': '222', 'color': 'red_7025'}}]"]

what happened？
why limit=5 is ok， but limit=10 will get error？and when limit=5， why only one data recalled？
anyone can help me？

xiaofan-luan · 2024-11-22T05:24:46Z

xiaofan-luan
Nov 22, 2024
Maintainer

i want to test the search function, but i have get magic search result

store data

res = client.query(
    collection_name="test_ez2",
    filter="",  # 如果需要，可以在这里添加过滤条件
    output_fields=["*"],  # 返回所有字段
    partition_names=["_default","partA"],  # 指定要查询的分区名称
    limit=1000  # 调整这个值以适应您的需求
)
data: ["{'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'}", "{'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'}"]

search 1

only return "file_name" and "color", it's work fine.

res = client.search(
    collection_name="test_ez2", # Replace with the actual name of your collection
    data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
    limit=10, # Max. number of search results to return
    search_params={"metric_type": "COSINE", "params": {"nprobe": 200, # Radius of the search circle
        # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                                  }}, # Search parameters
    output_fields=["file_name", "color"] # Output fields to return
)
data: ["[{'id': '222', 'distance': 0.9995954036712646, 'entity': {'file_name': '222', 'color': 'red_7025'}}, {'id': '111', 'distance': 0.9989381432533264, 'entity': {'file_name': '111', 'color': 'pink_8682'}}]"]

search 2

return "file_name" ,"color" and " vector", it's work error.

res = client.search(
    collection_name="test_ez2", # Replace with the actual name of your collection
    data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
    limit=10, # Max. number of search results to return
    search_params={"metric_type": "COSINE", "params": {"nprobe": 200, # Radius of the search circle
        # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                                  }}, # Search parameters
    output_fields=["file_name", "color", "vector"] # Output fields to return
)
RPC error: [search], <MilvusException: (code=65535, message=incomplete query result, missing id 222, len(searchIDs) = 2, len(queryIDs) = 1, collection=453236263229647026)>, <Time:{'RPC start': '2024-11-22 04:57:51.694358', 'RPC error': '2024-11-22 04:57:51.708931'}>
Failed to search collection: test_ez2

search 2

i change the limit=5 return "file_name" ,"color" and " vector", and it's work fine. but only return ont embedding

res = client.search(
    collection_name="test_ez2", # Replace with the actual name of your collection
    data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
    limit=5, # Max. number of search results to return
    search_params={"metric_type": "COSINE", "params": {"nprobe": 200, # Radius of the search circle
        # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                                  }}, # Search parameters
    output_fields=["file_name", "color", "vector"] # Output fields to return
)
data: ["[{'id': '222', 'distance': 0.9995954036712646, 'entity': {'vector': [3.200000047683716, 0.20000000298023224, 0.20000000298023224, 0.2199999988079071, 0.33000001311302185, 9.329999923706055], 'file_name': '222', 'color': 'red_7025'}}]"]

what happened？ why limit=5 is ok， but limit=10 will get error？and when limit=5， why only one data recalled？ anyone can help me？

because you may have duplicated PK in the system and when search both PK is searched and there is not enough result to return.
you can check duplicated PK using birdwatcher

8 replies

yhmo Nov 22, 2024
Collaborator

I could not reproduce your issue with this script:
This script creates a new collection and inserts the two rows data.

from pymilvus import (
    MilvusClient,
    DataType,
)

if __name__ == "__main__":
    client = MilvusClient(uri='http://localhost:19530')

    schema = client.create_schema(
        auto_id=False,
        enable_dynamic_schema=True,
    )

    schema.add_field(field_name="file_name", datatype=DataType.VARCHAR, max_length=100, is_primary=True)
    schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=6)
    schema.add_field(field_name="color", datatype=DataType.VARCHAR, max_length=100)

    index_params = client.prepare_index_params()
    index_params.add_index(field_name="vector", index_type="IVF_FLAT", metric_type="COSINE", params={'nlist': 512})

    client.drop_collection(collection_name="test_ez2")
    client.create_collection(collection_name="test_ez2", schema=schema, index_params=index_params)


    client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
    client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})


    client.load_collection(collection_name="test_ez2")
    print("loaded")
    print(client.query(collection_name="test_ez2", filter="", output_fields=["count(*)"], consistency_level="Strong"))

    print("##############################################################################################")
    results = client.search(
        collection_name="test_ez2",  # Replace with the actual name of your collection
        data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
        limit=10,  # Max. number of search results to return
        search_params={"metric_type": "COSINE",
                       "params": {"nprobe": 200,  # Radius of the search circle
                                  # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                  }},  # Search parameters
        output_fields=["file_name", "color"]  # Output fields to return
    )
    for res in results[0]:
        print(res)
    print("##############################################################################################")
    results = client.search(
        collection_name="test_ez2",  # Replace with the actual name of your collection
        data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
        limit=10,  # Max. number of search results to return
        search_params={"metric_type": "COSINE",
                       "params": {"nprobe": 200,  # Radius of the search circle
                                  # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                  }},  # Search parameters
        output_fields=["file_name", "color", "vector"]  # Output fields to return
    )
    for res in results[0]:
        print(res)
    print("##############################################################################################")
    results = client.search(
        collection_name="test_ez2",  # Replace with the actual name of your collection
        data=[[3.5, 0.2, 0.2, 0.22, 0.33, 9.33]],
        limit=5,  # Max. number of search results to return
        search_params={"metric_type": "COSINE",
                       "params": {"nprobe": 200,  # Radius of the search circle
                                  # "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
                                  }},  # Search parameters
        output_fields=["file_name", "color", "vector"]  # Output fields to return
    )
    for res in results[0]:
        print(res)

All three search requests return correct results.

loaded
data: ["{'count(*)': 2}"] 
##############################################################################################
{'id': '222', 'distance': 0.9995954036712646, 'entity': {'file_name': '222', 'color': 'red_7025'}}
{'id': '111', 'distance': 0.9989381432533264, 'entity': {'file_name': '111', 'color': 'pink_8682'}}
##############################################################################################
{'id': '222', 'distance': 0.9995954036712646, 'entity': {'vector': [3.200000047683716, 0.20000000298023224, 0.20000000298023224, 0.2199999988079071, 0.33000001311302185, 9.329999923706055], 'file_name': '222', 'color': 'red_7025'}}
{'id': '111', 'distance': 0.9989381432533264, 'entity': {'vector': [4.0, 0.20000000298023224, 0.20000000298023224, 0.2199999988079071, 0.33000001311302185, 9.329999923706055], 'file_name': '111', 'color': 'pink_8682'}}
##############################################################################################
{'id': '222', 'distance': 0.9995954036712646, 'entity': {'vector': [3.200000047683716, 0.20000000298023224, 0.20000000298023224, 0.2199999988079071, 0.33000001311302185, 9.329999923706055], 'file_name': '222', 'color': 'red_7025'}}
{'id': '111', 'distance': 0.9989381432533264, 'entity': {'vector': [4.0, 0.20000000298023224, 0.20000000298023224, 0.2199999988079071, 0.33000001311302185, 9.329999923706055], 'file_name': '111', 'color': 'pink_8682'}}

Process finished with exit code 0

AriesJin Nov 22, 2024
Author

i also reproduce your result! but if you repeat insert the two row data，you will get an interesting result, like:

replace the code of

client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})

as

client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})

client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})

and now, if your set the limit = 2 and limit =10, will get different result.

after that, you can change "output_fields=["file_name", "color", "vector"]" to "output_fields=["file_name", "color"]", got a new result.

xiaofan-luan Nov 22, 2024
Maintainer

i also reproduce your result! but if you repeat insert the two row data，you will get an interesting result, like:

replace the code of

client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})

as

client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})

client.insert(collection_name="test_ez2", data={'file_name': '111', 'vector': [4.0, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'pink_8682'})
client.insert(collection_name="test_ez2", data={'file_name': '222', 'vector': [3.2, 0.2, 0.2, 0.22, 0.33, 9.33], 'color': 'red_7025'})

and now, if your set the limit = 2 and limit =10, will get different result.

after that, you can change "output_fields=["file_name", "color", "vector"]" to "output_fields=["file_name", "color"]", got a new result.

I think this is due to your PK is duplicated.
The second insert need to be upsert.
Milvus didn't replace your PK

xiaofan-luan Nov 22, 2024
Maintainer

So if you want the second file_name = 111 overwrite the first one. You will need to do a upsert instead of insert.

@yanliang567 We need to test on this and fix.
I don't think we can resolve all the duplicate on read . Please invite @congqixia and @zhagnlu for a meeting

AriesJin Nov 23, 2024
Author

thanks! i think upsert can fix it. but as it is written in the documentation “If you plan to use the upsert operation instead of insert for large-scale data ingestion (e.g. millions of vectors), be aware that this can lead to high memory consumption on Milvus data nodes.” .

sometimes, it's hard to determine whether the "insert data" is existed in collection, so use upsert is must?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Magic Search #37927

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 8 replies

{{title}}

store data

search 1

search 2

search 2

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Magic Search #37927

AriesJin Nov 22, 2024

store data

search 1

search 2

search 2

Replies: 1 comment · 8 replies

xiaofan-luan Nov 22, 2024 Maintainer

store data

search 1

search 2

search 2

yhmo Nov 22, 2024 Collaborator

AriesJin Nov 22, 2024 Author

xiaofan-luan Nov 22, 2024 Maintainer

xiaofan-luan Nov 22, 2024 Maintainer

AriesJin Nov 23, 2024 Author

AriesJin
Nov 22, 2024

Replies: 1 comment 8 replies

xiaofan-luan
Nov 22, 2024
Maintainer

yhmo Nov 22, 2024
Collaborator

AriesJin Nov 22, 2024
Author

xiaofan-luan Nov 22, 2024
Maintainer

xiaofan-luan Nov 22, 2024
Maintainer

AriesJin Nov 23, 2024
Author