How is LlamaIndex storing data in Milvus ?

[joybanerjee08]

Hi guys,
I have been tasked with researching viability of using Milvus with LlamaIndex for storing more than a million documents.

I have used a simple code such as this to store a sample of 100 document, and retrieve some data:

vector_store = MilvusVectorStore(
    uri="http://localhost:19530/my_db", collection_name='my_collection', dim=384, overwrite=True
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)
retriever = index.as_retriever(similarity_top_k=3)
response_chunks = retriever.retrieve("Details of Internal Combustion Engines")
print(response_chunks[0])

I can see the node id, text and similarity scores in the output of the above code.

Next I have used pymilvus to describe the collection, it gave me this output:

{"collection_name": "my_collection", "auto_id": False, "num_shards": 1, "description": "", "fields": [{"field_id": 100, "name": "id", "description": "", "type": <DataType.VARCHAR: 21>, "params": {"max_length": 65535}, "is_primary": True}, {"field_id": 101, "name": "embedding", "description": "", "type": <DataType.FLOAT_VECTOR: 101>, "params": {"dim": 384}}], "aliases": [], "collection_id": 453383211425148447, "consistency_level": 1, "properties": {}, "num_partitions": 1, "enable_dynamic_field": True}

I can see two fields, id and embedding. So where are the texts and metadata ? If they are not stored, then how is LlamaIndex fetching the text ?

[zc277584121]

hello @joybanerjee08 , as you can see, "enable_dynamic_field": True in the description, it means by default , milvus use the dynamic field feature to storage these metadatas and other fields except for some basic fields such as id, vector. Simply speaking, you can regard it as containing fields such as metadata, and it can dynamically add any other fields. You can also dive into our documents to get a better understand. https://milvus.io/docs/enable-dynamic-field.md#Enable-Dynamic-Field

[joybanerjee08]

Hi @zc277584121, I understand that with dynamic fields as true, we can add new fields later on the go. But the new fields like metadata or text are never created by llama Index. After loading all the documents, I'm only seeing two fields: id and embeddings.

[zc277584121]

Because only id and embeddings are standard fields, while other fields like metadata and text are in the dynamic field.
You can try this code to get a random entity in your collection:

print(vector_store.client.search(
    collection_name="my_collection", 
    data=[[0.1] * 384],
    limit=2,
    output_fields=["*"],
))

If everything is right, you will see all the fields.

If you want to get a more directly view of your milvus collections, you can try to use attu, which is a visualization tool for milvus and user friendly to get started.