can't query large collection

[AndrewOYLK]

When doing a large number of deletions or write operations on the collection, will it have any impact on read requests? I see that querynode has a large number of delete requests for this collection. I don't know if this will affect normal queries.

[2024/11/11 06:57:18.793 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=450679418892396320] [segmentID=453849863457179825] [error="index not found[segmentID=453849863457179825]"]

[yhmo]

Similar issue: #34221, which is fixed in >= v2.4.6
Not sure your issue is the same one. You can raise a new issue if the log is different.

Yes, heavy deletion operations might cause unexpected errors. Milvus developers have put a lot of effort to improve the performance and stability of large-scale deletion.

From the release note we can see: https://milvus.io/docs/release_notes.md
v2.4.15 ---------- "Additionally, Milvus 2.4.15 improved delete performance and stability through significant optimizations in L0 handling. "
v2.4.14 ---------- "Additionally, this release includes numerous enhancements focused on improving stability in large-scale delete operations and compaction performance."
v2.4.11 ---------- "This release introduces new features to view expressions and improve delete stability."
And a lot of related issues were fixed with the milvus version moving forward.

[AndrewOYLK]

I did not use helm to deploy the entire milvus system. I deployed etcd and pulsar components independently, and s3 used aliyun oss. So for this upgrade, I will independently upgrade the version of the milvus component (from v2.4.5->v2.4.15). I am not sure whether the current versions of etcd or pulsar will have any impact if I continue to use them.

[yhmo]

To know the required versions of etcd/pulsar/minio, download the milvus-helm latest release tar file form here: https://github.com/zilliztech/milvus-helm/releases/tag/milvus-4.2.19
Extract the tar file, open the requirements.yaml to see the version.

dependencies:
- name: etcd
  version: 6.3.3
  repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
  condition: etcd.enabled
  tags:
    - etcd
- name: minio
  version: 8.0.17
  repository: https://zilliztech.github.io/milvus-helm
  condition: minio.enabled
  tags:
    - minio
- name: pulsar
  version: 2.7.8
  repository: https://pulsar.apache.org/charts
  condition: pulsar.enabled
  tags:
    - pulsar
- name: kafka
  version: 15.5.1
  repository: https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
  condition: kafka.enabled
  tags:
    - kafka

[yhmo]

You can try different versions of pulsar/minio/etcd by yourself, we can't promise the other versions can work, because we didn't make full tests for the other versions, we don't have the report.

[AndrewOYLK]

Thank you, I finally want to confirm that during the upgrade process from 2.4.5 to 2.4.15, the metadata structure in etcd and the s3 storage structure have not changed, right? So the upgrade does not require executing certain scripts to change the metadata. The data is right. I understand that I just need to replace the container image version with the version in helm, right? There are no special shell update scripts to be executed in helm, right?

[yhmo]

There is no difference in metadata between minor versions. The metadata of 2.4.5 is the same as 2.4.15. You don't need any scripts for the upgrading. Just update the image version.