knowhere-v1.3.21

What's Changed

• Add async thread pool for generating diskann cache and catch unexpected return. by @cqy123456 in #226
• Change async thread pool size to 1 by @cqy123456 in #234
• Add some log for diskann prepare by @cqy123456 in #235
• Fix diskann async cache by @liliu-z in #250

Full Changelog: v1.3.20...v1.3.21

knowhere-v1.3.20

What's Changed

• Generate diskann cache asynchronously. by @cqy123456 in #191

Full Changelog: v1.3.19...v1.3.20

knowhere-v1.3.19

What's Changed

• Reverted changes to modify knowhere threads scheduling priority

Full Changelog: v1.3.18-hotfix...v1.3.19

knowhere-v2.2.2

Release Notes

What's New

• SCANN without refine: (#77)

  • SCANN now supports a with_raw_data config option to determine whether the refine process is needed. Omitting the refine step can dramatically reduce memory usage (to approximately 1/17 of the original).

• Multi devices for GPU: (#83 #146)

  • GPU indexes can now be distributed in different cards to accelerate and the performance and enlarge the capacity.

• Remove IVF_FLAT_NM: (#39):

  • IVF_FLAT_NM was introduced to conserve disk resources by avoiding the encoding of raw data during serialization. However, this approach posed challenges for the support of our new features and when upgrading FAISS. In this release, we've reverted the implementation of IVF_FLAT to the original FAISS version.

• DiskANN filtering search optimization: (#143)

  • We adopt a strategy that randomly determines whether to add a filtered vector to the candidate list, aiming to enhance DiskANN's filtering search performance with a high filtering rate.

• Reprioritize thread pool: ( #130)

  • We deprioritize Knowhere's threads to support better resource allocation of Milvus.

• Support GetVectorById for IVFFLAT Cosine (#80)

Enhancements and Fixes

• Replace OMP with Knowhere's thread pool to better manage thread number. (#120)
• Implement Size() API to estimate the memory usage for HNSW and DiskANN. (#124 #126)
• Fix race condition in GetVectorById of IVF indexes. (#133 #134)

New Contributors

• @alexanderguzhva made their first contribution in #72

v1.3.18-hotfix

What's Changed

• Fix DiskANN LRU Set Invalid Medoid by @PwzXxm in #97

Full Changelog: v1.3.18...v1.3.18-hotfix

knowhere-v2.2.1

What's Changed

• Temporary solution to fix ivf_flat search result wrong when cosine by @cydrain in #36
• Fix scann add_with_ids logic by @chasingegg in #45
• Fix brute force search wrong when COSINE by @cydrain in #48
• Fix segment fault when use config load a list by @Presburger in #49

New Contributors

• @weiliu1031 made their first contribution in #32

Full Changelog: v2.2.0...v2.2.1

knowhere-v2.2.0

Release Notes

New Features

• FastScan Integration with FAISS: (#1)

  • Introducing support for FastScan from FAISS. Functionally analogous to SCANN, FastScan utilizes a 4-bit PQ with IVF to enhance the search speed, making it ideal for high-dimensional vector searches.

• New index type IVF_FLAT_CC: (milvus-io/knowhere#824)

  • Introducing new index type IVF_FLAT_CC for growing segment searching.

• Memory-Mapped Support: (milvus-io/knowhere#785 milvus-io/knowhere#1006 milvus-io/knowhere#1015)

  • mmap is now compatible with the FAISS series of indexes and HNSW.

• Extended Metric Support: (milvus-io/knowhere#812)

  • We've added support for the COSINE metric across all float indexes.

• Binary Data Handling: (milvus-io/knowhere#978)

  • HNSW can now process binary data with metric type HAMMING or JACCARD.

• Support prometheus metrics: (milvus-io/knowhere#756)

• GetVectorByIds API: (milvus-io/knowhere#791 milvus-io/knowhere#802 milvus-io/knowhere#805 milvus-io/knowhere#815)

  • This new API facilitates the retrieval of vectors from indexes.

  • Users must first invoke HasRawData prior to GetVectorByIds to confirm the vectors' presence. (milvus-io/knowhere#828)

    Support Matrix for GetVectorByIds

    Indexes	IP	L2	COSINE
    FLAT	Y	Y	Y
    IVF_FLAT	Y	Y	N
    IVF_PQ	N	N	N
    IVF_SQ	N	N	N
    HNSW	Y	Y	Y
    DiskANN	N	Y	Y
    SCANN	Y	Y	Y

Other Changes

• Dependency Management with Conan: (milvus-io/knowhere#728)

  • We've integrated conan to streamline our dependency management.

• Replace Easylogging++ by Glog: (milvus-io/knowhere#766)

• Integration with Folly: (milvus-io/knowhere#788)

  • We've incorporated folly as our preferred third-party standard library.

• Unified RangeSearch Semantics: (milvus-io/knowhere#1016)

  • The RangeSearch functions of Faiss IVF-serials have been updated to be parameter-less for indexes, providing a cohesive semantic experience.

• Enhanced Search Capabilities: (milvus-io/knowhere#794)

  • The search performance for HNSW has been optimized, especially for filtering.

• Thread Management Improvements: (#15)

  • To counteract thread contention, build and search thread pools have been separated. Additionally, Knowhere's thread priority is automatically lowered to prevent disruptions to the primary thread.

• Discontinued Metrics for Binary Indexes: (milvus-io/knowhere#994)

  • Support for metrics such as TANIMOTO, SUBSTRUCTURE, and SUPERSTRUCTURE has been phased out for Binary indexes.

• Discontinued index type ANNOY: (milvus-io/knowhere#796)

  • Support for index type ANNOY has been phased out.

• Improve code coverage: (milvus-io/knowhere#949 milvus-io/knowhere#955 milvus-io/knowhere#958 milvus-io/knowhere#959 milvus-io/knowhere#972)

knowhere-v1.3.18

What changes

• Deprioritize the thread pool to ensure the main thread's healthy.