Adding additional distance

[jwall149]

I'm building a service that finding similarity between malware and virus based TLSH.
TLSH is a high dimension vector represents the binary of the malware/virus, that the distance between two vectors satisfy the inequality triangle theorem.

I want to index and search these vector to find/group similar malware and virus, but the distance algorithm is quite unique that does not based on any distance Milvus supports, and can not be approximate to any of the supported distances (cosine, euclidean, dot, hamming....).

Can anybody please give me a good tutorial where should I start forking the code base and build the software.

I am completely new to Milvus, so please advise me on the index as well, I'm thinking of SCANN.

[yhmo]

It is not a simple job to add a new index type or new metric type into milvus.
Wait @liliu-z to comment.

[liliu-z]

Milvus has already support SCANN and so many other indexes, you can use any one since TLSH satisfies the inequality triangle theorem. For more suggestion, HNSW can provide better performance&recall but need more memory, DiskANN can utilize disk to enlarge your capacity, SCANN can provide not bad performance and suitable for k>1K scenarios like all other IVF-series indexes.

However, we only support IVFFlat for binary embedding. Not sure what is your data type in this case.

To support more metrics, @foxspy can you help on this?

[jwall149]

The value of a dimension mostly integer, and most of them are 2 bit integer (Just 0,1,2,3).

The distance also an integer can varies from 0 to 2000. Similar to Manhattan distance, I'm afraid of losing discriminatory power when it come to HNSW or any other index algorithm. What do you think about this?

I can spend time and code the custom metric, please guide me.

Also I need to work on indexing/search on 50 billions+ vectors, please advise

[xiaofan-luan]

I'm assuming we are talking about https://github.com/trendmicro/tlsh

[xiaofan-luan]

I think it's always better to start from https://github.com/zilliztech/knowhere.
Questions:

1. what does the vector looks like? [1,3,2,4]... multible 2 bit integers and it has to represent by a binary vector?
2. is there a fast cpp implementation to calculate TLSH already?
3. if the distance between two vectors satisfy the inequality triangle theorem, then I guess both graph index and ivf could support. But right now only faiss supported TLSH, so I guess it's wise to start from knowhere faiss implementation

[jwall149]

@yhmo Sure ofc, I'm ok with anything, zoom, google meets, etc...
I'm quite flexible on work day so please arrange some time of your spare time, for sure I can join.

[yhmo]

@yhmo Sure ofc, I'm ok with anything, zoom, google meets, etc... I'm quite flexible on work day so please arrange some time of your spare time, for sure I can join.

Please tell me your email address. So that we can send you the meeting invitation. We intend to use Zoom meeting.
This is my official email: [email protected]

[jwall149]

Please send to [email protected], thank you

[yhmo]

@jwall149 I have sent an email to you, please confirm.

[jwall149]

Replied, please confirm

[xiaofan-luan]

@liliu-z @yhmo @jwall149
thoughts?

[jwall149]

FYI: I found a similar feature request: #25701

[liliu-z]

UDF metrics is also in our radar. We can have to talk to see what we need to do next step