hierarchical_topics error : "Distance matrix cannot contain negative values"

[StephChev]

Hello Again,
I'm moving on with Bertopic dans all was running fine.
But today we have a new error as we are trying to build the hierarchical path of clusters :

The code

hierarchical_topics = r_bertopic.hierarchical_topics(tickets)
fig_cluster_hierarchy = r_bertopic.visualize_hierarchy(hierarchical_topics=hierarchical_topics, custom_labels=True)
fig_cluster_hierarchy.write_json(os.path.join(model_path, "hierarchical_topics.json"))

But we have this error fired by the first line : hierarchical_topics = r_bertopic.hierarchical_topics(tickets)
_[null-err-75] [INFO] [dku.utils] - File "/home/ml-app/DATA_DESIGN/code-envs/python/py_39_ticketing/lib/python3.9/site-packages/bertopic/_bertopic.py", line 977, in hierarchical_topics
[null-err-75] [INFO] [dku.utils] - X = validate_distance_matrix(X, embeddings.shape[0])
[null-err-75] [INFO] [dku.utils] - File "/home/ml-app/DATA_DESIGN/code-envs/python/py_39_ticketing/lib/python3.9/site-packages/bertopic/utils.py", line 147, in validate_distance_matrix
[null-err-75] [INFO] [dku.utils] - raise ValueError("Distance matrix cannot contain negative values.")
[null-err-75] [INFO] [dku.utils] - ValueError: Distance matrix cannot contain negative values.

I search the web and didn't found any real help.
So my here are some questions :

• Why could we have some negative distance between clusters ? (are they due to approximation ?)
• Can we force distance to have only absolute values ? or if it is approximation can we force negative values to be 0.0 ?
• if yes, how can we do that ?
• if no... what could we workaround this issue and run hierarchical_topics (even if it is a little false, or approx ?)

Thnaks a lot for all your answers
BR
Stéphane

[MaartenGr]

Thank you for sharing this. Since this seems to be an issue (you got an error) I believe this should be posted as an issue instead as there is some information missing that helps me understand the issue (such as reproducible example, BERTopic version, etc.). More specifically, I think this was fixed in the latest version but I'm not sure if that was included in your particular version.

[StephChev]

Hello @MaartenGr and first thnaks a lot for all your answers.
You're right I should have posted it as an issue... my version is 0.16.0. I will check if this has been corrected.
I'm just afraid of moving to last version as I have a lot of 0.16.0 pickle models that have been already generated and are used to transform points... Do you think this should lead to issues ?

[StephChev]

Hello, I solved the issue but that was not so simple as using the last Bertoptic version 0.16.3 was leading to many compatibility issues (

• huggingface_hub: snapshot_download() got an unexpected keyword argument 'ignore_files'",
• numba : _rebuild() got an unexpected keyword argument 'impl_kind', ...)
• • ...
    After many test/tries, here are libraries version I had to used with bertopic 0.16.3 (perhaps if it is already avaiable someweher else please forgive me)
    bertopic==0.16.3
    transformers>=4.11.3
    sentence-transformers==2.2.2
    huggingface_hub==0.10.0
    scikit-learn>=0.24.2
    numpy>=1.19.5
    hdbscan>=0.8.29
    umap-learn>=0.5.1
    numba==0.58.1

Hope this will help
BR
Stephane

[MaartenGr]

Glad to hear that the issue was solved. Note that using pickle requires you to pin the exact environment (including python and all dependencies) in order to properly load the pickled file. That's also why safetensors and pytorch were introduced.

[StephChev]

ah...
Do You mean that existing pickles generated with 0.16.0 and reloaded with 0.16.3 will not work properly ?
And do you suggest that one solution is to load old 0.16.0 with "previous environement" and then save them as safetensors ? then we will be able to reload them with 0.16.3 or future versions ?
One last point, if we use the save as safetensors, and reload them into bertopics, what will be the functions that could work differently or not work at all ? we are mainly using transform, fittransform and hierarchical-topics ?

Thanks a lot for your help
BR
Stephane

[MaartenGr]

Do You mean that existing pickles generated with 0.16.0 and reloaded with 0.16.3 will not work properly ?

Not necessarily. I mean that when you pickle something, you are (almost) getting an exact copy of what you are saving. As such, if you change your numpy version (and therefore how numpy might work) the resulting model could break. This is inherent to anything that you use pickle on, not just BERTopic. You can find a bit more about that in the scikit-learn documentation.

It is also the reason why I am so adamant about users using safetensors since there is a much bigger degree of reproducibility involved.

One last point, if we use the save as safetensors, and reload them into bertopics, what will be the functions that could work differently or not work at all ? we are mainly using transform, fittransform and hierarchical-topics ?

They will work but transform will work differently as no UMAP or HDBSCAN models are being used (it uses the cosine similarity between document and topic embeddings instead). Also note that you should not use fit_transform after loading the model since that will completely overwrite the model.