Getting different results each run even though I am setting the random seed in UMAP model

[GeorgeDittmar]

Are there other aspects of the framework that are somewhat stochastic in nature? I set the random seed in UMAP to hopefully keep runs the same but every time I run BERTopic over my data, the number of clusters returning changes each run. Anyone else notice this or know of another seed thats needed to be set?

[MaartenGr]

If you are in the same environment each time, then there should not be any other seeds necessary. For me at least, when I set random_state in UMAP, the output remains exactly the same. Could you share your exact code for running into this issue?

[GeorgeDittmar]

Its weird my issue has gone away randomly now after a few notebook restarts so I dunno whats going on now. Still investigating myself.

[MaartenGr]

Could you share also the import that you have as well as the versions of BERTopic, including its dependencies?

[drob-xx]

The way it is supposed to work is that there is no variation as long as you use the same random_state and are running on the same architecture (OS, chips etc. etc. may not return the same results.) There is a bug in the cuML UMAP implementation and random_state has to be coupled with init='random' (which has other implications) for the results to be the same. There should be no doubt - when we say "the same" it means exactly the same. I just compare the embeddings so you should be able to just compare embeddings:

np.all(bertopic_model_1.umap_model.embedding_ == bertopic_model_2.umap_model.embedding_)

This will be True if the UMAP fit was the same across multiple runs.

[jchen04]

@MaartenGr These are my imports:

from bertopic import BERTopic
from sentence_transformers import SentenceTransformer
from hdbscan import HDBSCAN
from bertopic.vectorizers import ClassTfidfTransformer
from umap import UMAP

And these are the versions:
BERTopic v0.13.0
hdbscan v 0.8.29
numpy v1.21.4
pandas v1.3.4
plotly v5.11.0
pyyaml v5.4.1
scikit-learn v1.2.0
sentence-transformers v2.2.2
tqdm v4.64.1
umap-learn v0.5.3

I'm running in a jupyter notebook on a windows 10 machine and not using cuML UMAP.

@drob-xx np.all(topic_model1.umap_model.embedding_ == topic_model2.umap_model.embedding_) is returning False for me.

Just to be extra safe, I did another test where I re-initialized all the objects in each run, as well as copied my input data and embeddings into new objects (I didn't see these being changed by the model fit, but just in case...)

df = df_base.copy()
umap_model = UMAP(random_state=42)
hdbscan_model = HDBSCAN(min_cluster_size=50)
embeddings = embeddings_base.copy()
ctfidf_model = ClassTfidfTransformer(reduce_frequent_words=True)
np.random.seed(100)
topic_model = BERTopic(calculate_probabilities=False,
                        nr_topics='auto',
                        ctfidf_model=ctfidf_model,
                        seed_topic_list=seed_topic_list,
                        umap_model=umap_model,
                        min_topic_size=100,
                       )
topics, probs = topic_model.fit_transform(df['text'].values,
                                           embeddings
                                           )

I sincerely hope this is user error somehow on my part!

[drob-xx]

Not sure what is going on. I'm sure @MaartenGr will see more than I do. However, I think you are making this more complicated than it needs to be - also your approach is going to use different default parameters than BERTopic uses out of the box. I would make a more straight forward test case e.g.

• Create two different BERTopic models
• Run them with the umap_model random_state set the same
• compare them - they should be the same - something like:

my_docs = df_base['My_Text_Column']

bt1 = BERTopic()
bt1.umap_model.random_state = 42
bt1.hdscan_model.min_cluster_size = 50
bt1.fit_transform(my_docs)

bt2 = BERTopic()
bt2.umap_model.random_state = 42
bt2.hdscan_model.min_cluster_size = 50
bt2.fit_transform(my_docs)

assert(bt1.umap_model.embedding_ == bt2.umap_model.embedding_)

All the output from both models, the labels, the topic model vocabularies etc. should be exactly the same.