Review of a BERTopic Model

[foesine]

Hey!

We are two master students (Dominik and Emma) from Germany and we used BERTopic on a Uni Project. We want to publish a paper and together with our project partner we are looking for some feedback on the model. We thought since you are the best to ask and it doesn't hurt to ask, we might as well try. Would you like to take a look at or model and give us some feedback?
I saw that you are also looking for use cases so maybe you would profit from it as well.
For the context: We are using it for abstracts of philosophical papers published in highly ranked journals.
Thanks so much!

[MaartenGr]

Sure, feel free to share your model here or via any of my other personal social media platforms. I can't promise that I can spend hours looking at everything but I'll definitely have a look.

[DominikMann]

Hi Maarten,

Thats amazing, thank you very much! I have uploaded the notebook for the topic model here: https://github.com/DominikMann/BertTopic-Philosophy-Topic-Modeling

Please dont look into the pre-processing functions, as we have just spent hours figuring out how to do it somehow. ^^
I contacted you once about the stop words and the CountVectorizer and you suggested to use the custom one. We implemented it but don't really think that it words but we just settled with merging all trash categories after fitting the model because it's not that bad.
We also wanted some criterion to argue why we chose the particular parameters for UMAP and HDBSCAN (because we need to write a term paper) so we random searched the parameters with the best DBCV value of HDBSCAN. Thats the part where we are not really sure if this is acceptable or not but for us, all the topics looked pretty good at different parameters, so we needed some argumentation why we chose these ones. Maybe you can give us your opinion on that procedure.
We also stole some code of you for calculating the coherence scores (at the end of the notebook). :) As there is some confusion about the values of the coherence score (for example: https://stackoverflow.com/questions/62032372/coherence-score-u-mass-18-is-good-or-bad?) do you know if coherence score values have any real meaning? For example, our topic model gets a u_mass score of -1, a LDA model with the same topic numbers gets a u_mass score of -8.4. As far as I can tell from the look on the formulas of the papers to this scores, a higher score means a better model but is -1 even good or just better than LDA but still pretty bad?

That were some long question and you don't need to answer them in detail or at all. We are just happy if you can give us some feedback on our model and everything else is just a bonus!

[MaartenGr]

We also wanted some criterion to argue why we chose the particular parameters for UMAP and HDBSCAN (because we need to write a term paper) so we random searched the parameters with the best DBCV value of HDBSCAN. Thats the part where we are not really sure if this is acceptable or not but for us, all the topics looked pretty good at different parameters, so we needed some argumentation why we chose these ones. Maybe you can give us your opinion on that procedure.

There are quite a number of ways that you can evaluate BERTopic, from topic coherence and diversity scores, to cluster and predictive evaluation metrics. This becomes more apparent when you add in the somewhat subjective nature of topic modeling. When do we have captured all topics? What is exactly a topic? Can we universally agree on what this document is about? Especially when stakeholders are involved, this becomes more and more complex.

The solution to an evaluation in general, in my opinion, is to start from the beginning. Why are we doing this analysis? What are the end goals? What are we trying to achieve with this model? For whom or what is it being developed? In your case, you mention "because we need to write a term paper". Although that is technically correct, there is a fundamental reason why you are performing your topic modeling. That reason, which is often the reason for developing the model/paper, can then be served to decide which evaluation metric you are going to use. For example, if you are writing a paper to show that BERTopic is horrible at making coherent topics, then coherence metrics might serve here. In contrast, if you want to create the best topic model, then you need to ask yourself what constitutes "best" in your paper/framework/philosophical underpinnings.

We also stole some code of you for calculating the coherence scores (at the end of the notebook). :) As there is some confusion about the values of the coherence score (for example: https://stackoverflow.com/questions/62032372/coherence-score-u-mass-18-is-good-or-bad?) do you know if coherence score values have any real meaning? For example, our topic model gets a u_mass score of -1, a LDA model with the same topic numbers gets a u_mass score of -8.4. As far as I can tell from the look on the formulas of the papers to this scores, a higher score means a better model but is -1 even good or just better than LDA but still pretty bad?

Unfortunately, there isn't necessarily a cut-off for when the value that you get back is "good" as the underlying formulas do not really allow for that. Who is to say when a u_mass score suffices or not? These kinds of coherence metrics are typically used for comparisons between models to show that one is "better" than another. The difficulty here is that these measures are an approximation and in practice, even academic research, typically more is needed, like diversity metrics or most preferably, human evaluation. The latter is quite difficult and resource intensive, so that is not always done. Personally, especially if you are not that familiar with coherence metrics, I would start with "c_v" which has been researched quite extensively and has been the most popular one in research. The latter is not necessarily the best reason for choosing it but if you are selecting something else, then some readers might want an explanation for that.

In practice, whenever you are choosing evaluation metrics, I would advise taking all of the above into account, arguing for why these metrics are relevant to your use case, and also being transparent in their shortcomings as they almost always have some.

[DominikMann]

Thank you for your insights! That are some nice questions that we did not consider fully yet!

About the coherence score, it is what I have thought that there is no fixed "good" values. Our coherence scores are all higher for the Bertopic model than the LDA model so it is kinda obvious that the bertopic is better. We also calculated topic diversity (Proportion of unique words) where LDA actually scored better but this is due to alot of "random" words in the LDA model so its not very meaningful. Actually, we have a source of human evaluation, which is a domain expert so atleast for model evaluation we would have everything covered you mentioned.