Extracting Topic Probabilities Per Year - Dynamic Topic Modeling

[javifern77]

Good evening!
Is there a way to extract the probabilities for each topic for each timestamp when doing dynamic topic modeling with BERTopic? I am interested in the topic probabilities as a dependent variable in a time series analysis. Any guidance would be greatly appreciated.

[MaartenGr]

That is currently not implemented since we are not recalculating probabilities at each timestamp but instead merely updating the topic representation. There is no change in the probabilities for each topic when running dynamic topic modeling.

[javifern77]

In that case, if I wanted to examine yearly changes in the probability of topic-x showing up in the corpus, would I have to run separate models for each year? In essence:

topics, probs = topic_model.get_topic_info(Year1data)
topics, probs = topic_model.get_topic_info(Year2data)
...

or would running the above merely fit Year2data to a topic model initially generated for Year1data, and thereby not return accurate probabilities of topic-x showing up in Year2data? I am also guessing that in order to ensure that data for year N is fitted into the same clusters as the year before, I would need to set a random_state to the UMAP parameter as well?

[MaartenGr]

You likely meant .fit instead of .get_topic_info. However, if you run .fit separately, you are likely getting different topics. Instead, you can run .fit once and then use manual topic modeling to fit a model for each timestamp.

[javifern77]

Awesome! I will be sure to give this a try, and reach back out if I have any remaining questions. Thank you again for your time! If I end up using this procedure in an academic paper, what would be the best way for me to reference your guidance?

[MaartenGr]

Great, let me know if you have any follow-up questions in case things do not work out as intended. Citing the paper would be great: https://github.com/MaartenGr/BERTopic#citation

[javifern77]

Understood. Thank you again!