How to fold back in new topics after redistributing outliers

[spookyuser]

Hi! First off I looove this library and can't believe it's something I can just download for free!

My question is this, I followed this line from the faq about reducing the number of outlier topics using the following line:

probability_threshold = 0.01
new_topics = [np.argmax(prob) if max(prob) >= probability_threshold else -1 for prob in probs]

This works perfectly, however after doing this the order of the topics is no longer correct because after adding in the outliers the frequency of each topic changes significantly, and the frequency of words in each topic is no longer correct because after the outliers are added the frequencies can change a lot, this also affects the bar chart because it relies on selecting the n most frequent words per topic.

So I was just wondering if there was a way to tell the model about the new_topics such that generate_topic_labels would work as expected and same for the ordering of topics and the bar chart.

What i've tried but doesn't seem to work:

  df2 = pd.DataFrame({"Topic": new_topics, "Document": data})
  model._update_topic_size(df2)
  model._sort_mappings_by_frequency(df2)

Would appreciate any insight! Thanks :)

[MaartenGr]

Although it is possible to add new_topics back into the model, there are a number of things that you should be careful of. First, when you assign these outliers to different topics it is difficult for the model to track which topics should be mapped to one another. For example, if you have two outliers but one is assigned to topic 1 and the other to topic 5 there is no fixed way to map one topic to another. Second, after updating the topics with your method, it is not possible to further merge topics since no clean mapping between topics will then exist. This means that updating topics that way might prevent other features from working. Third, it is necessary that all unique topics in topics can also be found in new_topics.

Having said that, as of version v0.12, you can use .update_topics(docs, new_topics) to update the internal representation to achieve the counts that you were looking for.

[spookyuser]

For future googlers this is the full working snippet I used :)
Note that you need to call a few methods after update_topics for it to be ordered correctly, more info here

    topics, probs = topic_model.fit_transform(documents=documents)
    probability_threshold = 0.01
    new_topics = [
        np.argmax(prob) if max(prob) >= probability_threshold else -1 for prob in probs
    ]
    topic_model.update_topics(documents, new_topics)

    #  Update topic frequencies
    documents = pd.DataFrame({"Document": documents, "Topic": new_topics})
    topic_model._update_topic_size(documents)
    documents = topic_model._sort_mappings_by_frequency(documents)

[MaartenGr]

Generally, following the FAQ with respect to HDBSCAN parameters for reducing outliers works best. Other than that, it might be worthwhile to explore the number of topics with HDBSCAN but then model that with an algorithm like k-Means that does not model outliers.

[spookyuser]

Thanks! I'll try that, to clarify, when you say explore the number of topics with HDBSCAN do you mean using HDBSCAN to do an initial clustering, see how many topics it makes, then ask Kmeans clustering to cluster using the number of topics HDBSCAN created?

[MaartenGr]

Yes, HDBSCAN can give you a rough idea of the number of topics in your data. Using those values, you can then indeed use k-Means based on that number of topics.

[spookyuser]

Amazing thanks so much for your help!