Deduping topics based on LLM generated topic name

[clfkenny]

Hi, I have found that when using TextGeneration, different topics/clusters can somehow end up with the exact same name (I don't think there's an easy way to prevent this from happening within the LLM itself). In this case I would like to merge the duplicate topic names.

Would the following snippet correctly merge those topics that have the same name? Thanks!

topics = topic_model.get_topic_info()
topics['llama_gen_topic'] = topics['llama'].apply(lambda x: x[0])

# Find duplicates
dupes = {}
for idx, row in topics.iterrows():
    topic_id = row['Topic']
    topic_name = row['llama_gen_topic']

    if topic_name not in dupes:
        dupes[topic_name] = [topic_id]
    else:
        dupes[topic_name].append(topic_id)
        
dupes = {k: v for k, v in dupes.items() if len(v) > 1} # filtering out for duplicate topics

# use merge_topics to combine the topics
if dupes:
    topic_model.merge_topics(docs_list, list(dupes.values()))


# Update custom labels with the newly assigned topic
topics = merged_model.get_topic_info()
topics['llama_gen_topic'] = topics['llama'].apply(lambda x: x[0])

topic_mapping = {x['Topic'] : x['llama_gen_topic'][:50] for _, x in topics.iterrows()}
merged_model.set_topic_labels(topic_mapping)

[MaartenGr]

This seems like a solid approach to deduping topics by using the topic labels that were generated before. Did you also make sure that they are indeed duplicated topics by inspecting other representations, such as the representative documents or c-TF-IDF representations?