When to perform outlier reduction? - Merged Models

[NullPxl]

Hi! I was wondering if anyone had insight about when would be best to perform outlier reduction in an incremental learning pipeline:

Option 1.

• train and save (as safetensors) model1, model2, model3 (note that I have all docs and embeddings saved for each as well)
• load each model in, and merge together merged_model = BERTopic.merge_models([model1, model2, model3], embedding_model=...)
• reduce outliers on merged_model, update_topics with new_topics
• save merged model
• new model comes in: merge it with the merged_model, reduce outliers on the new total merged_model, update_topics, save

Option 2.

• train and save model1, model2, model3
• load each model in, use reduce_outliers on each individual model, and then merge together, update_topics -> save merged_model
• new model comes in: reduce outliers, and then merge it with the merged_model, update_topics and save

In either case, I like having the original models (without any outlier reduction applied) saved. Option 2 makes most sense to me (and might be more accurate?), but it would be great to get any opinions. Thanks!

[NullPxl]

Also, will saving a model where outliers were reduced and topics updated with update_topics(docs, topics=new_topics) lead to issues if I then want to merge additional models onto it later (my use-case above)? I'm unsure if the logger warning about "Make sure that manually assigning topics is the last step in the pipeline" in update_topics is applicable here.

[MaartenGr]

I indeed think that option 2 will be a bit more accurate when reducing outliers since it is closer to the source topics/topic embeddings. That said, it wouldn't hurt trying out to see what kinds of results you get. Merging models is quite fast.

Also, will saving a model where outliers were reduced and topics updated with update_topics(docs, topics=new_topics) lead to issues if I then want to merge additional models onto it later (my use-case above)? I'm unsure if the logger warning about "Make sure that manually assigning topics is the last step in the pipeline" in update_topics is applicable here.

Good question, I'm not entirely sure as I haven't used it in this particular use case before. You could try it out by merging models 1 and 2, update topics and then merge it with model 3. This will give you a quick indication if everything works alright.