Stop words still in top_n_words of topics

[DominikMann]

Hi,

I am removing english and custom stop words with the CountVectorizer:

from sklearn.feature_extraction.text import CountVectorizer
from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))

custom_sw = set(['mathrsfs', 'amsmath', 'shrink', 'citation'\
             'available citation', 'citation abstract', \
             'citation abstract', 'abstract available', \
             'citation analysis', 'doi 10', '53 doi', \
             '1093 analys', 'australasian journal', \
             'volume', 'journal'])

stop_words.update(custom_sw)

vectorizer_model = CountVectorizer(ngram_range=(1, 2), stop_words=stop_words)

I am giving the vectorizer_model to the BERTopic model but still when inspecting the topics there are some with custom stop words as top_n_words. Is there a way to remove them (without the need to pre process the dataset by myself)?

[MaartenGr]

Could you also share your code for instantiating the BERTopic model? Also, could you share some of the topics themselves that still have those custom stop words? After having trained your BERTopic model, you can inspect the vectorizer to see if the stopwords were properly added to the model with:

topic_model.vectorizer_model.stop_words_

[MaartenGr]

Ah, I see what the problem is. The stop_words are removed before n-grams are created. This means that stopwords with an n-gram of 1 will get removed but higher n-grams will not. Instead, you will have to create your own custom analyzer or tokenizer and remove the n-gram stopwords there. After some googling, I found an example of doing this here.

[DominikMann]

That custom vectorizer worked like a charm! Thank you :-)

[MaartenGr]

Great! Glad to hear that it worked out.

[DominikMann]

Hi Maarten,

sorry for the late reopining of this topic but it seems that the CustomVectorizer did in fact not remove the custom stop ngrams.

This is the whole process of removing the stop words + ngrams:

class CustomVectorizer(CountVectorizer): 
       
    stop_grams = []    
    
    def __init__(self, stop_grams = [], **opts):
        self.stop_grams = stop_grams
        super().__init__(**opts)
    
    def remove_ngrams(self, doc):
        for stop_gram in self.stop_grams:
            doc = doc.replace(stop_gram, "")
        return doc
    
    # overwrite the build_analyzer method, allowing one to
    # create a custom analyzer for the vectorizer
    def build_analyzer(self):
        
        # load stop words using CountVectorizer's built in method
        stop_words = list(self.get_stop_words())
        
        preprocessor = self.build_preprocessor()
        tokenizer = self.build_tokenizer()
        remove_ngrams = self.remove_ngrams
        
        
        # create the analyzer that will be returned by this method
        def analyser(doc):
                
            # apply the preprocessing and tokenzation steps
            doc_clean = preprocessor(doc.lower())
            
            # remove phrase stopwords
            doc_clean = remove_ngrams(doc)
            
            # tokenize using default tokenizer
            tokens = tokenizer(doc_clean)            
            
            # use CountVectorizer's _word_ngrams built in method
            # to remove stop words and extract n-grams
            return(self._word_ngrams(tokens, stop_words))
        
        return(analyser)

import nltk
from sklearn.feature_extraction.text import CountVectorizer

nltk.download('stopwords')

stop_words = stopwords.words('english')

custom_sw = ['mathrsfs', 'amsmath', 'shrink', 'citation', 
             'volume', 'journal', 'Issue', 'Volume',
             'EarlyView', 'The', 'This', 'We', 'In', 'Journal', 'Philosophical', 'Studies', 'JW', 
             'Page', 'Australasian', '90', '2012', 'March', 'article', 'abstract', '737', 'any055',
             'analys', '10', 'Analysis', 'Wittgensteins', 'Tractatus', 'No'] #stop words + noise in abstracts

stop_words.extend(custom_sw)

stop_grams = ['available citation', 'citation abstract',
              'No available', 
              'citation abstract', 'abstract available',
              'citation analysis', 'doi 10', '53 doi',
              'usa philosophical', '1093 analys', 'australasian journal',
              'doi 10', 'research 104', '2022 philosophy',
              'australasian philosophy', 'philosophy 89',
              'philosophy 90', 'Philosophical Studies', '1573 0883', 'Australasian Journal', 
              'Philosophy 89', 'Philosophy 90', 'original publication', 'review article', 'Analysis 78', 'analys any055',
              'article abstract', 'citation Analysis', '1093 analys', '10 1093', 'Book review', 
              'Phenomenological Research', 'Philosophy Phenomenological', 'Research Philosophy', '2022 Philosophy',
              'phenomenological research', 'Page 676', 'available citation', 'No available', 
              'citation No', 'citation Book', 'Analysis 78', '1093 analys', 'Type Article', 'philosophy and phenomenological research',
              'Philosophy and Phenomenological Research', 'Philosophy 104', '2011 Philosophy']


vectorizer_model = CustomVectorizer(ngram_range=(1, 2), stop_words=list(stop_words), stop_grams = stop_grams)

when calling topic_model.vectorizer_model.stop_grams or topic_model.vectorizer_model.stop_words the correct words/ngrams are given back. However, when fitting the model:

model_bert = BERTopic(
    umap_model=umap_model,
    hdbscan_model=hdbscan_model,
    embedding_model=embedding_model,
    vectorizer_model=vectorizer_model,
    top_n_words=15,
    language='english',
    calculate_probabilities=True,
    verbose=True,
    diversity=0.35
)

I still get topics like:

79: [('Philosophy', 0.36217383674230064),
  ('2011 Philosophy', 0.29000327355819217),
  ('Philosophy 89', 0.2881971012343228),
  ('2011', 0.21669426757421156),
  ('September', 0.1623705781225231),
  ('June', 0.1505839502958349),
  ('December', 0.13552623744509845),
  ('September Philosophy', 0.12040718824325394),
  ('September 2011', 0.07959732570520407),
  ('377 June', 0.035225923358158856),
  ('608', 0.035225923358158856),
  ('Philosophy 205', 0.035225923358158856),
  ('Philosophy 616', 0.035225923358158856),
  ('June Philosophy', 0.035225923358158856),
  ('Philosophy 92', 0.033727111016124064)],

or

 76: [('Philosophy 104', 0.07424551349867395),
  ('2022 Philosophy', 0.07188484351729392),
  ('Philosophy', 0.06123363450601498),
  ('acts', 0.03222349656180911),
  ('beyond call', 0.02585958270726494),
  ('optionality', 0.02299810160468259),
  ('supererogatory acts', 0.02299810160468259),
  ('moral', 0.02218401188759087),
  ('duty', 0.021460520994500815),
  ('beyond', 0.021344214637107513),
  ('Philosophy 99', 0.01971265851829936),
  ('paradox', 0.01948416802366299),
  ('permissible', 0.018320726242470248),
  ('comparative cost', 0.01709493974057777),
  ('2019 Philosophy', 0.01709493974057777)],

where stop grams like "2022 Philosophy" should be removed.

[MaartenGr]

Without seeing the documents themselves, I cannot be completely sure but I believe this may the result of one of two things:

First, it seems that the code is somewhat case-sensitive and that it does take into account whether words are capitalized or not which might affect the result. Second, when you have the term 2022 Philosophy and there exists a stopword between those words, like 2022 in Philosophy it will not remove that stopgram but first remove the "in" and it will not check for stopgrams afterwards. You will have to do some additional checks afterward it seems.

To illustrate this, the following code shows that the stopgram is not removed:

from sklearn.feature_extraction.text import CountVectorizer, ENGLISH_STOP_WORDS
import pandas as pd
text_input = [
    "2022 in Philosophy",
]

df = pd.DataFrame(text_input, columns=["text"])

stop_grams = ["Fox news", "mother jones", '2022 Philosophy']

vectorizer = CustomVectorizer(ngram_range=(1, 2), stop_words="english", stop_grams = stop_grams)
X = vectorizer.fit_transform(df['text'])

text = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names())
vectorizer.get_feature_names()

However, when the text input is ["2022 Philosophy"] it will be removed. So you will have to adjust the code a bit I believe to remove those at the right stop.