Representative documents when using the MultimodalBackend

[eli-quereli]

Hey, thank you so much for the great work on BERTopic!

is it possible to extract the representative docs when not passing any docs, but only images to the model? I am using the MultiModalBackend, training topic models with images-only, and with text-image data.

get_representative_documents works in a text-only setting, but not for the images-only and text-images runs.

Thank you very much in advance.

[MaartenGr]

Thank you for the kind words! I believe if you use the bertopic.representation.VisualRepresentation representation model it should generate the aggregated image visualization that you see here. Extracting representative images requires a different method from extracting representative texts since the latter are calculated through c-TF-IDF and the form should be calculated using cosine similarity.

That said, it should be possible to do this yourself by calculating the cosine similarity between image embeddings and topic embeddings.

[eli-quereli]

Thank you for the quick response!

I am still working on understanding the differences between the text-based topic modeling and the image-based setting.
When I only pass images, I use the KeyBERTInspired Representation Model with the Image2Text Model nlpconnect/vit-gpt2-image-captioning.

Aren't these generated image captions used at some point to calculate the c-TF-IDF?
I am just reading the code for the fit_transform method in bertopic.py and there I came across this:

 # Create documents from images if we have images only
        if documents.Document.values[0] is None:
            custom_documents = self._images_to_text(documents, embeddings)

            # Extract topics by calculating c-TF-IDF
            self._extract_topics(custom_documents, embeddings=embeddings)
            self._create_topic_vectors(documents=documents, embeddings=embeddings)

            # Reduce topics
            if self.nr_topics:
                custom_documents = self._reduce_topics(custom_documents)

            # Save the top 3 most representative documents per topic
            self._save_representative_docs(custom_documents)
```

[MaartenGr]

Ah right, it seems that the captions of the images are saved as "representative documents" but not the images themselves. If I'm not mistaken, then .get_representative_docs should return those captions.

[eli-quereli]

Thank you for the reply! Unfortunately I was not able to extract the captions by calling topic_model.get_representative_docs( ) - this only returns an empty dictionary. I would like to be able to reproduce the selection of images for the VisualAspect you mentioned in your first reply. Is there any way to access the representative_images directly?