Better Topic Modeling Visualization/Saving for Image Only Analysis

[nabsiddiqui]

Feature request

I would like to see a swipe gallery of each topic. Each image would contain the top representative images of that topic in a grid (similar to what is shown at the end of the process described here: https://maartengr.github.io/BERTopic/getting_started/multimodal/multimodal.html#images-only) with the captions underneath. I would like the option to create the grid with the original images since the images right now are low quality in the grid. If creating a swipe gallery is too much, something that just saves the topics in a folder as 1_keyword1_keyword2_keyword3_keyword_4.jpg also works.

Motivation

Currently, at the end of the image modeling process (https://maartengr.github.io/BERTopic/getting_started/multimodal/multimodal.html#images-only), there is a grid that shows the list of keywords and the grid. This works well for exploratory analysis but is hard to display later. Right now, there is no way to display the results in a better way. In addition, even saving the grid image is inefficient because it is generated from the compressed images rather than the original.

Your contribution

I have created a save captions and grid function I could add, but I don't think it will help with the larger issue of not being able to utilize the original images to create a grid as I don't have the original dataset.

[MaartenGr]

Thank you for the request. If I'm not mistaken, you would like to see for each individual image to which topics they belong, correct? If so, I'm not sure what the best way is to implement that with the current set of dependencies. I think a 2D chart with images would be great, but it seems that I was wrong for choosing plotly here since it also requires dash to showcase the images.

Saving images could be a possibility but wouldn't it merely be reproducing the input images but with the associated keywords? It indeed seems a bit redundant.

[nabsiddiqui]

It depends on how much you want to focus on the image-only implementation of BERTopic. I looked around, and there isn't anything similar in terms of using only the representative images as inputs to be captioned. Other implementations require all the images to be captioned, and I think this is a good unique approach that can be implemented in fields like art history, online image culture, etc.

In regards to your question about reproducing the input images, redundancy would only occur if you create a grid of images with low-res versions of the images. Currently, that is what the code in the example does. This makes it hard to see some images. For instance, I am looking at cigarette cards for research, with text on them. They do not display well without the original images. In the easiest example, you would create the grid with the representative images but in the original resolution.

But, yes, I think the best idea would probably just be to implement the images as a 2d graph. Are you opposed to creating a visualization that is based on HTML or CSS? A couple of options:

• Create a basic image gallery of each topic in Python with captions underneath: https://github.com/haltakov/simple-photo-gallery
• Create something like pixPlot (https://dhlab.yale.edu/projects/pixplot/) or Collection Space navigator (https://collection-space-navigator.github.io/)
• Basic HTML or CSS output of image gallery. Various interesting effects to make it more visually appealing here: https://freefrontend.com/css-gallery/
• Provide additional fine-tuning opportunities and create something like imgs.ai: https://imgs.ai/interface

[MaartenGr]

Thank you for the suggestions! I would love to keep using the set of dependencies that are already here as that allows for minimum overhead.

Are you opposed to creating a visualization that is based on HTML or CSS?

That also means no custom HTML/CSS, as maintaining that would be a lot of work next to the main code.

But, yes, I think the best idea would probably just be to implement the images as a 2d graph.

This would then be similar to .visualize_documents where the input embeddings are reduced to 2D so that they can be plotted. The visualization technique, however, is a bit more difficult without additional packages. Plotly, unfortunately, requires dash to show the images. I believe altair would be a nice alternative but I'm a bit hesitant switching plotting backends.

Based on your suggested options, most of them are not maintained regularly and are quite small. Wouldn't more popular/stable packages be preferred for this kind of functionality?

[nabsiddiqui]

Hey Martin,

Thank you for the response. Unfortunately, the fact that they are poorly maintained is probably the reason for trying to come up with a different backend. I think because of the lack of options, these are the ones that people use that do need to do this kind of work.

One option may be the animation slider introduced in plotly 4.14: https://plotly.com/python/imshow/. The topics would be at the bottom and the images would be the display of the representative documents. I don't know how much better that is than the Grid, but still may be a useful option and doesn't look too difficult to implement.

Other than that, without having an HTML/CSS output, I think you are stuck with either needing to require dash or switching to altair.

[MaartenGr]

Oh, the animation slider is a good one! That would resolve a bunch of issues and allows you to iteratively go through each topic and show the individual representative images. That one should indeed be doable. I'll mark this issue as a good first issue for someone to work on. I have limited time at the moment but I do think this would be cool to implement for those interested.

[nabsiddiqui]

Sounds good. Thank you. I’m a little packed with other research right now, but it seems like it shouldn’t be too difficult. When I present on my research at a conference or something, I will see if I can do it and will send a pull request. Of course, if you or someone else wants to work on it, let me know.

I’ll also try to think of a way to update the current tutorial on images only to make it more clear. Quick question for updating the tutorial if you are interested and then you can probably close this discussion, is it possible to also use maximum marginal relevance on the images themselves or is BERT only able to do this with the descriptions later? Right now, I’m getting a lot of topics with similar images (all the backs of cigarette cards, for instance)

[MaartenGr]

Quick question for updating the tutorial if you are interested and then you can probably close this discussion, is it possible to also use maximum marginal relevance on the images themselves or is BERT only able to do this with the descriptions later?

I believe MMR currently only affects the images and indeed not the images themselves. It should technically be possible but would require some additional code in the MMR class I think.