Maintain external documents IDs

[falmanna]

Hey,

First of all, thank you for this fantastic work!

I want to maintain external document ids for multiple use cases. One of which is to get the representative docs ids instead of the actual documents. My use case is to use the most representative document title as a label for the cluster as the generated ones are not that descriptive.

What I am trying to achieve is to get the document closer to the centroid of the cluster and use its title as a label for the cluster. And from my understanding, get_representative_docs is what I should use (assuming the returned docs are sorted?), but it returns the document body.

Let me know if this makes sense.

[MaartenGr]

Thank you for the kind words! It is indeed useful to use the titles as examples of what constitutes a topic. Although it does not currently return the topic IDs, the body should be the exact same as the input. So it should be possible to do something like docs.index(my_representative_doc) to find the corresponding ID. Do note that the centroid often does not constitute the most representative document, especially when you consider that the cluster algorithm used is density based. In other words, the most representative documents that are returned are not the centroids but documents from several points of high density.

Just out of curiosity, what makes the generated labels not that descriptive?

[falmanna]

do something like docs.index(my_representative_doc) to find the corresponding ID

Sure, but it will require extra structuring and transformation of my data. I tried passing the ids in the fit_transform and attached them to the Dataframe, worked perfectly.

Just out of curiosity, what makes the generated labels not that descriptive?

So, in my case, I am clustering a set of skills attached to an occupation. These skills have a title, and a description (single or multiple sentences). I want to have a meaningful label for the generated cluster of skills that anyone can understand by reading it, but, here is an example of my outputs for a head chief occupation with 27 skills, I got the following:

======== Number topics reduction
   Topic  Count                                               Name
0     -1      7      -1_activities_trends_lead_maintenance kitchen
1      0      7     0_costs_organisation_recipes_required supplies
2      1      6  1_accordance_safety hygiene_area_visitors cust...
3      2      5     2_including_apply_cooking techniques_finishing
4      3      3              3_hospitality_address_paying_rotation
labels: ['activities', 'costs', 'accordance', 'including', 'hospitality']

I am using a CountVectorizer with ngram_range=(1, 2), stopwords, lemmatization, and diversity to generate these labels, but still they don't tell much about the skills clustered within.

Compare this to using the title of the first document returned in get_representative_docs per topic (I assumed they are sorted?):

======== Topics representations
1 comply with food safety and hygiene
0 plan menus
3 manage hospitality revenue
2 use cooking techniques

They might not be 100% represintitave of the skills inside, but they are much easier to understand.

Thanks!