Keyphrase extraction meant to extract meaningful attributes from fashion products. Python3 based. What implemented so far:
- Graph based methods
- Embedding based methods
TODO:
- Remove meaningless but frequent words and rebuild the LDA model
- Maybe try some DL supervised methods
pip3 install -r requirements.txtpython -m nltk.downloader stopwords
python -m nltk.download('averaged_perceptron_tagger')To enable the use of Embedding-based method, follow innstructions here
to download Sent2vec model and store it in the 'EmbeddingRank' folder. Name it as 's2v.bin' for example.
Since I have already trained with LDA, I stored the word-to-topic distribution, document-to-topic distribution and feauture_names as pickle file inside LDA folder. Feel free to train with your own corpus using lda.py and replace them with your trained distributions.
After download the whole repo and named it as kkExtract:
# Example of single-tpr extraction
import kkExtract
# tpr_rank = kkExtract.graph_methods.singleTPR(doc_index = 12, str_input=test_str) # you need to specify the document index in the corpus, build from string
tpr_rank = singleTPR(doc_index = 12, input_file=path/to/your/txt_file) # build from txt file
tpr_rank.weight_node()
res = tpr_rank.get_top_phrases(k=6)
print(res)# Example of EmbedRank extraction
path_to_sen2vecmodel = 's2v.bin'
emb_rank = EmbedRank(k=5, embedModel = path_to_sen2vecmodel)
emb_rank.load_text(test_str)
res = emb_rank.topK_phrase()Detailed examples are provided in the end of each methods python file
Currently implements the following keyphrase extraction models:
- Graph-based models
- SingleRank [documentation, article by (Wan and Xiao, 2008)]
- Single Topical PageRank [documentation]
- Emdedding-based models
- EmbedRank [documentation ]