Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sahyagiri/distinctkeywords
semantically distinct key phrase extraction using hilbert hashes.
https://github.com/sahyagiri/distinctkeywords
french-keyword-extraction german-keyword-extraction hilbert-curve keybert keyphrase-extraction keyphrase-extraction-algorithm keyphrase-generation keyword-detection keyword-extraction keyword-extractor keywords-extraction nlp python trie
Last synced: 12 days ago
JSON representation
semantically distinct key phrase extraction using hilbert hashes.
- Host: GitHub
- URL: https://github.com/sahyagiri/distinctkeywords
- Owner: sahyagiri
- License: mit
- Created: 2022-02-19T09:32:33.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2022-02-28T09:45:42.000Z (almost 3 years ago)
- Last Synced: 2024-06-03T14:10:21.468Z (7 months ago)
- Topics: french-keyword-extraction, german-keyword-extraction, hilbert-curve, keybert, keyphrase-extraction, keyphrase-extraction-algorithm, keyphrase-generation, keyword-detection, keyword-extraction, keyword-extractor, keywords-extraction, nlp, python, trie
- Language: Jupyter Notebook
- Homepage: https://dossiers.page/semantically-distinct-key-phrase-extraction/
- Size: 72 MB
- Stars: 46
- Watchers: 3
- Forks: 10
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://pepy.tech/project/distinct-keywords)
[](https://pypi.org/project/distinct-keywords)
[](https://pypi.org/project/distinct-keywords)
[](https://pypi.org/project/distinct-keywords)
[](https://pypi.org/project/distinct-keywords)
# DistinctKeywords
This is a utility function to extract semantically distinct keywords. This is an unsupervised method based on word2vec. Current implementation used a word2vec model trained in simplewiki(for English). Other language models and their sources are given below.
Please visit the [blog post](https://dossiers.page/semantically-distinct-key-phrase-extraction/) for more details
### Supported Languages (jupyter notebooks available in examples). Please see the word2vec citation to know the models used
1. English (default) using custom word2vec trained on simplewiki.
2. German (on test. Need support from native speakers).
3. French (on test. Need support from native speakers).
4. Italian (on test. Need support from native speakers).
5. Portuguese (on test. Need support from native speakers).
6. Spanish (on test. Need support from native speakers).
## Installation Instructions
1. conda create -n keyphrases python=3.8 --no-default-packages
2. conda activate keyphrases
3. pip install distinct-keywords
4. python -m spacy download en_core_web_sm
5. conda install --channel=conda-forge nb_conda_kernels jupyter
6. Optional multi-lingual support
```
import nltk
nltk.download('omw')
```
7. jupyter notebook
## Getting started
1. Clone the repository
2. Open the examples folder in jupyter notebook. The sub-folders contain the respective language files.
3. Select the language you wanted to try out
## Usage
### You need to have the respective language files in current directory. Please visit examples folder to download and to know how to use them in parameter.
```
from distinct_keywords.keywords import DistinctKeywords
doc = """
Supervised learning is the machine learning task of learning a function that
maps an input to an output based on example input-output pairs. It infers a
function from labeled training data consisting of a set of training examples.
In supervised learning, each example is a pair consisting of an input object
(typically a vector) and a desired output value (also called the supervisory signal).
A supervised learning algorithm analyzes the training data and produces an inferred function,
which can be used for mapping new examples. An optimal scenario will allow for the
algorithm to correctly determine the class labels for unseen instances. This requires
the learning algorithm to generalize from the training data to unseen situations in a
'reasonable' way (see inductive bias).
"""
distinct_keywords=DistinctKeywords()
distinct_keywords.get_keywords(doc)
```
## Output
['machine learning',
'pairs',
'mapping',
'vector',
'typically',
'supervised',
'bias',
'supervisory',
'task',
'algorithm',
'unseen',
'training']
## Methodology
After creating word2vec, the words are mapped to a hilbert space and the results are stored in a key-value pair (every word has a hilbert hash). Now for a new document, the words and phrases are cleaned, hashed using the dictionary. One word from each different prefix is then selected using wordnet ranking from NLTK (rare words are prioritized). The implementation of grouping and look up is made fast using Trie and SortedDict
## Benchmarks
Currently this is tested against KPTimes test dataset (20000 articles). A recall score of 31% is achieved when compared to the manual keywords given in the dataset.
Steps to arrive at the score:
1. Used both algorithms. Keybert was ran with additional parameter top_n=16 as the length of dstinct_keywords at 75% level was around 15.
2. Results of algorithms and original keywords were cleaned (lower case, space removal, character removal, but no lemmatization)
3. Take intersection of original keywords and generated keyword **word banks** (individual words)
4. For each prediction compare the length of intersecting words with length of total keyword words
Output is given below
## Word2vec citations
1. Spanish: Spanish Billion Word Corpus and Embeddings by Cristian Cardellino https://crscardellino.ar/SBWCE/
2. German: @thesis{mueller2015,
author = {{Müller}, Andreas},
title = "{Analyse von Wort-Vektoren deutscher Textkorpora}",
school = {Technische Universität Berlin},
year = 2015,
month = jun,
type = {Bachelor's Thesis},
url = {https://devmount.github.io/GermanWordEmbeddings}
}
3. French: @misc{fauconnier_2015,
author = {Fauconnier, Jean-Philippe},
title = {French Word Embeddings},
url = {http://fauconnier.github.io},
year = {2015}}
4. Italian: Nordic Language Processing Laboratory (NLPL) (http://vectors.nlpl.eu/repository/)
5. Portuguese: NILC - Interinstitutional Nucleus of Computational Linguistics http://nilc.icmc.usp.br/embeddings