https://github.com/knightron0/topicgeneration
Get current topic from captions using Latent Dirichlet Allocation.
https://github.com/knightron0/topicgeneration
captions lda sentences topic
Last synced: 13 days ago
JSON representation
Get current topic from captions using Latent Dirichlet Allocation.
- Host: GitHub
- URL: https://github.com/knightron0/topicgeneration
- Owner: knightron0
- Created: 2020-01-16T16:34:34.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2020-01-21T07:46:16.000Z (over 6 years ago)
- Last Synced: 2025-02-26T10:46:44.123Z (over 1 year ago)
- Topics: captions, lda, sentences, topic
- Language: Python
- Homepage:
- Size: 1.95 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Topic Generation
Finding out the current topic from caption. This uses a Latent Dirichlet Allocation (LDA) Model in order to predict the topic from captions. LDA is known to be most efficient way to do Topic Modelling
# Requirements
- nltk
- gensim
- sklearn
# How to use?
Clone the repository and open it in the terminal. Enter the following command:-
```
python main.py /pth/to/file
```
Note: The file should contain the paragraph for which the topic needs to be generated.
# Data Preprocessing
The most important part in Natural Language Processing or any type of Machine Learning is Data Preparation. For this project, in order to get the data ready for the model, the following steps were carried out:
- Tokenization: The document is split into sentences and the sentences are split into words. These words are further turned into lowercase and the punctuation is removed.
- Stopwords are removed and words which have length <= 3 are also removed.
- Lemmatization: The versbs are changed from past and future tense to present tense.
- Stemming: The words are stemmed and are converted to their root form.
After applying these steps, the dataset is then converted into a dictionary, which contains the word as the key and the number of occurences as the value. This is what is fed into the model.