https://github.com/davidjnevin/vocab_extractor
A python based vocabulary extractor from class notes
https://github.com/davidjnevin/vocab_extractor
Last synced: 3 months ago
JSON representation
A python based vocabulary extractor from class notes
- Host: GitHub
- URL: https://github.com/davidjnevin/vocab_extractor
- Owner: davidjnevin
- Created: 2023-04-14T19:13:10.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-05-03T21:15:57.000Z (about 1 year ago)
- Last Synced: 2024-12-31T02:23:46.935Z (5 months ago)
- Language: Python
- Size: 186 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Vocab Extractor (Early Beta Tokenizer)
This version of the Vocabulary Extractor script processes a collection `.docx` files and extracts unique words and their corresponding Part of Speech (POS) tags. It saves the extracted (word, pos) pairs and cumulative word_count into an SQLite database.
## Features
- Tokenizes the document content using the NLTK library.
- Extracts Part of Speech (POS) information for each word.
- Efficiently inserts unique (word, pos, word_count) into an SQLite database using batch inserts and in-memory data structure.
- Includes a test suite to ensure the functionality of the script.
- Command line interface for easy execution and configuration
## Test Suite
A comprehensive test suite is available to test the functionality of the Vocab Extractor. The test suite covers the following areas:
- Extracting vocabulary from text
- Reading file contents
- Creating a database connection
- Inserting words into the database
- Processing reports
- Batching files
- Processing all reports
- Command line interface inputs
To run the tests, make sure you have `pytest` installed: