{"id":23906838,"url":"https://github.com/maettuu/24hs-essentials-in-text-and-speech-processing","last_synced_at":"2025-07-20T21:32:59.856Z","repository":{"id":267839203,"uuid":"902517466","full_name":"maettuu/24HS-Essentials-in-Text-and-Speech-Processing","owner":"maettuu","description":"Repository for the course Essentials in Text and Speech Processing Fall 2024","archived":false,"fork":false,"pushed_at":"2025-01-08T20:17:21.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-19T19:52:50.630Z","etag":null,"topics":["amazon-books-reviews","approximate-nearest-neighbors","backend","content-based-recommendation","python","recommendation-system","sbert","tf-idf-vectorization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maettuu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-12T18:10:22.000Z","updated_at":"2025-01-08T20:17:25.000Z","dependencies_parsed_at":"2024-12-12T19:24:29.565Z","dependency_job_id":"85c54482-b662-462e-92ab-2a5d69490806","html_url":"https://github.com/maettuu/24HS-Essentials-in-Text-and-Speech-Processing","commit_stats":null,"previous_names":["maettuu/24hs-essentials-in-text-and-speech-processing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/maettuu/24HS-Essentials-in-Text-and-Speech-Processing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maettuu%2F24HS-Essentials-in-Text-and-Speech-Processing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maettuu%2F24HS-Essentials-in-Text-and-Speech-Processing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maettuu%2F24HS-Essentials-in-Text-and-Speech-Processing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maettuu%2F24HS-Essentials-in-Text-and-Speech-Processing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maettuu","download_url":"https://codeload.github.com/maettuu/24HS-Essentials-in-Text-and-Speech-Processing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maettuu%2F24HS-Essentials-in-Text-and-Speech-Processing/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266204533,"owners_count":23892364,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-books-reviews","approximate-nearest-neighbors","backend","content-based-recommendation","python","recommendation-system","sbert","tf-idf-vectorization"],"created_at":"2025-01-05T02:14:57.677Z","updated_at":"2025-07-20T21:32:59.839Z","avatar_url":"https://github.com/maettuu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 24HS Essentials in Text and Speech Processing (ETSP)\nThis repository includes the semester project of the course ETSP ([source repository](https://github.com/solodezaldivar/readAlike/tree/0d5913d29a7ab64459b55311e4176ca8ca8f992e)). The code is written in `Python v3.12`.\n\nMain packages: `annoy`, `sentene_transformers`, `torch`, `numpy`, `sklearn` : {`TfidfVectorizer`}, `scipy`, `pandas`, `matplotlib`, `tqdm`\n\n## readAlike\n\n`readAlike` is a book recommendation system that provides similar books to a given input book. The system leverages multiple techniques, including TF-IDF vectorization, Sentence-BERT embeddings, and Approximate Nearest Neighbors (ANN) for generating content-based book recommendations based on title, description, author, and category data ([dataset](https://www.kaggle.com/datasets/mohamedbakhet/amazon-books-reviews)).\n\n\n## How to Run\n\nTo run the recommendation engine:\n1. Open the command line and execute ``` pip install -r requirements.txt ```\n2. Execute `main.py`. Given an example book, the program will print the top five recommended books based on three methods: TF-IDF, SBERT, and ANN.\n\n\n## Project Structure\n\n### 1. Classes\n- **preprocessing/**: Manages data preprocessing.\n  - **`Preprocessor`**: Handles data cleaning and formatting from a CSV file of book data.\n- **core/**: Contains the core modules for recommendation.\n  - **`Library`** and **`Book`**: Models the library of books and individual book data.\n  - **`Vectorizer`**: Converts book text data into numerical vectors using TF-IDF and Sentence-BERT.\n  - **`DimensionalityReducer`**: Reduces the dimensionality of TF-IDF vectors using Truncated SVD.\n  - **`Ann`**: Creates an Approximate Nearest Neighbors model for efficient similarity search.\n  - **`Recommender`**: Main recommendation engine that integrates the above components to provide recommendations.\n\n### 2. Key Components\n- **`config.py`**: Configuration file with column names for title, description, authors, and categories.\n- **`main.py`**: Main entry point for running the recommendation pipeline.\n\n## Program Flow\n\n1. **Preprocessing**: The `Preprocessor` class reads the dataset and performs data cleaning.\n2. **Library Initialization**: `Library` is initialized with the cleaned dataset, storing each book as a `Book` object.\n3. **Vectorization**: `Vectorizer` creates TF-IDF and Sentence-BERT embeddings for each book.\n4. **Dimensionality Reduction**: `DimensionalityReducer` reduces TF-IDF embeddings for optimized ANN performance.\n5. **ANN Construction**: `Ann` constructs an ANN model based on the reduced vectors.\n6. **Recommendation**: `Recommender` classifies recommendations into TF-IDF, SBERT, and ANN-based results, outputting top similar books.\n\n## Classes and Methods\n\n### `Preprocessor`\n- **Attributes**:\n  - `df`: DataFrame containing cleaned book data.\n- **Methods**:\n  - `preprocess_data()`: Cleans and formats the data.\n  - `drop_items_with_short_entries()`, `drop_duplicates()`, `convert_strings_into_lists()`: Helper functions to clean the dataset.\n\n### `Library`\n- **Attributes**:\n  - `books`: List of `Book` objects.\n- **Methods**:\n  - `get_combined_data()`: Concatenates title, description, authors, and categories into a single string per book.\n  - `get_book_idx()`: Retrieves the index of a book within the library.\n\n### `Book`\n- **Attributes**:\n  - `title`, `description`, `authors`, `categories`: Fields describing the book.\n- **Methods**:\n  - `get_combined_data()`: Combines title, description, authors, and categories into a single string.\n\n### `Vectorizer`\n- **Attributes**:\n  - `tfidf_matrix`: Sparse matrix of TF-IDF vectors.\n  - `sbert_embeddings`: Sentence-BERT embeddings for each book.\n- **Methods**:\n  - `tfidf_vectorize()`: Vectorizes a book using TF-IDF.\n  - `sbert_vectorize()`: Vectorizes a book or library using SBERT.\n\n### `DimensionalityReducer`\n- **Attributes**:\n  - `reduced_matrix`: Dimensionality-reduced version of the TF-IDF matrix.\n- **Methods**:\n  - `reduce()`: Reduces a TF-IDF vector to the lower dimension.\n\n### `Ann`\n- **Attributes**:\n  - `ann_indices`: ANN model for similarity search.\n- **Methods**:\n  - `get_nearest_neighbors_by_index()`, `get_nearest_neighbors_by_vector()`: Retrieves nearest neighbors by item index or vector.\n\n### `Recommender`\n- **Attributes**:\n  - `lib`: Library of books.\n  - `vectorizer`: Vectorizer instance.\n  - `reducer`: Dimensionality reducer instance.\n  - `ann`: ANN instance.\n- **Methods**:\n  - `recommend()`: Provides top recommendations based on TF-IDF, SBERT, and ANN.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaettuu%2F24hs-essentials-in-text-and-speech-processing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaettuu%2F24hs-essentials-in-text-and-speech-processing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaettuu%2F24hs-essentials-in-text-and-speech-processing/lists"}