{"id":18634628,"url":"https://github.com/natnaelhhaile/Text-Similarity-Analysis","last_synced_at":"2025-04-11T07:33:30.209Z","repository":{"id":246457426,"uuid":"821189117","full_name":"natnaelhhaile/Text-Similarity-Analysis","owner":"natnaelhhaile","description":null,"archived":false,"fork":false,"pushed_at":"2024-06-28T02:31:20.000Z","size":6,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-05T11:53:49.834Z","etag":null,"topics":["bag-of-words","cosine-similarity","data-analysis","machine-learning","natural-language-processing","nltk-python","one-hot-encoding","python","stemming","stop-word-removal","stop-words","text-mining","text-processing","text-similarity-analysis","tf","tf-idf","tokenization"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/natnaelhhaile.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-28T02:21:39.000Z","updated_at":"2024-06-28T02:31:23.000Z","dependencies_parsed_at":"2024-06-28T03:49:13.993Z","dependency_job_id":null,"html_url":"https://github.com/natnaelhhaile/Text-Similarity-Analysis","commit_stats":null,"previous_names":["cur10usitydrives/text-similarity-analysis","curiousitydrives/text-similarity-analysis","natnaelhhaile/text-similarity-analysis"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/natnaelhhaile%2FText-Similarity-Analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/natnaelhhaile%2FText-Similarity-Analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/natnaelhhaile%2FText-Similarity-Analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/natnaelhhaile%2FText-Similarity-Analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/natnaelhhaile","download_url":"https://codeload.github.com/natnaelhhaile/Text-Similarity-Analysis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248358917,"owners_count":21090447,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bag-of-words","cosine-similarity","data-analysis","machine-learning","natural-language-processing","nltk-python","one-hot-encoding","python","stemming","stop-word-removal","stop-words","text-mining","text-processing","text-similarity-analysis","tf","tf-idf","tokenization"],"created_at":"2024-11-07T05:20:08.117Z","updated_at":"2025-04-11T07:33:29.917Z","avatar_url":"https://github.com/natnaelhhaile.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Text Similarity Analysis with Various Encoding Techniques\n\n## Overview\n\nThis project explores text similarity analysis using different encoding techniques and preprocessing steps. \nIt aims to evaluate how these factors impact the accuracy of measuring similarity between pairs of sentences.\n\n## Key Components\n\n### 1. Preprocessing\n\n- **Tokenization**: Sentences are tokenized into words using NLTK's `word_tokenize`.\n  \n- **Stop-word Removal**: Common English stop words (e.g., \"the\", \"is\", \"and\") are removed to focus on meaningful content words.\n  \n- **Stemming**: Words are stemmed to their root form using the Porter Stemmer algorithm, reducing inflectional forms and variants.\n\n### 2. Encoding Techniques\n\n- **One-Hot Encoding**: Represents each word in the sentence as a binary vector, where each word presence is marked as 1 or 0.\n  \n- **Bag-of-Words**: Represents each sentence as a vector of word frequencies in a predefined dictionary (vocabulary).\n  \n- **Term Frequency (TF)**: Represents each word in the sentence as its frequency of occurrence within the sentence.\n  \n- **Term Frequency-Inverse Document Frequency (TF-IDF)**: Weighs each word by its frequency in the document and inversely by its\n                                                          frequency across all documents (sentences).\n\n### 3. Similarity Measurement\n\nThe similarity between pairs of sentences is measured using the cosine similarity metric. Cosine similarity computes the cosine \nof the angle between two vectors of word counts (or TF-IDF scores), providing a measure of similarity irrespective of the sentence length.\n\n### 4. Evaluation Scenarios\n\nSeveral scenarios are evaluated to demonstrate the impact of preprocessing steps (stop-word removal, stemming) \nand encoding techniques on similarity measurement:\n\n- **Stop-word Removal**: Increases recall but may decrease precision.\n  \n- **Stemming**: Increases recall by reducing word variations but may decrease precision.\n  \n- **Encoding Techniques**: One-hot encoding, bag-of-words, TF, and TF-IDF are compared for their precision in capturing semantic similarity.\n\n### 5. IDF Calculation\n\nThe project includes a calculation of Inverse Document Frequency (IDF) for terms in a given corpus (sentences). IDF is used to \nweight terms by their rarity across the corpus, enhancing the discriminative power of TF-IDF.\n\n## Usage\n\n### Dependencies\n\n- Python 3.x\n- NLTK (Natural Language Toolkit)\n- scikit-learn (for CountVectorizer and TfidfVectorizer)\n\nInstall dependencies using `pip`:\n\n```bash\npip install -r ...\n```\n\n### Running the Project\n\n1. Clone the repository.\n\n2. Ensure you have the necessary NLTK corpora downloaded:\n\n```python\nimport nltk\nnltk.download('punkt')\nnltk.download('stopwords')\n```\n\n3. Run the script to execute evaluation scenarios.\n\n### Example Output\n\nThe script provides detailed output for each scenario, including original similarity scores and scores after preprocessing \n(stop-word removal, stemming) and encoding.\n\n## Conclusion\n\nThis project serves as a foundational exploration into text similarity analysis using various preprocessing techniques and \nencoding methods. It highlights the trade-offs between recall and precision and provides insights into optimizing text \nsimilarity computations for different applications.\n\n## Author\n\nNatnael Haile\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatnaelhhaile%2FText-Similarity-Analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnatnaelhhaile%2FText-Similarity-Analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatnaelhhaile%2FText-Similarity-Analysis/lists"}