{"id":23165225,"url":"https://github.com/akshatrajsaxena/implementing-lsh","last_synced_at":"2025-04-04T21:21:40.478Z","repository":{"id":262665596,"uuid":"873639340","full_name":"akshatrajsaxena/Implementing-LSH","owner":"akshatrajsaxena","description":"Implementation of LSH in order to find the similarity in a large dataset","archived":false,"fork":false,"pushed_at":"2024-11-13T15:54:56.000Z","size":3020,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-10T05:45:19.091Z","etag":null,"topics":["lsh","lsh-algorithm"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akshatrajsaxena.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-16T13:57:01.000Z","updated_at":"2024-11-13T15:55:00.000Z","dependencies_parsed_at":"2024-11-13T16:53:36.480Z","dependency_job_id":null,"html_url":"https://github.com/akshatrajsaxena/Implementing-LSH","commit_stats":null,"previous_names":["akshatrajsaxena/implementing-lsh"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshatrajsaxena%2FImplementing-LSH","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshatrajsaxena%2FImplementing-LSH/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshatrajsaxena%2FImplementing-LSH/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akshatrajsaxena%2FImplementing-LSH/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akshatrajsaxena","download_url":"https://codeload.github.com/akshatrajsaxena/Implementing-LSH/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247249865,"owners_count":20908263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lsh","lsh-algorithm"],"created_at":"2024-12-18T01:15:39.509Z","updated_at":"2025-04-04T21:21:40.459Z","avatar_url":"https://github.com/akshatrajsaxena.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LSH Implementation \n\nThe Implementation of the project is divided into four substeps which includes::\n\n## 1. **Reading the Input Content and Text Preprocessing**\n\nThis involves loading data from the following files:\n- `ids.txt`: Contains the document IDs.\n- `texts.txt`: Contains the text data corresponding to the document IDs.\n\nText preprocessing typically involves several operations:\n- **Lowercase conversion**: Ensures uniformity by converting all text to lowercase.\n- **Removing punctuation**: Cleans the text to focus on words.\n- **Tokenization**: Splits the text into individual words or tokens.\n\nFurther, the preprocessing may also include:\n- **Stopword removal**: Eliminates common words (e.g., \"the\", \"is\", \"and\") which do not contribute much meaning.\n- **Stemming/Lemmatization**: Reduces words to their base forms (e.g., \"running\" → \"run\") to standardize the text, improving the comparison process.\n\n## 2. **Vectorizing the Text with TF-IDF**\n\n**TF-IDF** (Term Frequency-Inverse Document Frequency) is a technique used to convert text into numerical vectors. The process weighs terms based on two factors:\n\n- **Term Frequency (TF)**: Measures how often a word appears in a document.\n- **Inverse Document Frequency (IDF)**: Measures how rare a word is across all documents.\n\nThis method captures the relative importance of words, allowing for meaningful comparisons between documents. TF-IDF representation is effective for identifying key terms and measuring document similarity.\n\n## 2. **Vectorizing the Text with TF-IDF**\n\n**TF-IDF** (Term Frequency-Inverse Document Frequency) is a technique used to convert text into numerical vectors. The process weighs terms based on two factors:\n\n- **Term Frequency (TF)**: Measures how often a word appears in a document.\n- **Inverse Document Frequency (IDF)**: Measures how rare a word is across all documents.\n\nThis method captures the relative importance of words, allowing for meaningful comparisons between documents. TF-IDF representation is effective for identifying key terms and measuring document similarity.\n\n---\n\n### Proper Working of TF-IDF:\n\n**TF-IDF** stands for Term Frequency-Inverse Document Frequency. It measures how important a word is in a document within a collection of documents.\n\nThe TF-IDF value for a term is calculated by multiplying two metrics:\n\n1. **Term Frequency (TF)**: The number of times a term appears in a document, normalized by the total number of terms in the document.\n\n   $$\n   TF(t, d) = \\frac{\\text{Number of times term } t \\text{ appears in document } d}{\\text{Total number of terms in document } d}\n   $$\n\n2. **Inverse Document Frequency (IDF)**: A measure of how rare a term is across all documents. The more rare a term is, the higher its IDF value.\n\n   $$\n   IDF(t) = \\log \\left(\\frac{N}{n_t}\\right)\n   $$\n\n   Where:\n   - \\( N \\) = Total number of documents in the corpus\n   - \\( n_t \\) = Number of documents containing the term \\( t \\)\n\nThe final **TF-IDF** score for a term is the product of the term frequency and inverse document frequency:\n\n---\n\nBy applying this formula, TF-IDF identifies important words in the document that are also relatively unique across the corpus.\n\n\n## 3. **Implementing LSH for Finding Similarity**\n\n**LSH** (Locality-Sensitive Hashing) is used to approximate similarity search, enabling efficient identification of similar documents. It works by:\n- Hashing similar items into the same \"buckets\" with high probability.\n\nFor text data, **MinHash** is commonly employed. It creates multiple hash functions to generate a signature for each document, allowing quick retrieval of potential similar items without performing exhaustive pairwise comparisons.\n\n---\n\n## 4. **Evaluating the Accuracy by Comparing with Ground Truth**\n\nThis step evaluates the performance of the LSH implementation by comparing its results to the provided ground truth in the `items.json` file. The evaluation process involves:\n- Calculating **intersection scores** between the predicted and actual similar items for each document.\n- Computing the **average score** across all documents to assess the overall model performance.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakshatrajsaxena%2Fimplementing-lsh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakshatrajsaxena%2Fimplementing-lsh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakshatrajsaxena%2Fimplementing-lsh/lists"}