{"id":22615491,"url":"https://github.com/sovit-123/local_file_search","last_synced_at":"2025-04-11T12:12:39.553Z","repository":{"id":247452407,"uuid":"825885523","full_name":"sovit-123/local_file_search","owner":"sovit-123","description":"Local file search using embedding techniques","archived":false,"fork":false,"pushed_at":"2025-03-02T11:20:42.000Z","size":113,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-25T08:38:15.568Z","etag":null,"topics":["embedding-models","embeddings","nlp","vector-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sovit-123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-08T17:33:43.000Z","updated_at":"2025-03-02T11:20:46.000Z","dependencies_parsed_at":"2025-03-02T15:15:16.251Z","dependency_job_id":null,"html_url":"https://github.com/sovit-123/local_file_search","commit_stats":null,"previous_names":["sovit-123/local_file_search"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovit-123%2Flocal_file_search","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovit-123%2Flocal_file_search/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovit-123%2Flocal_file_search/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sovit-123%2Flocal_file_search/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sovit-123","download_url":"https://codeload.github.com/sovit-123/local_file_search/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248398564,"owners_count":21097292,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embedding-models","embeddings","nlp","vector-search"],"created_at":"2024-12-08T19:07:43.306Z","updated_at":"2025-04-11T12:12:39.543Z","avatar_url":"https://github.com/sovit-123.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Local File Search using Embeddings\n\nScripts to replicate simple file search and RAG in a directory with embeddings and Language Models.\n\nFrom scratch implementation, ***no Vector DBs yet.***\n\n***A simplified use case***: You have thousands of research papers but don't know which are the ones containing content that you want. You do a search according to a rough query and get an adequately good results. \n\n\nhttps://github.com/user-attachments/assets/0b1bfc91-868b-4aa7-80ba-9e0a730c4b4b\n\n## Setup\n\nBefore moving forward with any of the installation steps, ensure that you have CUDA \u003e 12.1 installed globally on your system. Necessary for building Flash Attention.\n\n### Ubuntu\n\nRun the following in terminal in your preferred virtual/conda environment.\n\n```\nsh setup.sh\n```\n\nIt will install the the requirements from the `requirements.txt` file.\n\n### Windows\n\nInstall the required version of PyTorch first, preferably the latest stable supported by this repository. This is a necessary step to build Flash Attention correctly on Windows.\n\n```\npip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1\n```\n\nNext install rest of the requirements.\n\n```\npip install -r requirements.txt\n```\n\n## Updates\n\n* September 4, 2024: Added image, PDF, and text file chat to `app.py` with multiple Phi model options.\n\n* September 1, 2024: Now you can upload PDFs directly to the Gradio UI (`python app.py`) and start chatting.\n\n## Steps to Chat with Any PDF in Graio UI\n\n\u003cspan style=\"color: green\"\u003e***You can run `app.py` and select the any PDF file in the Gradio UI to interactively chat with the document.***\u003c/span\u003e \u003cspan style=\"color: purple\"\u003e***(Just execute `python app.py` and start chatting)***\u003c/span\u003e\n\n## Steps to Run Through CLI\n\n* (**Optional**) Download the `papers.csv` file from [here](https://www.kaggle.com/datasets/benhamner/nips-papers?select=papers.csv) and keep in the `data` directory. \u003cspan style=\"color: red\"\u003e**You can also keep PDF files in the directory and pass the directory path**.\u003c/span\u003e\n\n* (**Optional**) \u003cspan style=\"color: red\"\u003e*Execute this step only if you download the above CSV file. Not needed if you have your own text files or PDFs in a directory*\u003c/span\u003e. Run the `csv_to_text_files.py` script to generate a directory of text files from the CSV file. \n\n* Run `create_embeddings.py` to generate the embeddings that are stored in JSON file in the `data` directory. Check the scripts for the respective file names. ***Check `src/create_embedding.py`*** for relevant command line arguments to be passed.\n\n  * Generate example:\n\n    ```\n    python create_embeddings.py --index-file-name index_file_to_store_embeddings.json --directory-path path/to/directory/containing/files/to/embed\n    ```\n\n  * Additional command line arguments:\n\n    * `--add-file-content`: To store text chunks in JSON file if planning to do RAG doing file file search.\n    * `--model`: Any [Sentence Transformer](https://www.sbert.net/docs/sentence_transformer/pretrained_models.html) model tag. Default is `all-MiniLM-L6-v2`.\n    * `--chunk-size` and `--overlap`: Chunk size for creating embeddings and overlap between chunks.\n    * `--njobs`: Number of parallel processes to use. Useful when creating embeddings for hundreds of files in a directory.\n\n* Then run the `search.py` with the path to the respective embedding file to start the search. Type in the search query.\n\n  * General example:\n\n    ```\n    python search.py --index-file path/to/index.json\n    ```\n  \n    The above command just throws a list of TopK files that matches the query.\n\n  * Additional command line arguments:\n\n    * `--extract-content`: Whether to print the related content or not. Only works if `--add-file-content` was passed during creation of embeddings.\n    * `--model`: Sentence Transformer model tag if a model other than `all-MiniLM-L6-v2` was used during the creation of embeddings.\n    * `--topk`: Top K embeddings to match and output to the user.\n    * `--llm-call`: Use an LLM to restructure the answer for the question asked. Only works if `--extract-content` is passed as the model will need context. Currently the Phi-3 Mini 4K model is used. \n  \n\n\n## Datasets\n\n* [NIPS Research Papers](https://www.kaggle.com/datasets/benhamner/nips-papers?select=papers.csv)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsovit-123%2Flocal_file_search","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsovit-123%2Flocal_file_search","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsovit-123%2Flocal_file_search/lists"}