{"id":23246551,"url":"https://github.com/itsyaasir/pdf-intellect","last_synced_at":"2025-07-27T09:09:37.616Z","repository":{"id":184126344,"uuid":"671349271","full_name":"itsyaasir/pdf-intellect","owner":"itsyaasir","description":"PDF Intellect is a smart AI-powered tool designed to extract and analyze information from PDF document","archived":false,"fork":false,"pushed_at":"2023-09-21T17:15:26.000Z","size":5846,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-18T22:00:21.139Z","etag":null,"topics":["ai","analyzer","conda","llama2","llamacpp","machine-learning","pdf","project","python","smart"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/itsyaasir.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-27T05:55:12.000Z","updated_at":"2024-05-27T06:14:02.000Z","dependencies_parsed_at":"2023-07-27T07:45:14.328Z","dependency_job_id":"92f98aef-b310-4a59-b272-f027f906c034","html_url":"https://github.com/itsyaasir/pdf-intellect","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"f183e894b0c59d759de21abbc53f31732335cf73"},"previous_names":["itsyaasir/pdf-vector-db"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsyaasir%2Fpdf-intellect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsyaasir%2Fpdf-intellect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsyaasir%2Fpdf-intellect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/itsyaasir%2Fpdf-intellect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/itsyaasir","download_url":"https://codeload.github.com/itsyaasir/pdf-intellect/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230394166,"owners_count":18218707,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","analyzer","conda","llama2","llamacpp","machine-learning","pdf","project","python","smart"],"created_at":"2024-12-19T07:15:57.534Z","updated_at":"2024-12-19T07:15:58.108Z","avatar_url":"https://github.com/itsyaasir.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n---\n\n# PDF Intellect\n\nPDF Embedding Indexer is a CLI tool designed to process text content from PDF files, generate meaningful embeddings from the text using Sentence Transformers, and store those embeddings in a PostgreSQL database. This allows for quick and efficient similarity searching, providing a useful tool for managing and navigating through a large number of PDF files.\n\n## Features\n\n- Currently supports PDF files only, and is only limited to a single PDF file per command.\n- Extracts text content from PDF files and generates embeddings using Sentence Transformers.\n- Stores the embeddings in a PostgreSQL database for quick and efficient similarity searching.\n- Allows for sentence-level indexing, offering granular search results.\n- Stores additional metadata for each document, including file hash, timestamp, and title.\n- Prevents duplicate PDFs from being indexed.\n\n## Requirements\n\n- Python 3.8 or higher\n- PostgreSQL\n- [sentence-transformers](https://github.com/UKPLab/sentence-transformers)\n- psycopg2-binary\n- sqlalchemy\n- pgvector\n- python-magic\n- pdfminer.six\n- nltk\n\n## Setup\n\n1. Ensure that you have Python 3.8 or higher installed.\n\n2. Install PostgreSQL and setup a database for this project.\n\n3. Clone this repository:\n\n    ```bash\n    git clone https://github.com/itsyaasir/pdf-intellect.git\n    ```\n\n4. Change into the project directory:\n\n    ```bash\n    cd pdf-intellect\n    ```\n\n5. You can create and activate a Conda or a virtual environment:\n\n    - For Conda environment:\n\n        Run the provided setup script to create a Conda environment and install the necessary packages:\n\n        ```bash\n        bash conda_setup.sh\n        ```\n\n    - For virtual environment:\n\n        Create a virtual environment:\n\n        ```bash\n        python -m venv venv\n        ```\n\n        Activate the virtual environment:\n        - On Unix or MacOS, run:\n\n            ```bash\n            source venv/bin/activate\n            ```\n\n        - On Windows, run:\n\n            ```bash\n            venv\\Scripts\\activate\n            ```\n\n        Install the required packages:\n\n        ```bash\n        pip install -r requirements.txt\n        ```\n\n6. Run the provided setup script to setup the database:\n\n    You will need to prov\n\n    ```bash\n    bash db_setup.sh\n    ```\n\n7. Modify the environment variables in `config.py` if necessary.\n\n8. Depending on your model, you might need to adjust the prompt template to match the model's input format.\n    You can check the default template in `app/llama.py`.\n\n## Usage\n\nTo index a PDF:\n\n```bash\npython main.py index \u003cpdf_file\u003e\n```\n\nTo search for similar content given a query:\n\n```bash\npython main.py search \"\u003cquery\u003e\"\n```\n\nTo use the PDF with LLM:\n\n```bash\npython main.py query \u003c\"query\"\u003e\n```\n\nThese commands will print their results to the console.\n\n## Contributing\n\nPlease feel free to fork this repository and contribute. When submitting your changes, please ensure that your code is well-commented and that you have tested your changes.\n\n## License\n\nThis project is licensed under the terms of the MIT license. See [LICENSE](LICENSE) for more details.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitsyaasir%2Fpdf-intellect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fitsyaasir%2Fpdf-intellect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fitsyaasir%2Fpdf-intellect/lists"}