{"id":29031646,"url":"https://github.com/poacosta/pdf-2-chroma","last_synced_at":"2026-05-04T23:33:38.204Z","repository":{"id":300475408,"uuid":"1006275751","full_name":"poacosta/pdf-2-chroma","owner":"poacosta","description":"A production-ready Python script that transforms PDF document collections into locally-stored, semantically searchable vector databases using ChromaDB's persistent storage.","archived":false,"fork":false,"pushed_at":"2025-06-21T22:26:43.000Z","size":12,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-21T23:24:20.363Z","etag":null,"topics":["chromadb","embeddings","openai","pdf-document-processor","python","rag","sentence-transformers","vector-database"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/poacosta.png","metadata":{"files":{"readme":"README.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-21T22:17:16.000Z","updated_at":"2025-06-21T22:26:46.000Z","dependencies_parsed_at":"2025-06-21T23:34:38.022Z","dependency_job_id":null,"html_url":"https://github.com/poacosta/pdf-2-chroma","commit_stats":null,"previous_names":["poacosta/pdf-2-chroma"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/poacosta/pdf-2-chroma","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poacosta%2Fpdf-2-chroma","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poacosta%2Fpdf-2-chroma/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poacosta%2Fpdf-2-chroma/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poacosta%2Fpdf-2-chroma/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/poacosta","download_url":"https://codeload.github.com/poacosta/pdf-2-chroma/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poacosta%2Fpdf-2-chroma/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262044447,"owners_count":23249750,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chromadb","embeddings","openai","pdf-document-processor","python","rag","sentence-transformers","vector-database"],"created_at":"2025-06-26T10:04:52.082Z","updated_at":"2026-05-04T23:33:33.179Z","avatar_url":"https://github.com/poacosta.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PDF2Chroma\n\n[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)\n[![ChromaDB](https://img.shields.io/badge/ChromaDB-1.0.0+-green.svg)](https://www.trychroma.com/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nA production-ready Python script that transforms PDF document collections into locally-stored, semantically searchable\nvector databases using ChromaDB's persistent storage. This script bridges the gap between static document repositories\nand intelligent information retrieval systems.\n\n## The Vector Database Transformation Pipeline\n\nModern knowledge management faces a fundamental challenge: converting unstructured document content into computationally\naccessible representations. This script implements a pipeline that addresses three critical transformation stages:\n\n**Stage 1: Document Deconstruction**\n\n- Intelligent PDF text extraction with structural preservation\n- Page-level content organization and metadata capture\n- Quality filtering to eliminate processing artifacts\n\n**Stage 2: Semantic Chunking**\n\n- Boundary-aware text segmentation that respects semantic coherence\n- Configurable overlap strategies for context preservation\n- Content-based unique identifier generation\n\n**Stage 3: Vector Storage**\n\n- Embedding generation using state-of-the-art transformer models\n- Persistent local storage in ChromaDB's optimized format\n- Hierarchical metadata preservation for advanced filtering\n\n**Mechanical Insights:** The script leverages ChromaDB's embedded database architecture, storing all vector data locally\nin a self-contained directory structure. This eliminates external dependencies while providing enterprise-grade search\ncapabilities.\n\n**Implications:** By maintaining complete local control over document embeddings, organizations can implement RAG (\nRetrieval Augmented Generation) systems without external API dependencies or data privacy concerns.\n\n## Local Storage Architecture\n\nThe script creates a persistent ChromaDB instance that stores all document vectors, metadata, and search indices in your\nlocal filesystem:\n\n```\npdf-2-chroma/\n├── docs/                      # Source PDF directory\n├── knowledge_base/            # ChromaDB persistent storage\n│   ├── chroma.sqlite3         # Vector database file\n└── main.py                    # This script\n```\n\n**Storage Characteristics:**\n\n- **Self-Contained:** All data remains on your local machine\n- **Persistent:** Survives script restarts and system reboots\n- **Portable:** Database directory can be copied between systems\n- **Scalable:** Handles collections from hundreds to millions of chunks\n\n## Installation and Setup\n\n### Dependencies\n\n```bash\n# Core requirements\npip install chromadb pdfplumber\n\n# Optional: OpenAI embeddings\npip install openai\n```\n\n## Script Configuration and Usage\n\n### Basic Document Processing\n\n```python\n# Initialize the document processor\nloader = ChromaDBDocumentLoader(\n    persist_directory=\"./my_knowledge_base\",  # Local storage location\n    embedding_model=\"all-MiniLM-L6-v2\"  # Local embedding model\n)\n\n# Process all PDFs in a directory\nresults = loader.load_pdfs_from_folder(\n    folder_path=\"./company_documents\",\n    collection_name=\"corporate_kb\"\n)\n```\n\n**What Happens During Processing:**\n\n1. **PDF Discovery:** Script scans the specified directory for PDF files\n2. **Text Extraction:** Each PDF is processed page-by-page using pdfplumber\n3. **Intelligent Chunking:** Text is segmented with semantic boundary detection\n4. **Embedding Generation:** Chunks are converted to vectors using the specified model\n5. **Local Storage:** All data is persisted to the ChromaDB directory\n\n### Advanced Configuration Options\n\n#### Embedding Model Selection\n\nThe script supports multiple embedding strategies, each with distinct performance characteristics:\n\n**Local Sentence Transformers:**\n\n```python\n# Balanced performance and quality\nembedding_model = \"all-MiniLM-L6-v2\"  # 384 dimensions, fast processing\n\n# Superior semantic understanding\nembedding_model = \"all-mpnet-base-v2\"  # 768 dimensions, slower but higher quality\n```\n\n**OpenAI Cloud Embeddings:**\n\n```python\nloader = ChromaDBDocumentLoader(\n    use_openai_embeddings=True,\n    openai_model=\"text-embedding-3-small\",  # Cost-effective cloud processing\n    # Requires OPENAI_API_KEY environment variable\n)\n```\n\n**Insights:** Local models provide complete data privacy and eliminate API costs, while cloud models offer superior\nsemantic understanding at the expense of external dependencies.\n\n#### Chunking Strategy Optimization\n\n```python\nloader = ChromaDBDocumentLoader(\n    chunk_size=1200,  # Larger contexts for complex documents\n    chunk_overlap=200,  # Enhanced context preservation\n    min_chunk_size=100  # Filter out fragmentary content\n)\n```\n\n**Chunking Algorithm Details:**\n\n- **Boundary Detection:** Prioritizes sentence endings over word boundaries\n- **Overlap Strategy:** Maintains semantic continuity between adjacent chunks\n- **Size Optimization:** Balances context preservation with computational efficiency\n\n### Local Database Querying\n\nOnce documents are loaded, the local ChromaDB instance enables sophisticated search operations:\n\n```python\n# Semantic similarity search\nresults = loader.search_documents(\n    collection_name=\"corporate_kb\",\n    query=\"quarterly financial performance metrics\",\n    n_results=5\n)\n\n# Metadata-filtered search\nfiltered_results = loader.search_documents(\n    collection_name=\"corporate_kb\",\n    query=\"product specifications\",\n    filter_metadata={\"department\": \"engineering\"}\n)\n```\n\n**Query Processing Flow:**\n\n1. **Query Embedding:** Search text is converted to vector representation\n2. **Similarity Calculation:** ChromaDB computes cosine distances across all stored vectors\n3. **Ranking:** Results are sorted by semantic relevance\n4. **Metadata Enrichment:** Source documents and page numbers are included\n\n## Script Architecture Deep Dive\n\n### PDF Processing Pipeline\n\nThe document processing pipeline implements several approaches to common PDF extraction challenges:\n\n**Text Extraction Challenges and Solutions:**\n\n- **Challenge:** PDF formatting artifacts and layout inconsistencies\n    - **Solution:** pdfplumber's advanced parsing with post-processing cleanup\n- **Challenge:** Maintaining document structure and page relationships\n    - **Solution:** Hierarchical metadata preservation across file, page, and chunk levels\n\n**Chunking Innovation:**\nTraditional PDF processors often fragment content at arbitrary character boundaries, destroying semantic coherence. This\nscript implements boundary-aware chunking:\n\n```python\ndef chunk_text(self, text: str, base_metadata: Dict[str, Any]) -\u003e List[DocumentChunk]:\n# Intelligent boundary detection hierarchy:\n# 1. Paragraph breaks (\\n\\n) - Primary preference\n# 2. Sentence endings (., !, ?) - Secondary preference  \n# 3. Word boundaries - Fallback mechanism\n# 4. Character limits - Hard constraint\n```\n\n**Insights:** By respecting natural language boundaries, the chunking algorithm preserves semantic\nrelationships that would be lost in naive character-based splitting approaches.\n\n### Local Database Persistence\n\nChromaDB's embedded architecture provides enterprise-grade persistence without external infrastructure:\n\n**Storage Technology Stack:**\n\n- **SQLite Backend:** ACID-compliant relational storage for metadata\n- **Vector Indices:** Optimized similarity search structures\n- **File-Based Persistence:** Self-contained storage requiring no database servers\n\n**Data Integrity Mechanisms:**\n\n- **Unique Chunk IDs:** Content-based hashing prevents duplicates\n- **Incremental Updates:** Skip processing for previously loaded documents\n- **Error Recovery:** Graceful handling of corrupted or protected PDFs\n\n### Embedding Model Integration\n\nThe script abstracts embedding generation to support multiple model architectures:\n\n**Local Model Advantages:**\n\n- Complete data privacy and offline operation\n- No usage-dependent costs or rate limits\n- Deterministic results for reproducible experiments\n\n**Cloud Model Advantages:**\n\n- Superior semantic understanding from large-scale training\n- Regular model updates and improvements\n- Reduced local computational requirements\n\n**Performance Comparison:**\n\n```\nModel                    | Dimensions | Processing Speed | Semantic Quality\n------------------------|------------|------------------|------------------\nall-MiniLM-L6-v2        | 384        | ~1000 docs/min  | Good\nall-mpnet-base-v2       | 768        | ~400 docs/min   | Excellent  \ntext-embedding-3-small  | 1536       | ~600 docs/min   | Excellent\ntext-embedding-3-large  | 3072       | ~200 docs/min   | Superior\n```\n\n## Running the Script\n\n### Command Line Execution\n\n```bash\n# Basic usage\npython main.py\n\n# With custom configuration\npython main.py --source-dir ./docs --db-dir ./knowledge_base --model all-mpnet-base-v2\n```\n\n## Security and Privacy Considerations\n\nThe local storage architecture provides several security advantages:\n\n**Data Sovereignty:**\n\n- All document content remains on local infrastructure\n- No external API calls for basic embedding models\n- Complete control over access patterns and usage logs\n\n**Privacy Preservation:**\n\n- Document text never transmitted to external services (with local models)\n- Embedding vectors stored locally without external dependencies\n- Metadata filtering enables sensitive document segregation\n\n**Compliance Benefits:**\n\n- Supports GDPR, HIPAA, and other data protection requirements\n- Enables air-gapped deployment scenarios\n- Facilitates audit trails through local logging\n\n## Troubleshooting Common Issues\n\n### PDF Processing Errors\n\n* **Issue:** Some PDFs fail to process\n  * **Solution:** Check file permissions and PDF encryption status\n\n* **Issue:** Extracted text contains formatting artifacts  \n  * **Solution:** Adjust `min_chunk_size` parameter to filter fragments\n\n### Database Performance\n\n* **Issue:** Slow query responses on large collections\n  * **Solution:** Implement collection partitioning or increase SSD storage\n\n* **Issue:** High memory usage during processing\n  * **Solution:** Reduce `batch_size` parameter for available RAM\n\n### Embedding Model Selection\n\n* **Issue:** Poor search relevance with default model\n  * **Solution:** Upgrade to `all-mpnet-base-v2` for better semantic understanding\n\n* **Issue:** OpenAI API rate limits\n  * **Solution:** Switch to local models for unlimited processing\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoacosta%2Fpdf-2-chroma","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpoacosta%2Fpdf-2-chroma","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoacosta%2Fpdf-2-chroma/lists"}