{"id":20095208,"url":"https://github.com/chrissmartin/dataset-miner","last_synced_at":"2026-02-04T07:04:02.576Z","repository":{"id":260608796,"uuid":"860264628","full_name":"chrissmartin/dataset-miner","owner":"chrissmartin","description":"Generates standardised datasets from given docs for use in LLM training.","archived":false,"fork":false,"pushed_at":"2024-11-02T08:14:18.000Z","size":21,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-13T16:54:22.606Z","etag":null,"topics":["ai","context","context-extraction","dataset-generation","fine-tuning","llm","llm-framework"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrissmartin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-20T05:54:05.000Z","updated_at":"2024-11-09T00:56:15.000Z","dependencies_parsed_at":"2024-11-01T11:20:11.356Z","dependency_job_id":"9547369c-e933-432e-ba8a-ed438d1163bf","html_url":"https://github.com/chrissmartin/dataset-miner","commit_stats":null,"previous_names":["chrissmartin/dataset-miner"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrissmartin%2Fdataset-miner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrissmartin%2Fdataset-miner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrissmartin%2Fdataset-miner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrissmartin%2Fdataset-miner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrissmartin","download_url":"https://codeload.github.com/chrissmartin/dataset-miner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233690482,"owners_count":18714782,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","context","context-extraction","dataset-generation","fine-tuning","llm","llm-framework"],"created_at":"2024-11-13T16:54:26.027Z","updated_at":"2026-02-04T07:04:02.549Z","avatar_url":"https://github.com/chrissmartin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dataset Miner\n\nDataset Miner is a powerful Python tool designed to generate high-quality question-answer (Q\u0026A) pairs from various document formats using AI models. It processes documents such as PDFs, text files, Word documents, programming code files, JSON, CSV, and Excel files to create datasets suitable for fine-tuning language models or other NLP tasks.\n\n## 🌟 Features\n\n- **Multiple Format Support**:\n  - Documents: PDF, TXT, DOCX\n  - Data Files: JSON, CSV, XLSX, XLS\n  - Code Files: Python, Java, JavaScript, TypeScript, C++, and many more\n- **Advanced Processing**:\n  - Smart text chunking with support for code-aware splitting\n  - Intelligent header detection for markdown and HTML files\n  - Robust encoding handling for various file formats\n  - Support for table extraction from Word documents\n- **AI Integration**:\n  - Compatible with both Ollama and Groq AI models\n  - Customizable prompt templates for Q\u0026A generation\n  - Optional verification of generated Q\u0026A pairs\n- **Performance \u0026 Control**:\n  - Rate limiting for API calls\n  - Token usage tracking and cost analysis\n  - Progress tracking with detailed logging\n  - Support for processing large documents efficiently\n\n## 📋 Requirements\n\n- Python 3.8 or higher\n- Dependencies (automatically installed):\n  - langchain \u0026 langchain-community\n  - langchain-groq (for Groq integration)\n  - langchain-ollama (for Ollama integration)\n  - PyPDF2 (PDF processing)\n  - python-docx (Word document processing)\n  - pandas \u0026 openpyxl (Excel/CSV processing)\n  - tiktoken (token counting)\n  - Additional utilities: tqdm, colorama, python-dotenv\n\n## 💻 Installation\n\n1. Install from PyPI:\n\n   ```bash\n   pip install dataset-miner\n   ```\n\n2. Or install from source:\n   ```bash\n   git clone https://github.com/chrissmartin/dataset-miner.git\n   cd dataset-miner\n   pip install -e .\n   ```\n\n## 🚀 Usage\n\n### Command Line Interface\n\n```bash\ndataset-miner -source \u003cinput_directory\u003e -model \u003cai_model_name\u003e [options]\n```\n\nRequired arguments:\n\n- `-source`: Directory containing files to process\n- `-model`: AI model identifier (e.g., \"gpt-4o-mini\" for Ollama)\n\nOptional arguments:\n\n- `--output`: Output JSON file path (default: mined_dataset.json)\n- `--use-groq`: Use Groq instead of Ollama\n- `--verify`: Enable verification of generated Q\u0026A pairs\n- `--debug`: Enable debug logging\n- `--remove-empty-columns`: Remove empty columns from CSV/Excel files\n\n### Python API\n\n```python\nfrom dataset_miner import mine_documents\n\n# Basic usage\nmined_data, output_path = mine_documents(\n    source_dir=\"./documents\",\n    model=\"gpt-4o-mini\",\n    output_file=\"dataset.json\"\n)\n\n# Advanced usage with all options\nmined_data, output_path = mine_documents(\n    source_dir=\"./documents\",\n    model=\"gpt-4o-mini\",\n    output_file=\"dataset.json\",\n    use_groq=True,\n    remove_empty_columns=True,\n    verify=True,\n    debug=True\n)\n```\n\n## 📤 Output Format\n\nThe tool generates a JSON file containing Q\u0026A pairs in the Alpaca dataset format:\n\n```json\n[\n  {\n    \"instruction\": \"Question text here\",\n    \"input\": \"Additional context (if any)\",\n    \"output\": \"Answer text here\"\n  }\n]\n```\n\nWhen verification is enabled, each entry includes additional verification metadata:\n\n```json\n[\n  {\n    \"instruction\": \"Question text here\",\n    \"input\": \"Additional context (if any)\",\n    \"output\": \"Answer text here\",\n    \"verification\": {\n      \"status\": \"CORRECT\",\n      \"explanation\": \"Verification details\"\n    }\n  }\n]\n```\n\n## 📊 Cost Analysis\n\nThe tool provides detailed cost analysis and usage statistics:\n\n- Token usage tracking (input/output)\n- Cost breakdown by operation type\n- Verification costs (if enabled)\n- Average cost per Q\u0026A pair\n- Total cost summary\n\n## 🔍 Logging\n\nThe tool provides comprehensive logging with different verbosity levels:\n\n- Basic progress updates\n- Token usage and cost tracking\n- Error reporting and debugging information\n- Color-coded console output for better visibility\n\nEnable debug logging with the `--debug` flag for more detailed information.\n\n## 🔐 Environment Variables\n\n- `GROQ_API_KEY`: Required when using Groq integration (set in .env file)\n\n## 🤝 Contributing\n\nContributions are welcome! Please feel free to submit pull requests or open issues for:\n\n- Bug fixes\n- Feature enhancements\n- Documentation improvements\n- Test coverage expansion\n\n## 📝 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## ⚠️ Disclaimer\n\nWhile Dataset Miner strives for accuracy in Q\u0026A pair generation, the output should be reviewed for quality and appropriateness before use in production systems or datasets. The generated content depends on the AI model used and may require manual verification for critical applications.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrissmartin%2Fdataset-miner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrissmartin%2Fdataset-miner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrissmartin%2Fdataset-miner/lists"}