{"id":50419047,"url":"https://github.com/michael-borck/extracta","last_synced_at":"2026-05-31T07:30:52.403Z","repository":{"id":354391342,"uuid":"1095746417","full_name":"michael-borck/extracta","owner":"michael-borck","description":"Analyzes academic content across multiple media types for research and integrity checking.","archived":false,"fork":false,"pushed_at":"2026-04-28T10:13:38.000Z","size":216,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-28T12:18:08.384Z","etag":null,"topics":["academic-integrity","ai-analysis","api","citation-analysis","cli-tool","content-analysis","edtech","modular-architecture","plagiarism-detection","python","research-tool"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michael-borck.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-13T13:16:52.000Z","updated_at":"2026-04-28T10:13:39.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/michael-borck/extracta","commit_stats":null,"previous_names":["michael-borck/extracta"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/michael-borck/extracta","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Fextracta","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Fextracta/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Fextracta/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Fextracta/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michael-borck","download_url":"https://codeload.github.com/michael-borck/extracta/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michael-borck%2Fextracta/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33723548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["academic-integrity","ai-analysis","api","citation-analysis","cli-tool","content-analysis","edtech","modular-architecture","plagiarism-detection","python","research-tool"],"created_at":"2026-05-31T07:30:51.209Z","updated_at":"2026-05-31T07:30:52.395Z","avatar_url":"https://github.com/michael-borck.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Extracta\n\n\u003e **Deprecated.** This package is no longer maintained.\n\u003e For document text extraction use [markitdown](https://github.com/microsoft/markitdown) directly.\n\u003e Existing installs continue to work but no further updates will be published.\n\n\u003c!-- BADGES:START --\u003e\n[![edtech](https://img.shields.io/badge/-edtech-4caf50?style=flat-square)](https://github.com/topics/edtech) [![academic-integrity](https://img.shields.io/badge/-academic--integrity-blue?style=flat-square)](https://github.com/topics/academic-integrity) [![ai-analysis](https://img.shields.io/badge/-ai--analysis-blue?style=flat-square)](https://github.com/topics/ai-analysis) [![api](https://img.shields.io/badge/-api-blue?style=flat-square)](https://github.com/topics/api) [![citation-analysis](https://img.shields.io/badge/-citation--analysis-blue?style=flat-square)](https://github.com/topics/citation-analysis) [![cli-tool](https://img.shields.io/badge/-cli--tool-blue?style=flat-square)](https://github.com/topics/cli-tool) [![content-analysis](https://img.shields.io/badge/-content--analysis-blue?style=flat-square)](https://github.com/topics/content-analysis) [![modular-architecture](https://img.shields.io/badge/-modular--architecture-blue?style=flat-square)](https://github.com/topics/modular-architecture) [![plagiarism-detection](https://img.shields.io/badge/-plagiarism--detection-blue?style=flat-square)](https://github.com/topics/plagiarism-detection) [![python](https://img.shields.io/badge/-python-3776ab?style=flat-square)](https://github.com/topics/python)\n\u003c!-- BADGES:END --\u003e\n\n**Modular Content Analysis Platform** for research, assessment, and academic integrity checking.\n\nExtracta provides a unified interface for extracting and analyzing content from diverse media types including documents, images, repositories, and web content. It supports both research-focused deep analysis and assessment-oriented quality evaluation, with specialized tools for academic integrity validation.\n\n## ✨ Key Features\n\n- **🧩 Modular Architecture**: Pluggable lenses and analyzers for different content types\n- **📚 Academic Integrity**: Citation-reference validation, bibliography checking, URL verification, AI conversation analysis\n- **🤖 AI Conversation Analysis**: Cognitive intent classification for AI-assisted learning assessment\n- **🔍 Multiple Analysis Modes**: Research and assessment workflows\n- **📄 Rich Content Support**: Text, images, documents, repositories, presentations, spreadsheets, AI conversations\n- **🎯 Rubric-Based Assessment**: Custom rubrics for structured evaluation\n- **🛡️ Security First**: Input sanitization, URL validation, malicious content detection\n- **🧠 Intelligent Analysis**: Pattern detection, quality scoring, integrity validation, learning pattern recognition\n- **💻 Multiple Interfaces**: CLI, Python API, and Web API\n- **🔧 Modern Python**: Built with uv, ruff, mypy, and pytest\n\n## Installation\n\n### From PyPI\n\n```bash\npip install extracta\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/michaelborck-education/extracta.git\ncd extracta\npip install -e .\n```\n\n### Optional Dependencies\n\nInstall with specific feature support:\n\n```bash\npip install extracta[audio]     # Audio processing (faster-whisper for Apple Silicon)\npip install extracta[video]     # Video processing\npip install extracta[text]      # Enhanced text analysis (spaCy, NLTK)\npip install extracta[image]     # Image analysis with OCR\npip install extracta[code]      # Code analysis\npip install extracta[citation]  # Academic integrity (CrossRef, URL validation)\npip install extracta[conversation]  # AI conversation analysis (Gemini default)\npip install extracta[openai]    # OpenAI LLM provider\npip install extracta[claude]    # Anthropic Claude LLM provider\npip install extracta[openrouter] # OpenRouter unified API\npip install extracta[api]       # Web API server (FastAPI, Uvicorn)\npip install extracta[all]       # All features\n```\n\n## Usage\n\n### Command Line\n\n#### Basic Content Analysis\n```bash\n# Analyze document for research insights\nextracta analyze research_paper.pdf --mode research --output analysis.json\n\n# Assess student submission quality\nextracta analyze essay.docx --mode assessment --output feedback.json\n\n# Analyze repository structure and content\nextracta analyze https://github.com/user/repo --mode assessment\n```\n\n#### Academic Integrity Checking\n```bash\n# Comprehensive citation and reference validation\nextracta citation analyze student_paper.pdf --output integrity_check.json\n\n# AI conversation cognitive intent analysis (with different LLM providers)\nextracta citation conversation chatgpt_export.json --provider gemini --output analysis.json\nextracta citation conversation chat.json --provider claude --model claude-3-sonnet-20240229\nextracta citation conversation chat.json --provider openai --model gpt-4\nextracta citation conversation chat.json --provider openrouter --model anthropic/claude-3-haiku\n\n# Results include:\n# - Citation-reference relationship validation\n# - Bibliography padding detection\n# - URL accessibility and domain reputation\n# - AI conversation learning pattern analysis\n# - Academic integrity scoring\n```\n\n### Python API\n\n#### Basic Content Analysis\n```python\nfrom extracta import TextAnalyzer\n\nanalyzer = TextAnalyzer()\nresult = analyzer.analyze(text_content, mode=\"research\")\nprint(result)\n```\n\n#### Academic Integrity Analysis\n```python\nfrom extracta.analyzers import CitationAnalyzer, ReferenceAnalyzer, URLAnalyzer, ConversationAnalyzer\n\n# Citation-reference validation\ncitation_analyzer = CitationAnalyzer()\ncitation_result = citation_analyzer.analyze(document_text)\n\n# Bibliography quality assessment\nreference_analyzer = ReferenceAnalyzer()\nreference_result = reference_analyzer.analyze(document_text)\n\n# URL validation and reputation checking\nurl_analyzer = URLAnalyzer()\nurl_result = url_analyzer.analyze(document_text)\n\n# AI conversation cognitive intent analysis (with different providers)\nconversation_analyzer = ConversationAnalyzer(provider=\"claude\", model=\"claude-3-sonnet-20240229\")\nconversation_result = conversation_analyzer.analyze(conversation_json_data)\n\n# Or use OpenAI\nconversation_analyzer = ConversationAnalyzer(provider=\"openai\", model=\"gpt-4\")\nconversation_result = conversation_analyzer.analyze(conversation_json_data)\n\n# Combined integrity score\nintegrity_score = citation_result['citation_analysis']['academic_integrity_score']\nlearning_quality = conversation_result['conversation_analysis']['learning_assessment']['learning_quality_score']\nprint(f\"Academic Integrity Score: {integrity_score}/100\")\nprint(f\"AI Learning Quality Score: {learning_quality}/100\")\n```\n\n### Grading and Assessment\n\n```python\nfrom extracta.grading.rubric_manager import RubricRepository, get_default_rubric\nfrom extracta.grading.feedback_generator import FeedbackGenerator\n\n# Load or create a rubric\nrepo = RubricRepository(\"rubrics\")\nrubric = get_default_rubric(\"academic\")  # or repo.load(\"my-rubric\")\n\n# Generate feedback based on analysis results\ngenerator = FeedbackGenerator()\nfeedback = generator.generate_feedback(\n    rubric=rubric,\n    analysis_data=analysis_result,\n    audience=\"student\",\n    detail=\"detailed\"\n)\n```\n\n## 🎓 Academic Integrity Features\n\nExtracta provides comprehensive tools for detecting academic integrity issues and validating scholarly work:\n\n### Citation Analysis\n- **Citation-Reference Validation**: Ensures all references have corresponding in-text citations\n- **Bibliography Padding Detection**: Identifies references without citations\n- **Citation Stuffing Detection**: Flags excessive citations in single sentences\n- **Style Recognition**: Supports APA, MLA, Chicago, Harvard, and Numeric styles\n\n### Reference Validation\n- **DOI Verification**: Validates Digital Object Identifiers with CrossRef API\n- **URL Accessibility**: Checks if referenced URLs are accessible (404 detection)\n- **Domain Reputation**: Analyzes source credibility (academic vs. commercial domains)\n- **Format Validation**: Ensures proper reference formatting and completeness\n\n### AI Conversation Analysis\n- **Cognitive Intent Classification**: Uses configurable LLM to classify user prompts as Delegation vs. Scaffolding\n- **Multi-Provider Support**: Gemini, OpenAI GPT, Anthropic Claude, OpenRouter unified API\n- **Learning Pattern Recognition**: Analyzes conversation flow for active learning behaviors\n- **Session Quality Scoring**: Provides learning quality assessment (0-100)\n- **Platform Support**: ChatGPT, Claude, Bard, and generic conversation formats\n\n### Security \u0026 Privacy\n- **Input Sanitization**: Detects and prevents malicious content, hidden text, and LLM jailbreaks\n- **URL Protection**: SSRF prevention with academic domain whitelisting\n- **Content Validation**: Size limits, encoding validation, and integrity checking\n- **Privacy First**: No data persistence, user-controlled processing, ephemeral analysis\n- **Safe Processing**: Static analysis only, no code execution or external script running\n\n### Repository Analysis\n- **WordPress Detection**: Identifies WordPress projects and analyzes themes/plugins\n- **Code Quality Assessment**: Evaluates repository structure and practices\n- **File Type Analysis**: Comprehensive analysis of all repository contents\n\n### Integrity Scoring\n- **Academic Integrity Score**: 0-100 scale based on multiple validation criteria\n- **Detailed Reporting**: Specific issues and recommendations\n- **Pattern Detection**: Identifies suspicious citation and reference patterns\n\n## Development\n\n### Setup\n\n```bash\n# Clone repository\ngit clone https://github.com/michaelborck-education/extracta.git\ncd extracta\n\n# Create virtual environment\nuv venv\nsource .venv/bin/activate\n\n# Install dependencies\nuv pip install -e \".[dev]\"\n```\n\n### Testing\n\n```bash\n# Run tests\npytest\n\n# With coverage\npytest --cov=extracta\n```\n\n### Linting and Type Checking\n\n```bash\n# Lint with ruff\nruff check .\n\n# Type check with mypy\nmypy extracta\n\n# Format code\nruff format .\n```\n\n### Building and Publishing\n\n```bash\n# Build package\nuv build\n\n# Publish to PyPI\nuv venv  # if not already\nsource .venv/bin/activate\nuv pip install twine\ntwine upload dist/* --repository pypi\n```\n\n## Project Structure\n\n```\nextracta/\n├── extracta/\n│   ├── lenses/              # Content extraction modules\n│   │   ├── audio_lens/      # Audio file processing\n│   │   ├── video_lens/      # Video file processing\n│   │   ├── image_lens/      # Image processing with OCR\n│   │   ├── document_lens/   # Text \u0026 Office document processing\n│   │   ├── presentation_lens/ # Presentation file analysis\n│   │   ├── repo_lens/       # Repository-level analysis\n│   │   └── base_lens.py     # Common lens interface\n│   ├── analyzers/           # Content analysis modules\n│   │   ├── text_analyzer/   # Text quality and readability\n│   │   ├── image_analyzer/  # Image quality assessment\n│   │   ├── citation_analyzer/ # Citation-reference validation\n│   │   ├── reference_analyzer/ # Bibliography quality assessment\n│   │   ├── url_analyzer/    # URL validation and reputation\n│   │   └── base_analyzer.py # Common analyzer interface\n│   ├── grading/             # Assessment and grading\n│   │   ├── rubric_manager/  # Rubric creation and management\n│   │   └── feedback_generator.py # AI-powered feedback\n│   ├── orchestration/       # Workflow management\n│   ├── shared/              # Common utilities\n│   └── cli/                 # Command-line interface\n├── tests/                   # Test suite\n├── docs/                    # Documentation\n├── examples/                # Usage examples\n├── pyproject.toml           # Package configuration\n└── README.md               # This file\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Make your changes\n4. Add tests\n5. Run the test suite\n6. Submit a pull request\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## 🚀 Current Status \u0026 Roadmap\n\n### ✅ Implemented Features\n- [x] **Text Analysis**: Readability, sentiment, vocabulary, quality metrics\n- [x] **Image Analysis**: OCR, quality assessment, accessibility\n- [x] **Document Processing**: PDF, DOCX, Office docs (PPTX, Excel, CSV)\n- [x] **Citation Validation**: Citation-reference relationships, academic integrity\n- [x] **Reference Analysis**: Bibliography quality, DOI validation, CrossRef integration\n- [x] **URL Validation**: Accessibility checking, domain reputation, robots.txt\n- [x] **AI Conversation Analysis**: Cognitive intent classification, learning pattern recognition\n- [x] **Repository Analysis**: GitHub repo analysis, WordPress detection\n- [x] **Rubric System**: Custom rubrics, structured assessment\n- [x] **CLI Interface**: Multiple commands for different analysis types\n- [x] **Web API**: REST API for integration\n- [x] **Python API**: Programmatic access\n\n### 🔄 In Development\n- [ ] **Audio Lens**: Speech-to-text, audio quality analysis\n- [ ] **Video Lens**: Frame analysis, transcript processing\n- [ ] **Code Analyzer**: Code quality metrics, best practices\n- [ ] **Screenshot Integration**: Visual URL validation\n- [ ] **Wayback Machine**: Archive URL checking\n\n### 📋 Future Enhancements\n- [ ] **URL Conversation Input**: Direct analysis of conversations from URLs (ChatGPT share links, etc.)\n- [ ] **GUI Application**: Web-based interface\n- [ ] **LMS Integration**: Canvas, Blackboard, Moodle\n- [ ] **Advanced ML Models**: Fine-tuned for educational content\n- [ ] **Collaborative Features**: Multi-user assessment workflows\n- [ ] **Plugin Architecture**: Custom lenses and analyzers","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-borck%2Fextracta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichael-borck%2Fextracta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichael-borck%2Fextracta/lists"}