{"id":15515031,"url":"https://github.com/ucbepic/docetl","last_synced_at":"2026-10-28T20:22:42.637Z","repository":{"id":257623084,"uuid":"826111692","full_name":"ucbepic/docetl","owner":"ucbepic","description":"A system for agentic LLM-powered data processing and ETL","archived":false,"fork":false,"pushed_at":"2026-10-09T22:53:17.000Z","size":89215,"stargazers_count":4138,"open_issues_count":47,"forks_count":452,"subscribers_count":32,"default_branch":"main","last_synced_at":"2026-10-22T11:09:29.269Z","etag":null,"topics":["agents","data","data-pipelines","document-analysis","document-processing","elt","etl","llm","python","semantic-data","unstructured-data","unstructured-data-analysis","workflow"],"latest_commit_sha":null,"homepage":"https://docetl.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ucbepic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"claude":null,"gemini":null,"cursor":null,"copilot":null,"dco":null,"cla":null,"disclosure":null}},"created_at":"2024-07-09T05:57:16.000Z","updated_at":"2026-10-22T07:44:06.000Z","dependencies_parsed_at":"2026-08-10T16:24:17.092Z","dependency_job_id":null,"html_url":"https://github.com/ucbepic/docetl","commit_stats":null,"previous_names":["shreyashankar/docetl"],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/ucbepic/docetl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbepic%2Fdocetl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbepic%2Fdocetl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbepic%2Fdocetl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbepic%2Fdocetl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ucbepic","download_url":"https://codeload.github.com/ucbepic/docetl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ucbepic%2Fdocetl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":345082042,"owners_count":38477967,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-22T02:00:06.372Z","response_time":122,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","data","data-pipelines","document-analysis","document-processing","elt","etl","llm","python","semantic-data","unstructured-data","unstructured-data-analysis","workflow"],"created_at":"2024-10-02T10:01:06.242Z","updated_at":"2026-10-22T20:16:24.613Z","avatar_url":"https://github.com/ucbepic.png","language":"Python","funding_links":[],"categories":["Python","Machine Learning and AI","Platforms","5. Retrieval-Augmented Generation (RAG) \u0026 Knowledge","Document Processing / AI Pipelines"],"sub_categories":["Data Science and Analytics","Open Source Platforms"],"readme":"\u003cdiv align=\"center\"\u003e\n\n# DocETL: Declarative \u0026 Agentic Map-Reduce\n\n[![Website](https://img.shields.io/badge/Website-docetl.org-blue)](https://docetl.org)\n[![Documentation](https://img.shields.io/badge/Docs-ucbepic.github.io/docetl-green)](https://ucbepic.github.io/docetl)\n[![Discord](https://img.shields.io/discord/1285485891095236608?label=Discord\u0026logo=discord)](https://discord.gg/fHp7B2X3xx)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n[What is DocETL](#what-is-docetl) · [Install](#install) · [Python API](#python-api-recommended) · [YAML](#yaml-low-code) · [DocWrangler UI](#docwrangler-ui) · [Docs](#documentation)\n\n\u003c/div\u003e\n\n---\n\n## What is DocETL\n\nDocETL helps you process large collections of data (structured and unstructured) with LLMs. You write each operation in natural language, e.g., \"pull out every complaint in this ticket,\" and DocETL\n\n- provides the operators you need (map, reduce, filter, and more) and orchestrates them, parallelizing work across your data,\n- optimizes your pipeline automatically, swapping models, rewriting prompts, decomposing operations, and replacing subtasks with code wherever possible, to raise accuracy and cut cost, and\n- returns tables, easy to query in your favorite database.\n\nWithout DocETL, you write each LLM call yourself, wire them together, and tune the result for accuracy, cost, and latency by hand.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/assets/docetl-overview.svg\" alt=\"DocETL pipeline overview\" width=\"720\"\u003e\u003c/p\u003e\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cstrong\u003eCLI\u003c/strong\u003e\u003cbr\u003e\n\u003cimg src=\"docs/assets/progress-view/tui-real-complete.png\" alt=\"DocETL CLI\" width=\"100%\"\u003e\n\u003c/td\u003e\n\u003ctd width=\"50%\"\u003e\n\u003cstrong\u003eDocWrangler UI\u003c/strong\u003e\u003cbr\u003e\n\u003cimg src=\"docs/assets/tutorial/one-operation.png\" alt=\"DocWrangler\" width=\"100%\"\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n---\n\n## Install\n\n```bash\npip install docetl\nexport OPENAI_API_KEY=your_key   # or any LLM provider key\n```\n\n---\n\n## Need Help Writing Your Pipeline?\n\nUse Claude Code (recommended): run `docetl install-skill` and describe your task. See the [quickstart](https://ucbepic.github.io/docetl/quickstart-claude-code/).\n\nIf you'd rather use ChatGPT or the Claude app, copy the prompt at [docetl.org/llms-full.txt](https://docetl.org/llms-full.txt) into the chat before describing your task.\n\n---\n\n## Python API (recommended)\n\nBest for production code, notebooks, and scripting. [Full guide](https://ucbepic.github.io/docetl/python/)\n\n```python\nimport docetl\n\ndocetl.default_model = \"gpt-4o-mini\"\ndocetl.rate_limits = {\n    \"llm_call\": [{\"count\": 500, \"per\": 1, \"unit\": \"minute\"}],\n    \"llm_tokens\": [{\"count\": 200_000, \"per\": 1, \"unit\": \"minute\"}],\n}\n\n# Classify support tickets, then summarize each category\npipeline = docetl.read_json(\"tickets.json\")\n\npipeline = pipeline.map(\n    prompt=\"Classify this support ticket: {{ input.text }}\",\n    output={\"schema\": {\"category\": \"str\", \"priority\": \"str\"}},\n)\n\npipeline = pipeline.reduce(\n    reduce_key=\"category\",\n    prompt=\"Summarize these tickets: {% for t in inputs %}{{ t.text }}{% endfor %}\",\n    output={\"schema\": {\"summary\": \"str\"}},\n)\n\npipeline.schema()  # {'category': 'str', 'summary': 'str'}\npipeline.show()  # run on 5 docs and print results\nrows = pipeline.collect()  # full run\nprint(f\"Cost: ${pipeline.total_cost:.4f}\")\n```\n\n---\n\n## YAML (low-code)\n\nDeclare your pipeline in a config file, no Python needed. [Tutorial](https://ucbepic.github.io/docetl/tutorial/)\n\n```yaml\ndatasets:\n  tickets:\n    type: file\n    path: tickets.json\n\ndefault_model: gpt-4o-mini\n\noperations:\n  - name: classify\n    type: map\n    prompt: \"Classify this support ticket and assign a priority level.\"\n    output:\n      schema:\n        category: str\n        priority: str\n\npipeline:\n  steps:\n    - name: triage\n      input: tickets\n      operations: [classify]\n  output:\n    type: file\n    path: output.json\n```\n\n```bash\ndocetl run pipeline.yaml\n```\n\n---\n\n## DocWrangler UI\n\nVisual playground for interactive prompt development. Edit prompts, see results in real time. Try it at [docetl.org/playground](https://docetl.org/playground) or [run it locally](https://ucbepic.github.io/docetl/playground/).\n\n---\n\n## Documentation\n\n| | |\n|---|---|\n| [Python API Guide](https://ucbepic.github.io/docetl/python/) | Frame API reference: operations, config, optimization |\n| [YAML Tutorial](https://ucbepic.github.io/docetl/tutorial/) | Step-by-step walkthrough of declarative pipelines |\n| [Operators](https://ucbepic.github.io/docetl/operators/map/) | Map, filter, reduce, resolve, split, gather, extract, and more |\n| [Optimization](https://ucbepic.github.io/docetl/optimization/python-api/) | Automatic cost-accuracy optimization with MOAR |\n| [DocWrangler Setup](https://ucbepic.github.io/docetl/playground/) | Run the interactive UI locally or via Docker |\n| [Claude Code Quick Start](https://ucbepic.github.io/docetl/quickstart-claude-code/) | Describe your task and let Claude build the pipeline |\n\n---\n\n## Community\n\n[Discord](https://discord.gg/fHp7B2X3xx) · [Conversation Generator](https://github.com/PassionFruits-net/docetl-conversation) · [Text-to-Speech](https://github.com/PassionFruits-net/docetl-speaker) · [YouTube Transcript Topics](https://github.com/rajib76/docetl_examples)\n\n---\n\n## Development\n\n```bash\ngit clone https://github.com/ucbepic/docetl.git \u0026\u0026 cd docetl\nmake install\nmake tests-basic  # \u003c $0.01 with OpenAI\n```\n\n---\n\n## Papers\n\nDocETL was created at the [EPIC Data Lab](https://epic.berkeley.edu/) and [Data Systems and Foundations](https://dsf.berkeley.edu/) group at UC Berkeley.\n\n**DocETL**, VLDB 2025 ([paper](https://arxiv.org/abs/2410.12189))\n\n```bibtex\n@article{shankar2025docetl,\n  title={DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing},\n  author={Shankar, Shreya and Chambers, Tristan and Shah, Tarak and Parameswaran, Aditya G and Wu, Eugene},\n  journal={Proceedings of the VLDB Endowment},\n  volume={18}, number={9}, pages={3035--3048}, year={2025}\n}\n```\n\n**DocWrangler**, UIST 2025, Best Paper Honorable Mention ([paper](https://arxiv.org/abs/2504.14764))\n\n```bibtex\n@inproceedings{shankar2025docwrangler,\n  title={Steering Semantic Data Processing With DocWrangler},\n  author={Shankar*, Shreya and Chopra*, Bhavya and Hasan, Mawil and Lee, Stephen and Hartmann, Bj{\\\"o}rn and Hellerstein, Joseph M and Parameswaran, Aditya G and Wu, Eugene},\n  booktitle={Proceedings of the ACM Symposium on User Interface Software and Technology (UIST)},\n  year={2025}\n}\n```\n\n**MOAR**, VLDB 2026 ([paper](https://arxiv.org/abs/2512.02289))\n\n```bibtex\n@article{wei2026moar,\n  title={Multi-Objective Agentic Rewrites for Unstructured Data Processing},\n  author={Wei*, Lindsey Linxi and Shankar*, Shreya and Zeighami, Sepanta and Chung, Yeounoh and Ozcan, Fatma and Parameswaran, Aditya G},\n  journal={Proceedings of the VLDB Endowment}, year={2026}\n}\n```\n\n*\\*Co-first authors*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucbepic%2Fdocetl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fucbepic%2Fdocetl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fucbepic%2Fdocetl/lists"}