{"id":32856175,"url":"https://github.com/datajuicer/data-juicer","last_synced_at":"2026-10-20T20:27:33.364Z","repository":{"id":185316735,"uuid":"673277958","full_name":"datajuicer/data-juicer","owner":"datajuicer","description":"Data processing for and with foundation models!  🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷","archived":false,"fork":false,"pushed_at":"2026-10-20T03:04:52.000Z","size":227806,"stargazers_count":7164,"open_issues_count":65,"forks_count":435,"subscribers_count":20,"default_branch":"main","last_synced_at":"2026-10-20T04:27:39.663Z","etag":null,"topics":["data","data-analysis","data-pipeline","data-processing","data-science","data-visualization","foundation-models","instruction-tuning","large-language-models","llm","llms","multi-modal","pre-training","synthetic-data"],"latest_commit_sha":null,"homepage":"https://datajuicer.github.io/data-juicer/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datajuicer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"accessibility":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"claude":null,"gemini":null,"cursor":null,"copilot":null,"dco":null,"cla":null,"disclosure":null}},"created_at":"2023-08-01T09:16:41.000Z","updated_at":"2026-10-20T03:44:11.000Z","dependencies_parsed_at":"2026-10-20T04:37:59.715Z","dependency_job_id":null,"html_url":"https://github.com/datajuicer/data-juicer","commit_stats":{"total_commits":223,"total_committers":28,"mean_commits":7.964285714285714,"dds":0.7713004484304933,"last_synced_commit":"8e9b4c0b21d099fb950e74e6e176a5f730dd39eb"},"previous_names":["alibaba/data-juicer","modelscope/data-juicer","datajuicer/data-juicer"],"tags_count":32,"template":false,"template_full_name":null,"purl":"pkg:github/datajuicer/data-juicer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajuicer%2Fdata-juicer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajuicer%2Fdata-juicer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajuicer%2Fdata-juicer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajuicer%2Fdata-juicer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datajuicer","download_url":"https://codeload.github.com/datajuicer/data-juicer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datajuicer%2Fdata-juicer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":344872684,"owners_count":38388486,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-10-03T21:59:58.778Z","status":"online","status_checked_at":"2026-10-20T02:00:06.669Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-analysis","data-pipeline","data-processing","data-science","data-visualization","foundation-models","instruction-tuning","large-language-models","llm","llms","multi-modal","pre-training","synthetic-data"],"created_at":"2025-11-08T15:00:38.265Z","updated_at":"2026-10-20T20:27:33.343Z","avatar_url":"https://github.com/datajuicer.png","language":"Python","funding_links":[],"categories":["data","llm","Python","Core Machine Learning Research","Machine Learning and AI"],"sub_categories":["Data-Centric AI and Data Curation","Data Science and Analytics"],"readme":"#  Data-Juicer: The Data Operating System for the Foundation Model Era\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://pypi.org/project/py-data-juicer\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/py-data-juicer?logo=pypi\u0026color=026cad\" alt=\"PyPI\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pepy.tech/projects/py-data-juicer\"\u003e\u003cimg src=\"https://static.pepy.tech/personalized-badge/py-data-juicer?period=total\u0026units=INTERNATIONAL_SYSTEM\u0026left_color=grey\u0026right_color=green\u0026left_text=downloads\" alt=\"Downloads\"\u003e\u003c/a\u003e\n   \u003ca href=\"https://hub.docker.com/r/datajuicer/data-juicer\"\u003e\u003cimg src=\"https://img.shields.io/docker/v/datajuicer/data-juicer?logo=docker\u0026label=Docker\u0026color=498bdf\" alt=\"Docker\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://datajuicer.github.io/data-juicer/\"\u003e\u003cimg src=\"https://img.shields.io/badge/📖_Docs-Website-026cad\" alt=\"Docs\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://datajuicer.github.io/data-juicer/en/main/docs/Operators.html\"\u003e\u003cimg src=\"https://img.shields.io/badge/🧩_Operators-200+-blue\" alt=\"Operators\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/datajuicer/data-juicer-hub\"\u003e\u003cimg src=\"https://img.shields.io/badge/🍳_Recipes-50+-brightgreen\" alt=\"Recipes\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://datajuicer.github.io/data-juicer/zh_CN/main/index_ZH.html\"\u003e\u003cimg src=\"https://img.shields.io/badge/🇨🇳_文档-主页-red\" alt=\"Chinese\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2501.14755\"\u003e\u003cimg src=\"https://img.shields.io/badge/NeurIPS'25_Spotlight-2.0-B31B1B?logo=arxiv\" alt=\"Paper\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/datajuicer/data-juicer\"\u003e\n    \u003cimg src=\"https://img.shields.io/endpoint?style=flat\u0026url=https%3A%2F%2Fgist.githubusercontent.com%2FHYLcool%2Ff856b14416f08f73d05d32fd992a9c29%2Fraw%2Ftotal_cov.json\u0026label=coverage\u0026logo=codecov\u0026color=4c1\" alt=\"Coverage\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cb\u003eMultimodal | Cloud-Native | AI-Ready | Large-Scale \u003c/b\u003e\n\u003c/p\u003e\n\nData-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as *composable infrastructure*—providing modular building blocks to clean, synthesize, and analyze data across the entire AI lifecycle, unlocking latent value in every byte.\n\nWhether you're deduplicating web-scale pre-training corpora, curating agent interaction traces, or preparing domain-specific RAG indices, DJ scales seamlessly from your laptop to thousand-node clusters—no glue code required.\n\n\u003e **Alibaba Cloud PAI** has deeply integrated Data-Juicer into its data processing products.  See **[Quickly submit a DataJuicer job](https://www.alibabacloud.com/help/en/pai/user-guide/quickly-submit-a-datajuicer-task)**.\n\n\u003e 🧃 **Juicer**, our natural-language data-refinement model, turns cleaning instructions, filtering rules, and semantic-tagging requirements into structured outputs. Try it on [HuggingFace](https://huggingface.co/datajuicer/Juicer-35B-A3B) or [ModelScope](https://www.modelscope.cn/models/Data-Juicer/Juicer-35B-A3B), or see the **[Juicer docs](docs/Juicer.md)**.\n\n---\n\n## 🚀 Quick Start\n\n**Zero-install exploration**:\n- [JupyterLab Playground with Tutorials](http://8.138.149.181/)\n- [Ask DJ Copilot](https://datajuicer.github.io/data-juicer/en/main/docs_index.html)\n\n**Install \u0026 run**:\n```bash\nuv pip install py-data-juicer\ndj-process --config demos/process_simple/process.yaml\n```\n\n**Or compose in Python**:\n```python\nfrom data_juicer.core.data import NestedDataset\nfrom data_juicer.ops.filter import TextLengthFilter\nfrom data_juicer.ops.mapper import WhitespaceNormalizationMapper\n\nds = NestedDataset.from_dict({\n    \"text\": [\"Short\", \"This passes the filter.\", \"Text   with   spaces\"]\n})\nres_ds = ds.process([\n    TextLengthFilter(min_len=10),\n    WhitespaceNormalizationMapper()\n])\n\nfor s in res_ds:\n    print(s)\n```\n\n\n---\n\n## ✨ Why Data-Juicer?\n\n### 1. Modular \u0026 Extensible Architecture\n- **200+ operators** spanning text, image, audio, video, and multimodal data\n- **Recipe-first**: Reproducible YAML pipelines you can version, share, and fork like code\n- **Composable**: Drop in a single operator, chain complex workflows, or orchestrate full pipelines\n- **Hot-reload**: Iterate on operators without pipeline restarts\n\n### 2. Full-Spectrum Data Intelligence\n- **Foundation Models**: Pre-training, fine-tuning, RL, and evaluation-grade curation\n- **Agent Systems**: Clean tool traces, structure context, de-identification, and quality gating\n- **RAG \u0026 Analytics**: Extraction, normalization, semantic chunking, deduplication, and data profiling\n\n\n### 3. Production-Ready Performance\n- **Scale**: Process 70B samples in 2h on 50 Ray nodes (6400 cores)\n- **Efficiency**: Deduplicate 5TB in 2.8h using 1280 cores\n- **Optimization**: Automatic OP fusion (2-10x speedup), adaptive parallelism, CUDA acceleration, robustness\n- **Observability**: Built-in tracing for debugging, auditing, and iterative improvement\n\n\u003e *⭐ If Data-Juicer saved you time or improved your data work, please consider starring the repo.* It helps more people discover the project and keeps you notified of new releases and features.\n\n---\n\n## 📰 News\n\n\u003cdetails open\u003e\n\u003csummary\u003e[2026-09-08] Release v1.6.0: \u003cb\u003eJuicer Model Release; Cluster-Aware Partitioning; Config Validation; LiteLLM Backend\u003c/b\u003e\u003c/summary\u003e\n\n* 🧮 *Cluster-Aware Partitioning* — Automatic partition counts use live Ray cluster resources. Manual `partition.size` targets split data at row boundaries, including inputs with fewer blocks than partitions.\n* ✅ *Config Validation* — Pipeline preflight catches invalid operator settings and executor/schema mismatches before processing. Reader defaults now apply consistently across execution and analysis.\n* 🔌 *LiteLLM Backend* — Select `api_backend=\"litellm\"` in `prepare_api_model` for chat, embedding, and Responses requests through provider-specific model routing; the existing OpenAI-compatible backend remains the default.\n* 📚 *Documentation Refresh* — Rewritten English and Chinese guides cover installation, processing, analysis, configuration, export, and the playground. Added documentation for 28 existing operators, corrected examples, and separated guide and API navigation with incremental versioned documentation builds.\n* 🗄️ *Unified Remote Export* — Local, S3, and HDFS export share filesystem dispatch; JSONL export now serializes Python dates and datetimes in ISO format.\n* 🖼️ *Image OHEM Selector* — New `image_ohem_selector` selects high-loss image samples using a user-supplied scoring function and a top-k or ratio budget.\n* ⚡ *Bounded Tokenizer Batches* — Token-count filters limit tokenizer batch sizes to reduce peak memory on long inputs.\n* 🔧 *Robustness Fixes* — Fixed fused-filter cache isolation, MinHash state reuse and empty inputs, deduplicator execution-mode declarations, empty text chunks, gzip JSONL HPO sampling, and pandas extension-dtype handling. HPO modules can now be imported without starting a sweep.\n\u003c/details\u003e\n\n\u003cdetails open\u003e\n\u003csummary\u003e[2026-08-25] Juicer Model Release: \u003cb\u003eNatural-Language Data Refinement; Local Deployment; Playground\u003c/b\u003e\u003c/summary\u003e\n\n* 🧃 *Natural-Language Data Refinement* — [Juicer](docs/Juicer.md) follows natural-language instructions for text cleaning, filtering, and semantic labeling.\n* 🏠 *Local Deployment* — Download the [model](https://huggingface.co/datajuicer/Juicer-35B-A3B) for local deployment.\n* 🧪 *Playground* — Try data-refinement recipes in the [Juicer Playground](https://github.com/datajuicer/data-juicer-hub/tree/main/juicer_playground).\n\u003c/details\u003e\n\n\u003cdetails open\u003e\n\u003csummary\u003e[2026-08-07] Release v1.5.5: \u003cb\u003eExternal OP Plugins; HDFS I/O \u0026 Ray Data Optimizations; Elastic Multi-node Sharding\u003c/b\u003e\u003c/summary\u003e\n\n* 🧩 *External OP Plugins* — Third-party operators can now be shipped as standalone pip packages and auto-registered via Python entry points.\n* 🗄️ *HDFS I/O* — Added `hdfs://` support for dataset loading and exporting, aligned with the existing S3 path.\n* ⚡ *Ray Data Optimizations* — Removed eager actions that forced premature execution, migrated to the public `TaskPoolStrategy`/`ActorPoolStrategy` APIs, parallelized partitioned execution, preserved elastic actor pool concurrency, and kept partition checkpoints valid across block-layout changes.\n* 🌐 *Elastic Multi-node Sharding* — New runnable reference workflow under `demos/elastic_sharding/` that shards large JSONL datasets across nodes via node-local Ray executors, with retries and ordered merge.\n* 📊 *Distributed RayAnalyzer* — New `RayAnalyzer` computes and aggregates dataset stats with Ray native operators, avoiding pandas materialization.\n* 🧮 *Memory Optimizations* — Streamed n-gram counting in repetition filters and block-wise MinHash permutations cut peak memory (768 → 272 MiB RSS) without changing results.\n* 🔧 *Robustness \u0026 Dependency Fixes* — Fixed `text_chunk_mapper` delimiter leakage, `calibrate_response_mapper` `output_pattern` handling, and null captions in `image_diffusion_mapper`; added membership operators to `general_field_filter`; relaxed `numpy`/`fsspec`/`pandas` bounds for Python 3.13+ and pyarrow\u003e=17.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[2026-07-17] Release v1.5.4: \u003cb\u003eHumanVBench Video OPs; Batch-local Stage Fusion; Robustness Fixes\u003c/b\u003e\u003c/summary\u003e\n\n* 🧑‍🤝‍🧑 *New OPs* — Added 9 human-centric video understanding operators (human track extraction, active-speaker detection, audio ASR, speech emotion \u0026 age/gender detection, face demographic \u0026 attribute/emotion captioning, face-ratio filtering) for building HumanVBench (CVPR'26)-style pipelines.\n* ⚡ *Batch-local Stage Fusion* — New `FusedSequentialBatchOp` fuses consecutive OPs within a batch to cut inter-op overhead and speed up sequential processing.\n* 🔧 *Robustness \u0026 Install Fixes* — Fixed Ray deduplicator shared-state handling, resolved a relative-path issue in the Ray checkpoint writer, made `clean_html_mapper` robust to null text values, handled boolean stat columns in `ColumnWiseAnalysis`, and unblocked ARM64 (aarch64) installation via precise `decord`/`torchcodec` platform markers.\n* 🧪 *Test Coverage \u0026 Cleanup* — Expanded tests for utility functions and model handling, plus assorted code cleanup.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[2026-06-26] Release v1.5.3: \u003cb\u003eVLA Ops Enhancements; Ray Repartition Pipeline; Scalability \u0026 Robustness\u003c/b\u003e\u003c/summary\u003e\n\n* 🤖 *VLA Ops Enhancements* — Expanded embodied-AI processing with 10+ new/renamed VLA operators (camera calibration via DeepCalib/DroidCalib/MoGe, atomic action segmentation, hand action computation \u0026 motion smoothing, clip reassembly, trajectory overlay, LeRobot export) and a complete VLA pipeline demo.\n* 🔄 *Ray Repartition Pipeline* — New `ray_repartition_pipeline` for dataset-level block repartitioning in Ray mode.\n* ⚡ *Scalable Ray Data Reads* — Wired `override_num_blocks` through the full call chain for controlling block parallelism on PB-scale datasets.\n* 🧪 *Test Coverage Expansion* — Added 409 new test cases across 18 test files.\n* 🐳 *Stability \u0026 Robustness Fixes* — JSONStreamDatasource schema unification, OP env version resolution, FUSE-safe rmtree for PartitionedRayExecutor, deprecated model name updates, and num_proc handling fixes.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[2026-05-29] Release v1.5.2: \u003cb\u003eSemantic LLM OPs, Cross-doc Line Dedup \u0026 Leaner Dependencies\u003c/b\u003e\u003c/summary\u003e\n\n* 🧹 *New Deduplicator* — Added `DocumentLineDeduplicator` for cross-document line-level dedup, removing boilerplate lines (templates, copyright notices, navigation bars) by global document frequency.\n* 🤖 *Agent Data Quality Toolkit* — Shipped interaction-quality OPs \u0026 recipe, a bad-case HTML report, and more robust JSONL / HuggingFace meta loading.\n* 📦 *Leaner \u0026 Faster Install* — Slimmed the default dependency set (Ray, audio, spaCy, av, etc. moved to on-demand extras) to speed up installation.\n* 🐳 *Stability \u0026 Robustness Fixes* — Library-safe error handling (raise over `exit(1)`), Ray init/temp-dir fixes, valid API params (drop invalid `max_new_tokens`), PyArrow 20+ batch JSON reading, local-path aesthetics model support, and more performance/bug fixes.\n* 🧠 *Semantic LLM Operators* — Introduced `llm_extract_mapper`, `llm_condition_filter`, and `llm_structured_ops` with unified `llm_*` naming and configurable inference strategies (join/agg/top-k planned).\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[2026-03-17] Release v1.5.1: \u003cb\u003eLaTeX OPs; Compressed Format Support; Operator Robustness Fixes\u003c/b\u003e\u003c/summary\u003e\n\n* 📄 Two new LaTeX-focused mapper OPs shipped, extending data-juicer's document processing capabilities to handle `.tex` archives and figure contexts.\n* 🗜️ Compressed dataset format support: `json[l].gz` files can now be loaded directly, and Ray datasets gain proper support for reading compressed JSON files.\n* 📚 New documentation added covering cache, export, and tracing workflows to help users better understand and debug data processing pipelines.\n* 🤖 Major refactor and upgrade of data-juicer-agents completed: The project architecture and CLI/session capabilities were comprehensively redesigned for better maintainability and extensibility. See [date-juicer-agents](https://github.com/datajuicer/data-juicer-agents) for more details.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[2026-02-12] Release v1.5.0: \u003cb\u003ePartitioned Ray Executor, OP-level Env Management, and More Embodied-AI OPs\u003c/b\u003e\u003c/summary\u003e\n\n- 🚀 *Enhanced Distributed Execution Framework* -- Introduced partitioned Ray executor and OP-level isolated environments to improve fault tolerance, scalability, and dependency conflict resolution.\n- 🤖 *Expanded Embodied AI Video Processing* -- Added specialized operators for camera calibration, video undistortion, hand reconstruction, and pose estimation to strengthen multi-view video handling.\n- 💪🏻 *System Performance \u0026 Developer Experience Optimizations* -- Enabled batch inference, memory/log reduction, core logic refactoring, and updated documentation/templates.\n- 🐳 *Critical Bug Fixes \u0026 Stability Improvements* -- Resolved duplicate tracking, parameter conflicts, homepage rendering issues, and outdated docs for higher reliability.\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e[2026-02-02] Release v1.4.6: \u003cb\u003eCopilot, Video Bytes I/O \u0026 Ray Tracing \u003c/b\u003e\u003c/summary\u003e\n\n- 🤖 *Q\u0026A Copilot* —  Now live on our [Doc Site](https://datajuicer.github.io/data-juicer/en/main/index.html) | [DingTalk](https://qr.dingtalk.com/action/joingroup?code=v1,k1,N78tgW54U447gJP5aMC95B6qgQhlkVQS4+dp7qQq6MpuRVJIwrSsXmL8oFqU5ajJ\u0026_dt_no_comment=1\u0026origin=11?) | [Discord](https://discord.gg/ngQbB9hEVK). Feel free to ask anything related to Data-Juicer ecosystem!\n    - Check 🤖 [Data-Juicer Agents](https://github.com/datajuicer/data-juicer-agents/blob/main) | 📃 [Deploy-ready codes](https://github.com/datajuicer/data-juicer-agents/blob/main/qa-copilot) | 🎬[ More demos](https://github.com/datajuicer/data-juicer-agents/blob/main/qa-copilot/DEMO.md) for more details.\n- 🎬 *Video Bytes I/O* — Direct bytes processing for video pipelines\n- 🫆 *Ray Mode Tracer* — Track changed samples in distributed processing\n- 🐳 *Enhancements \u0026 fixes* — refreshed Docker image, small perf boosts, GitHub Insights traffic workflow, Ray compatibility updates, and bug/doc fixes.\n\u003c/details\u003e\n\nView [All Release](https://github.com/datajuicer/data-juicer/releases) and [News Archive](docs/news.md)\n\n---\n\n## 🔌 Users \u0026 Ecosystems\n\u003e The below list focuses on *developer-facing integration and usages* in *alphabetical order*.\n\u003e Missing your project / name? Feel free to [open a PR](https://github.com/datajuicer/data-juicer/pulls) or [reach out](#contributing--community).\n\nData-Juicer plugs into your existing stack and evolves with community contributions:\n\n### Extensions\n- **[data-juicer-agents](https://github.com/datajuicer/data-juicer-agents)** — DJ Copilot and agentic workflows\n- **[data-juicer-hub](https://github.com/datajuicer/data-juicer-hub)** — Community recipes and best practices\n- **[data-juicer-sandbox](https://github.com/datajuicer/data-juicer-sandbox)** — Data-model co-development with feedback loops\n\n\n### Frameworks \u0026 Platforms\n[AgentScope](https://github.com/agentscope-ai/agentscope) · [Apache Arrow](https://github.com/apache/arrow) · [Apache HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html) · [Apache Hudi](https://hudi.apache.org/) · [Apache Iceberg](https://iceberg.apache.org/) · [Apache Paimon](https://paimon.apache.org/) · [Alibaba PAI](https://www.alibabacloud.com/en/product/machine-learning?_p_lc=1) · [Delta Lake](https://delta.io/) · [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) · [EasyAnimate](https://github.com/aigc-apps/EasyAnimate) · [Eval-Scope](https://github.com/modelscope/evalscope) · [Huawei Ascend](https://www.huawei.com/en/products/cloud-computing-dc/atlas/ascend) · [Hugging Face](https://huggingface.co/) · [LanceDB](https://lancedb.github.io/lance/) · [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) · [ModelScope](https://modelscope.cn/) · [ModelScope Swift](https://github.com/modelscope/ms-swift) · [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) · [Ray](https://docs.ray.io/) · [RM-Gallery](https://github.com/modelscope/RM-Gallery) · [Trinity-RFT](https://github.com/modelscope/Trinity-RFT) · [Volcano Engine](https://www.volcengine.com/)\n\n### Industry\nAlibaba Group, Ant Group, BYD Auto, ByteDance, DTSTACK, JD.com, NVIDIA, OPPO, Xiaohongshu, Xiaomi, Ximalaya, and more.\n\n### Academia\nCAS, Nanjing University, Peking University, RUC, Tsinghua University, UCAS, Zhejiang University, and more.\n\n\n###  Contributing \u0026 Community\nWe believe in *building together*. Whether you're fixing a typo, crafting a new operator, or sharing a breakthrough recipe, every contribution shapes the future of data processing.\n\nWe welcome contributions at all levels:\n- **[Good First Issues](https://github.com/datajuicer/data-juicer/labels/good%20first%20issue)** — Add operators, improve docs, report issues, or fix bugs\n- **[Developer Guide](https://datajuicer.github.io/data-juicer/en/main/docs/DeveloperGuide.html)** — Optimize engines, add features, or enhance core infrastructure\n- **[DJ-Hub](https://github.com/datajuicer/data-juicer-hub)** — Share knowledge: recipes, papers, and best practices\n- **Connect**: [Slack](https://join.slack.com/t/data-juicer/shared_invite/zt-23zxltg9d-Z4d3EJuhZbCLGwtnLWWUDg) · [DingTalk](https://qr.dingtalk.com/action/joingroup?code=v1,k1,N78tgW54U447gJP5aMC95B6qgQhlkVQS4+dp7qQq6MpuRVJIwrSsXmL8oFqU5ajJ\u0026_dt_no_comment=1\u0026origin=11?) · [Discord](https://discord.gg/ngQbB9hEVK)\n\n| Discord | DingTalk |\n|:---:|:---:|\n| \u003cimg src=\"https://gw.alicdn.com/imgextra/i1/O1CN011Oj8CB1f8Bw5JpgJA_!!6000000003961-0-tps-762-769.jpg\" width=\"100\"\u003e | \u003cimg src=\"https://gw.alicdn.com/imgextra/i3/O1CN01bBPoaX1EwZsiYudtd_!!6000000000416-2-tps-656-660.png\" width=\"100\"\u003e |\n\n\nData-Juicer is made possible by the users and community:\n- **Initiated by**: Alibaba Tongyi Lab\n- **Co-developed with**: Alibaba Cloud PAI, Anyscale (Ray team), Sun Yat-sen University, NVIDIA (NeMo team), and [contributors worldwide](https://github.com/datajuicer/data-juicer/graphs/contributors)\n- **Inspired by**: Apache Arrow, Ray, Hugging Face Datasets, BLOOM, RedPajama-Data, ...\n\n---\n\n\n## Documentation\n\nFor detailed documentation, please see [here](https://datajuicer.github.io/data-juicer/en/main/docs_index.html).\n\n**Quick Links:**\n- **[operator zoo](https://datajuicer.github.io/data-juicer/en/main/docs/Operators.html)** — Browse 200+ operators with examples\n- **[Agent interaction quality \u0026 bad-case](demos/agent/README.md)** — In-repo recipe, JSONL pipeline, HTML report (`demos/agent/`; operators such as `agent_bad_case_signal_mapper` are also listed in [docs/Operators.md](docs/Operators.md))\n- **[data-juicer-hub](https://github.com/datajuicer/data-juicer-hub)** — Community-driven recipes and best practices\n- **[developer guide](https://datajuicer.github.io/data-juicer/en/main/docs/DeveloperGuide.html)** — Build your own code and contribute to DJ\n- **[data-juicer-cookbook](https://datajuicer.github.io/data-juicer/en/main/docs/tutorial/DJ-Cookbook.html)** — resource archive\n- **[awesome_llm_data](https://datajuicer.github.io/data-juicer/en/main/docs/awesome_llm_data)** —  “Awesome List” for data-model co-development\n\n\n---\n\n## 📄 License \u0026 Attribution\n\nData-Juicer is released under the [Apache License 2.0](LICENSE).\nAttribution is appreciated: please use our [badge](https://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/data_juicer/assets/DJ-Org-Logo.jpeg), or text as \"This project uses Data-Juicer: https://github.com/datajuicer\".\n\n---\n\n## 📖 Citation\nIf you find Data-Juicer useful in your work, please cite:\n\n```bibtex\n@inproceedings{djv1,\n  title={Data-Juicer: A One-Stop Data Processing System for Large Language Models},\n  author={Chen, Daoyuan and Huang, Yilun and Ma, Zhijian and Chen, Hesen and Pan, Xuchen and Ge, Ce and Gao, Dawei and Xie, Yuexiang and Liu, Zhaoyang and Gao, Jinyang and Li, Yaliang and Ding, Bolin and Zhou, Jingren},\n  booktitle={SIGMOD},\n  year={2024}\n}\n\n@article{djv2,\n  title={Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models},\n  author={Chen, Daoyuan and Huang, Yilun and Pan, Xuchen and Jiang, Nana and Wang, Haibin and Zhang, Yilei and Ge, Ce and Chen, Yushuo and Zhang, Wenhao and Ma, Zhijian and Huang, Jun and Lin, Wei and Li, Yaliang and Ding, Bolin and Zhou, Jingren},\n  journal={NeurIPS},\n  year={2025}\n}\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eMore Publications\u003c/b\u003e (Click to expand)\u003c/summary\u003e\n\n- (ICML'25 Spotlight) [Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development](https://arxiv.org/abs/2407.11784)\n\n- (CVPR'25) [ImgDiff: Contrastive Data Synthesis for Vision Large Language Models](https://arxiv.org/abs/2408.04594)\n\n- (TPAMI'25) [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://arxiv.org/abs/2407.08583)\n\n- (NeurIPS'25) [Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data](https://arxiv.org/abs/2502.04380)\n\n- (NeurIPS'25) [MindGYM: What Matters in Question Synthesis for Thinking-Centric Fine-Tuning?](https://arxiv.org/abs/2503.09499)\n\n- (CVPR'26) [HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data](https://arxiv.org/abs/2412.17574)\n\n- (ICML'26) [DetailMaster: Can Your Text-to-Image Model Handle Long Prompts?](https://www.arxiv.org/abs/2505.16915)\n\n- (Data Scaling) [BiMix: A Bivariate Data Mixing Law for Language Model Pretraining](https://arxiv.org/abs/2405.14908)\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatajuicer%2Fdata-juicer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatajuicer%2Fdata-juicer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatajuicer%2Fdata-juicer/lists"}