{"id":29182137,"url":"https://github.com/teilomillet/kushim","last_synced_at":"2026-03-15T23:01:51.685Z","repository":{"id":299900704,"uuid":"1001956132","full_name":"teilomillet/kushim","owner":"teilomillet","description":"eval creator","archived":false,"fork":false,"pushed_at":"2025-06-27T09:12:17.000Z","size":517,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-27T09:32:01.746Z","etag":null,"topics":["dataset","dataset-generation","eval","evaluation","evaluation-framework","llm","openai"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/teilomillet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-14T12:00:47.000Z","updated_at":"2025-06-23T04:21:05.000Z","dependencies_parsed_at":null,"dependency_job_id":"80bfafe1-3fe4-4fa1-80f7-1b197c3a7592","html_url":"https://github.com/teilomillet/kushim","commit_stats":null,"previous_names":["teilomillet/kushim"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/teilomillet/kushim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fkushim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fkushim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fkushim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fkushim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/teilomillet","download_url":"https://codeload.github.com/teilomillet/kushim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/teilomillet%2Fkushim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274291823,"owners_count":25258162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","dataset-generation","eval","evaluation","evaluation-framework","llm","openai"],"created_at":"2025-07-01T20:02:08.359Z","updated_at":"2025-09-19T18:27:59.269Z","avatar_url":"https://github.com/teilomillet.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kushim: A Framework for Verifiable, Self-Optimizing LLM Evaluation Datasets\n\nKushim is a framework for generating high-quality, verifiable Question \u0026 Answer datasets. In an era of generative models, creating reliable evaluation data is one of the biggest challenges. Kushim addresses this by providing an end-to-end workflow built on two core principles: **verifiability by design** and **self-optimizing quality**.\n\nIt's not just about generating data; it's about generating *trustworthy* data that gets *better* on its own.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"500px-Kushim_Illustration.png\" width=\"500\" alt=\"Kushim Illustration\"/\u003e\n\u003c/p\u003e\n\n## The Kushim Philosophy: Core Concepts\n\n### 1. Verifiable by Design\n\nThe biggest risk with synthetic data is factual inconsistency. Kushim is built to mitigate this risk. Every single question-answer pair generated by the pipeline is subjected to a strict validation step. An LLM-based validator checks if the generated answer is factually and unambiguously supported by the original source text. If a pair fails this check, it's discarded.\n\nThis ensures that your final dataset isn't just a collection of plausible-sounding questions, but a set of verifiable facts grounded in a source of truth.\n\n### 2. Self-Optimizing Quality with DSPy\n\nA static, one-size-fits-all prompt is not optimal. The best way to phrase a question depends on the source material. Kushim leverages the power of **DSPy** to create a self-improving pipeline.\n\nInstead of just running a prompt, Kushim can \"compile\" it. It uses DSPy's optimizers (`teleprompters`) to:\n1.  Generate a small training set from your source documents.\n2.  Test multiple variations of prompts to see which ones produce the highest-quality, most verifiable Q\u0026A pairs *for your specific data*.\n3.  Save this \"compiled\" program, which contains the optimized, high-performance prompts.\n\nThis means Kushim learns from your data to improve its own performance, leading to a significantly higher-quality final dataset.\n\n## How It Works: The Kushim Pipeline Workflow\n\nThe Kushim pipeline integrates these concepts into an efficient, streaming workflow that proceeds in the following stages:\n\n1.  **Source \u0026 Fetch**: The process begins by fetching raw documents from a designated `Source`, such as a Wikipedia article or a local file directory.\n\n2.  **Chunking**: The fetched documents are broken down into smaller, manageable text chunks. This is a standard practice in RAG-style pipelines and prepares the data for the generation models.\n\n3.  **Self-Optimization (A One-Time \"Compile\" Step)**: This is the heart of Kushim's quality assurance process. Instead of using a static prompt, the pipeline:\n    *   **Generates a Training Set**: It takes a small sample of the chunks to create a temporary training set.\n    *   **Optimizes Prompts**: It uses DSPy to test multiple prompt variations, identifying the one that produces the highest-quality, most verifiable Q\u0026A pairs for *your specific data*.\n    *   This \"compiled\" program, containing the optimized prompts, is saved and used for the main generation task.\n\n4.  **Generation**: Using the high-performance prompts from the compilation step, the pipeline generates question-answer pairs from all of the text chunks.\n\n5.  **Validation \u0026 Filtering**: Each generated Q\u0026A pair is rigorously validated. An LLM checks if the answer is factually supported by its original source chunk. Pairs that pass validation proceed to the final dataset; those that fail are discarded.\n\nThis multi-stage process ensures that the final output is not only relevant but also verifiable and of the highest possible quality.\n\n## Getting Started\n\nAfter installing Kushim (`uv add kushim` or `pip install kushim`), you can use its core components directly. The key is to instantiate the pipeline and run it. The optimization is handled for you—the first run compiles and saves the best prompts, and subsequent runs are fast.\n\n```python\n# A conceptual example of using the Kushim pipeline\nfrom kushim import pipeline, config, source\n\n# 1. Choose your source and model\ndata_source = source.WikipediaSource()\npipeline_config = config.KushimConfig(\n    model_name=\"openrouter/openai/gpt-4.1\",\n    fetch_kwargs={\"mode\": \"search\", \"query\": \"History of coffee\"}\n)\n\n# 2. Instantiate the pipeline\nkushim_pipeline = pipeline.KushimPipeline(\n    source=data_source,\n    config=pipeline_config\n)\n\n# 3. Run it!\n# This will automatically handle compiling and saving the optimized\n# generator to a .json file for you on the first run.\nvalidated_dataset, _ = kushim_pipeline.run(\n    optimize=True,\n    compiled_generator_path=\"compiled_coffee_generator.json\"\n)\n\nprint(validated_dataset)\n```\n\nFor complete, runnable scripts demonstrating the full dataset creation lifecycle (merging, encryption, and pushing to the Hugging Face Hub), please see the `examples/` directory in the [GitHub repository](https://github.com/teilomillet/kushim).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteilomillet%2Fkushim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fteilomillet%2Fkushim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fteilomillet%2Fkushim/lists"}