{"id":51873570,"url":"https://github.com/andyyaro/local-ai-orchestrator","last_synced_at":"2026-07-25T03:01:30.613Z","repository":{"id":369334713,"uuid":"1288392111","full_name":"andyyaro/local-ai-orchestrator","owner":"andyyaro","description":"Local-first multi-agent AI quality pipeline for drafting, critique, revision, judging, code verification, and run history on a Mac with Ollama.","archived":false,"fork":false,"pushed_at":"2026-07-04T19:58:50.000Z","size":1047,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-04T20:05:44.103Z","etag":null,"topics":["local-ai","multi-agent","ollama","portfolio-project","python","sqlite","streamlit"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andyyaro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-03T14:54:45.000Z","updated_at":"2026-07-04T19:57:50.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/andyyaro/local-ai-orchestrator","commit_stats":null,"previous_names":["andyyaro/local-ai-orchestrator"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/andyyaro/local-ai-orchestrator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyyaro%2Flocal-ai-orchestrator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyyaro%2Flocal-ai-orchestrator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyyaro%2Flocal-ai-orchestrator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyyaro%2Flocal-ai-orchestrator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andyyaro","download_url":"https://codeload.github.com/andyyaro/local-ai-orchestrator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyyaro%2Flocal-ai-orchestrator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35863583,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"online","status_checked_at":"2026-07-25T02:00:06.922Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["local-ai","multi-agent","ollama","portfolio-project","python","sqlite","streamlit"],"created_at":"2026-07-25T03:01:29.962Z","updated_at":"2026-07-25T03:01:30.599Z","avatar_url":"https://github.com/andyyaro.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Local AI Orchestrator\n\n\u003e A local-first, free-to-run multi-agent AI pipeline that improves outputs\n\u003e through structured critique, revision, scoring, code verification, and final synthesis.\n\u003e Runs on a MacBook Pro with no paid APIs. No internet connection required after setup.\n\n---\n\n## The Problem\n\nSingle-call AI responses are inconsistent. You send one prompt, get one answer,\nand have no way to know whether it is the best possible output or a mediocre first\ndraft. Large AI companies solve this with internal feedback loops and verification\nsystems — but those systems are usually invisible to users and often depend on\npaid API calls.\n\nThis project brings that quality loop to a local machine: one goal in, multiple\nagent passes, critique, revision, scored judgment, optional code verification,\nand a polished final output — all running locally through Ollama.\n\n---\n\n## Features\n\n- **7-agent quality pipeline**: Supervisor → Planner → Builder → Critic →\n  Fixer → Judge → Synthesizer\n- **Scored improvement loop**: Output improves across up to N iterations until\n  it passes a configurable quality threshold\n- **Structured JSON scoring**: The Judge returns category scores, pass/fail,\n  hard failures, and rationale\n- **Workflow modes**: Writing, Coding, Planning, Debugging, Study, and General\n- **Real code verification**: In coding mode, generated Python is executed and\n  pytest tests are run; failures are fed back into the next loop\n- **Hard-fail safety for broken code**: Code that fails execution or pytest cannot\n  pass just because the Judge model scored it highly\n- **Streamlit dashboard**: Local web UI with live agent output, score progression,\n  model controls, final output display, and run history\n- **SQLite run history**: Runs and iterations are saved locally for later review\n- **Structured logging**: JSON logs are written for debugging and run inspection\n- **Config-driven models**: Change model profiles without editing agent code\n- **Provider-ready adapter layer**: Ollama is the default; OpenAI and Anthropic\n  stubs exist for future expansion\n- **Fully local by default**: No API keys, no subscription, no paid model calls\n\n---\n\n## Architecture\n\n```text\nUser Goal\n    │\n    ▼\n┌─────────────┐\n│ Supervisor  │  Refines goal and chooses workflow mode\n└──────┬──────┘\n       │\n       ▼\n┌─────────────┐\n│  Planner    │  Creates a structured execution plan\n└──────┬──────┘\n       │\n       ▼\n┌─────────────┐\n│  Builder    │  Produces the first complete draft\n└──────┬──────┘\n       │\n       ▼\n┌─────────────────────────────────────────────┐\n│              Improvement Loop               │\n│                                             │\n│  ┌─────────┐   ┌────────┐   ┌──────────┐   │\n│  │ Critic  │ → │ Fixer  │ → │ Verify*  │   │\n│  └─────────┘   └────────┘   └────┬─────┘   │\n│                                  │         │\n│                              ┌───▼───┐     │\n│                              │ Judge │     │\n│                              └───┬───┘     │\n│                                  │         │\n│      repeat if score is below threshold     │\n└─────────────────────────────────────────────┘\n       │\n       ▼\n┌─────────────┐\n│ Synthesizer │  Polishes the best-scoring draft\n└─────────────┘\n```\n\n`Verify` runs only in coding mode. It extracts Python code, executes it safely,\nruns pytest when tests exist, and feeds failures back into the next revision loop.\n\n---\n\n## Agent Roles\n\n### Supervisor\n\nThe Supervisor receives the raw goal, rewrites it into a cleaner task, and chooses\nthe workflow mode.\n\n### Planner\n\nThe Planner converts the refined goal into a structured plan for the Builder.\n\n### Builder\n\nThe Builder creates the first complete draft or implementation.\n\n### Critic\n\nThe Critic reviews the draft and identifies weaknesses, missing pieces, errors,\nor unclear sections.\n\n### Fixer\n\nThe Fixer revises the draft based on the Critic’s feedback.\n\n### Code Verifier\n\nIn coding mode, the verifier extracts Python code blocks, blocks unsafe patterns,\nruns the code, and runs pytest tests when present.\n\n### Judge\n\nThe Judge scores the revised draft and returns structured JSON containing the score,\npass/fail decision, hard failures, and rationale.\n\n### Synthesizer\n\nThe Synthesizer takes the best-scoring draft and produces the final polished output.\n\n---\n\n## Repository Structure\n\n```text\nlocal-ai-orchestrator/\n├── agents/\n│   ├── supervisor.py\n│   ├── planner.py\n│   ├── builder.py\n│   ├── critic.py\n│   ├── fixer.py\n│   ├── judge.py\n│   └── synthesizer.py\n├── app/\n│   └── streamlit_app.py\n├── config/\n│   ├── models.yaml\n│   └── modes.yaml\n├── orchestrator/\n│   ├── adapters.py\n│   ├── code_runner.py\n│   ├── config_loader.py\n│   ├── database.py\n│   ├── logger.py\n│   └── pytest_runner.py\n├── prompts/\n│   ├── supervisor.txt\n│   ├── planner.txt\n│   ├── builder.txt\n│   ├── critic.txt\n│   ├── fixer.txt\n│   ├── judge.txt\n│   └── synthesizer.txt\n├── runs/\n│   └── .gitkeep\n├── logs/\n│   └── .gitkeep\n├── run.py\n├── run_langgraph.py\n├── show_history.py\n├── activate.sh\n├── requirements.txt\n└── README.md\n```\n\nGenerated run folders, SQLite history, logs, virtual environments, and environment\nfiles are ignored by Git.\n\n---\n\n## Requirements\n\nThis project was built and tested on:\n\n- macOS\n- Apple Silicon MacBook Pro\n- 24GB unified memory\n- Python 3.14\n- Ollama\n- Streamlit\n- SQLite\n- Local open-weight models\n\nRecommended Ollama models:\n\n```bash\nollama pull llama3.2:3b\nollama pull llama3.1:8b\nollama pull qwen2.5:14b\nollama pull qwen2.5-coder:14b\nollama pull gemma3:12b\nollama pull phi4:14b\n```\n\nThe system can run with smaller models for faster testing.\n\n---\n\n## Setup\n\nClone the repository:\n\n```bash\ngit clone https://github.com/andyyaro/local-ai-orchestrator.git\ncd local-ai-orchestrator\n```\n\nCreate and activate the virtual environment:\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\n```\n\nInstall dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\nMake sure Ollama is running:\n\n```bash\ncurl http://localhost:11434\n```\n\nExpected output:\n\n```text\nOllama is running\n```\n\nOptional helper activation script:\n\n```bash\nsource activate.sh\n```\n\n---\n\n## Terminal Usage\n\nRun the terminal pipeline:\n\n```bash\npython run.py \\\n  --goal \"Write a 300-word explanation of why sleep deprivation hurts productivity.\" \\\n  --max-loops 2 \\\n  --threshold 65\n```\n\nExample coding run:\n\n```bash\npython run.py \\\n  --goal \"Write a Python function called fibonacci(n: int) -\u003e list[int] that returns the first n Fibonacci numbers. Include pytest tests for n=0, n=1, and n=10.\" \\\n  --model-main qwen2.5-coder:14b \\\n  --model-fast llama3.2:3b \\\n  --max-loops 3 \\\n  --threshold 70\n```\n\nExample stress run with lighter models:\n\n```bash\npython run.py \\\n  --goal \"Write a comprehensive guide to Python decorators: what they are, how they work, three practical examples, and when not to use them.\" \\\n  --model-main llama3.1:8b \\\n  --model-fast llama3.2:3b \\\n  --max-loops 4 \\\n  --threshold 88 \\\n  --min-improvement 2\n```\n\nEach run creates a folder under `runs/\u003ctimestamp\u003e/` containing intermediate\nartifacts such as:\n\n```text\n00_supervisor.json\n01_planner_plan.txt\n02_builder_draft_v0.txt\nloop01_critic.txt\nloop01_fixer.txt\nloop01_judge.json\nbest_draft.txt\nfinal_output.txt\nrun_summary.json\n```\n\nIn coding mode, code verification artifacts are saved as:\n\n```text\nloop01_code_run.txt\n```\n\n---\n\n## Streamlit Dashboard\n\nLaunch the local dashboard:\n\n```bash\nstreamlit run app/streamlit_app.py\n```\n\nThen open:\n\n```text\nhttp://localhost:8501\n```\n\nThe dashboard lets you:\n\n- Enter a goal\n- Select workflow mode\n- Choose model overrides\n- Set loop count and score threshold\n- Watch agent outputs live\n- See score progression\n- Download the final output\n- Browse run history from SQLite\n\n---\n\n## Run History\n\nRuns are saved locally in SQLite:\n\n```text\nruns/history.db\n```\n\nInspect recent runs from the terminal:\n\n```bash\npython show_history.py\n```\n\nShow database stats:\n\n```bash\npython show_history.py --stats\n```\n\nInspect a specific run:\n\n```bash\npython show_history.py --run-id 1\n```\n\nReset run history:\n\n```bash\npython show_history.py --reset\n```\n\nThe database is intentionally ignored by Git because it contains personal run data.\n\n---\n\n## Configuration\n\nModel behavior is controlled by:\n\n```text\nconfig/models.yaml\n```\n\nSwitch between profiles by changing:\n\n```yaml\nactive_profile: serious\n```\n\nAvailable profiles include:\n\n```text\nbootstrap\nfast\nserious\ncoding\nlow_memory\n```\n\nEach profile uses at most one distinct 14B-class model to avoid the model\nswapping a 24GB Mac incurs when two 14B-class models compete for memory.\nSee [`docs/model-profiles.md`](docs/model-profiles.md) for the full memory\nbudget rationale and when to use `low_memory` over `serious` or `coding`.\n\nWorkflow behavior is controlled by:\n\n```text\nconfig/modes.yaml\n```\n\nPrompts are stored in:\n\n```text\nprompts/\n```\n\nThis makes it possible to improve agent behavior without rewriting the main pipeline.\n\n---\n\n## Code Verification\n\nCoding mode includes real execution checks.\n\nThe verifier:\n\n1. Extracts Python fenced code blocks.\n2. Blocks dangerous patterns.\n3. Runs the extracted Python code.\n4. Runs pytest if test functions are present.\n5. Marks the iteration as failed if execution or tests fail.\n6. Feeds the verification output into the next Critic/Fixer loop.\n\nA Judge pass cannot override broken code. If code verification fails, the verdict is\nforced to a `broken_code` hard failure.\n\n---\n\n## Logging\n\nStructured pipeline logs are written to:\n\n```text\nlogs/pipeline.log\n```\n\nEach log entry is JSON and includes events such as:\n\n```text\nrun_start\nagent_start\nagent_end\nscore\ncode_verification\nrun_stop\nerror\n```\n\nThe log file is ignored by Git.\n\n---\n\n## Testing Status\n\nThe full system was tested with six end-to-end scenarios:\n\n```text\n✅ Writing test\n✅ Planning test\n✅ Coding test with pytest verification\n✅ Debugging test\n✅ Study/explanation test\n✅ Stress test with multiple loops\n```\n\nThe stress test verified that the system can run multiple improvement loops,\ntrack score progression, save the highest-scoring draft, and complete while memory\npressure stays green.\n\n---\n\n## Release Gates\n\nFour distinct checks, from fastest/always-required to slowest/optional.\nEach one tests something different — running a slower check does not make\na faster one redundant, and passing a faster one does not substitute for\na slower one where it applies.\n\n**Unit/CI gate** — required on every PR, runs in GitHub Actions:\n\n```bash\npython -m ruff check .\npython -m pytest tests/ -v\n```\n\n**Quick release check** — required before tagging a release. Adds a\nlightweight end-to-end smoke test with a small local model:\n\n```bash\npython -m ruff check .\npython -m pytest tests/ -v\nbash scripts/local_acceptance.sh\n```\n\n`scripts/local_acceptance.sh` verifies Ollama is running and the pipeline\ncompletes end-to-end (or fails safely) with a small model\n(`llama3.2:3b`). It intentionally does **not** require a small model to\nsatisfy a strict output-quality constraint like an exact word count — a\nsmall model correctly detecting and reporting its own constraint\nviolation is a pass here, not a failure. It does still guard against the\nPhase 6b regression where the pipeline silently reported success despite\na violated constraint.\n\n**Full release-candidate check** — slow (many real model calls,\n10–20+ minutes observed); run after major pipeline/model behavior\nchanges, not before every tag:\n\n```bash\npython -m eval.run_eval_suite\n```\n\n**Strict local quality check** — optional/recommended for this immediate\nv2.0 tag, since it is slower and model-dependent:\n\n```bash\nbash scripts/strict_acceptance.sh\n```\n\n`scripts/strict_acceptance.sh` uses a stronger local model\n(`llama3.1:8b`) and requires the final output to genuinely satisfy a\nprecise constraint (an exact word count), failing loudly if it doesn't.\nIt never downloads models automatically — if `llama3.1:8b` isn't pulled\nlocally, it reports the exact `ollama pull` command and skips.\n\n---\n\n## Screenshots\n\n### Minimal Streamlit dashboard\n\n![Local AI Orchestrator dashboard](docs/screenshot_dashboard_1.jpeg)\n\n### Run controls and local model settings\n\n![Local AI Orchestrator controls](docs/screenshot_dashboard_2.jpeg)\n\n### Run history and output review\n\n![Local AI Orchestrator run history](docs/screenshot_dashboard_3.jpeg)\n\n## Limitations\n\n- Local model quality depends on the selected Ollama model.\n- 14B models can be slow on long prompts.\n- Large models may timeout on memory-heavy runs.\n- Code verification executes generated code locally, so safety rules matter.\n- The Judge score is model-dependent and may not be perfectly calibrated.\n- This is a single-user local system, not a hosted multi-user product.\n\n---\n\n## Safety Notes\n\n- The project runs locally by default.\n- No data is sent to paid APIs by default.\n- `.env`, `.venv/`, `runs/`, and `logs/` are ignored by Git.\n- Never commit API keys, tokens, passwords, or local run databases.\n- Review generated code before trusting it outside the sandboxed verification flow.\n\n---\n\n## Planned Improvements\n\n- [ ] Add dashboard screenshots to the README\n- [ ] Add a visual diff between Fixer iterations\n- [ ] Add export buttons for Markdown and PDF\n- [ ] Add run comparison view\n- [ ] Add RAG over previous runs or local documents\n- [ ] Add optional OpenAI or Anthropic provider support through config\n- [ ] Add more detailed model-performance benchmarking\n\n---\n\n## License\n\nMIT License.\n\n---\n\nBuilt with Python, Ollama, Streamlit, SQLite, and local open-weight models.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyyaro%2Flocal-ai-orchestrator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandyyaro%2Flocal-ai-orchestrator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyyaro%2Flocal-ai-orchestrator/lists"}