{"id":50969334,"url":"https://github.com/databricks-solutions/maxgenie","last_synced_at":"2026-06-19T00:30:38.727Z","repository":{"id":362563695,"uuid":"1251434690","full_name":"databricks-solutions/maxgenie","owner":"databricks-solutions","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-04T20:53:48.000Z","size":12699,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-04T22:11:30.241Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-solutions.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.txt","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-27T15:16:19.000Z","updated_at":"2026-06-04T20:53:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/databricks-solutions/maxgenie","commit_stats":null,"previous_names":["databricks-solutions/maxgenie"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/databricks-solutions/maxgenie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fmaxgenie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fmaxgenie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fmaxgenie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fmaxgenie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-solutions","download_url":"https://codeload.github.com/databricks-solutions/maxgenie/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fmaxgenie/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34513020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-19T00:30:37.660Z","updated_at":"2026-06-19T00:30:38.702Z","avatar_url":"https://github.com/databricks-solutions.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MaxGenie\n\n\u003cp align=\"center\"\u003e\u003cstrong\u003eRelease v0.1.0\u003c/strong\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/databricks-solutions/maxgenie/actions/workflows/tests.yml\"\u003e\u003cimg src=\"https://github.com/databricks-solutions/maxgenie/actions/workflows/tests.yml/badge.svg\" alt=\"Tests\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/maxgenie-demo.gif\" alt=\"MaxGenie demo\" width=\"720\"\u003e\n\u003c/p\u003e\n\nMaxGenie is a Databricks workspace skill for optimizing one Genie space at a time. It runs inside the Databricks workspace, creates a disposable optimization clone, reads the space benchmarks, proposes structured improvements through a Databricks serving endpoint, rejects regressions with benchmark gates, and writes a final report with checkpoints and rollback artifacts.\n\nThe source Genie space is never modified directly.\n\n## Architecture\n\n```mermaid\nflowchart LR\n  src[\"Source Genie space\u003cbr/\u003e(read-only)\"] --\u003e clone[\"Export + managed clone\u003cbr/\u003e(edits only here)\"]\n  clone --\u003e assets[\"Editable assets\"]\n\n  clone --\u003e bench{\"Benchmarks?\"}\n  bench -- \"No\" --\u003e adv[\"Advisory best-practices pass\"] --\u003e advReport[\"Not benchmark-verified\u003cbr/\u003eno score\"]\n\n  bench -- \"Yes\" --\u003e split[\"Train / validation / hidden holdout\"]\n  split --\u003e base[\"Baseline eval\"] --\u003e cand[\"Candidate change\u003cbr/\u003evisible evidence only\"]\n  assets --\u003e cand\n\n  cand --\u003e gate[\"Train + validation gate\"]\n  gate --\u003e pass{\"Passes?\"}\n  pass -- \"No\" --\u003e reject[\"Reject / restore\"]\n  pass -- \"Yes\" --\u003e accept[\"Accept / checkpoint\"]\n\n  reject --\u003e loop{\"Budget left?\"}\n  accept --\u003e loop\n  loop -- \"Yes\" --\u003e cand\n  loop -- \"No\" --\u003e audit[\"Final full eval\u003cbr/\u003ehidden holdout included\"]\n\n  split -. \"holdout not shown to candidates\" .-\u003e audit\n\n  audit --\u003e final{\"Improves full score\u003cbr/\u003eand holdout safe?\"}\n  final -- \"Yes\" --\u003e promote[\"Promotable clone state\"]\n  final -- \"No\" --\u003e safe[\"Safest checkpoint\u003cbr/\u003enot promotable\"]\n\n  promote --\u003e report[\"Final report\"]\n  safe --\u003e report\n```\n\nThe local `maxgenie` command is an operator wrapper for syncing the skill, launching Databricks jobs, checking status, and running the same workflow from a shell when needed. It is not the primary product surface.\n\n## Promotion Evidence\n\nThe one promotable MaxGenie path is the clone-safe workspace skill or Databricks job flow with result-based comparison, a fresh canonical baseline, full benchmark coverage, hidden-holdout isolation, checkpointed rollback, and terminal selection that requires full-score improvement while blocking holdout regression.\n\nRuns that use SQL-string comparison, partial benchmark coverage, observed-only baselines, or noncanonical splits are diagnostics. They can be useful for smoke tests and hypothesis generation, but final reports mark them non-comparable and they must be rerun under the strict result-based protocol before they count as leaderboard or release evidence.\n\n## What It Does\n\n- Parses a Genie space URL and validates Databricks authentication.\n- Exports space configuration and benchmark questions.\n- Creates or reuses a MaxGenie-managed optimization clone.\n- Builds train, validation, and hidden holdout splits.\n- Reuses the latest completed Genie benchmark run when requested, so the initial score does not need to be rerun on the source space.\n- Applies safe metadata curation and benchmark-gated candidate changes.\n- Uses a Databricks serving endpoint for strict JSON candidate proposals.\n- Replays compatible prior accepted candidate sequences when local run memory is available.\n- Repairs stale replay payloads so required pass-guard examples are preserved.\n- Runs train and validation gates before accepting a candidate.\n- Restores explicit recovery checkpoints if a clone needs to be recreated mid-run.\n- Runs terminal selection so a final state is promoted only when full score improves and holdout does not regress.\n- Runs one advisory best-practices pass when no benchmark exists, labels it not benchmark-verified, and never reports a fabricated score.\n- Preserves checkpoints, iteration logs, candidate payloads, benchmark summaries, and owner-facing reports.\n\n## Documentation\n\n- [Operator reference](docs/operator-reference.md): local job submission, status checks, audit/replay jobs, and advanced flags.\n- [Candidate modes](docs/candidate-modes.md): default strict artifact-patch behavior and controlled diagnostic modes.\n- [Public release checklist](PUBLIC_RELEASE.md): publication scope, security checks, and review requirements.\n- [Changelog](CHANGELOG.md): release notes.\n\n## Public Release Guardrails\n\nThis repository is intended to contain only the reusable MaxGenie runtime, workspace skill files, templates, documentation, and tests. Do not publish generated workspace exports, benchmark payloads, query-history exports, local reports, transcripts, credentials, or customer data.\n\nExamples and tests use synthetic fixture data. Any domain-specific product names, metric names, table names, and SQL snippets in tests are illustrative fixtures used to exercise optimizer behavior, not customer-provided data.\n\nBefore making a repository public, complete the checklist in `PUBLIC_RELEASE.md`, including license selection, third-party dependency license review, peer review, owner assignment, and confirmation that no non-public information is tracked.\n\n## License\n\nCopyright 2026 Databricks, Inc. All rights reserved. The source is provided subject to the Databricks License in `LICENSE.md`. Included or referenced third-party libraries are subject to their own licenses.\n\n| library | description | license | source |\n|---|---|---|---|\n| `databricks-sdk` | Databricks SDK for Python | Apache-2.0 | \u003chttps://github.com/databricks/databricks-sdk-py\u003e |\n| `httpx` | HTTP client | BSD-3-Clause | \u003chttps://github.com/encode/httpx\u003e |\n| `pydantic` | Data validation | MIT | \u003chttps://github.com/pydantic/pydantic\u003e |\n| `PyYAML` | YAML parser/emitter | MIT | \u003chttps://pyyaml.org/\u003e |\n| `typer` | CLI framework | MIT | \u003chttps://github.com/fastapi/typer\u003e |\n| `python-dotenv` | Local env-file loading | BSD-3-Clause | \u003chttps://github.com/theskumar/python-dotenv\u003e |\n| `pytest` | Test runner | MIT | \u003chttps://github.com/pytest-dev/pytest\u003e |\n| `mlflow` | Optional tracking integration | Apache-2.0 | \u003chttps://github.com/mlflow/mlflow\u003e |\n\n## Primary Use: Databricks Workspace Skill\n\nThe workspace skill lives under `genie_skills/maxgenie`.\n\nSync it to a user-level skill path:\n\n```bash\npython genie_skills/maxgenie/scripts/sync_to_workspace.py \\\n  --profile \u003cprofile\u003e \\\n  --host \"https://\u003cworkspace\u003e\" \\\n  --destination \"/Workspace/Users/\u003cuser\u003e@\u003cdomain\u003e/.assistant/skills/maxgenie\"\n```\n\nOr sync it to a workspace-level skill path:\n\n```bash\npython genie_skills/maxgenie/scripts/sync_to_workspace.py \\\n  --profile \u003cprofile\u003e \\\n  --host \"https://\u003cworkspace\u003e\" \\\n  --destination \"/Workspace/.assistant/skills/maxgenie\"\n```\n\nAfter sync, open a Genie space and invoke the `MaxGenie` skill. In workspace authoring runtimes, use the in-process Python entrypoint because child shell processes may not inherit the runtime authentication context. Load the synced workspace runner explicitly; some Genie Code runtimes do not put the skill root on `sys.path`:\n\n```python\nimport importlib.util\nfrom pathlib import Path\n\nfrom databricks.sdk import WorkspaceClient\n\n\ndef load_maxgenie_run_skill():\n    user_name = WorkspaceClient().current_user.me().user_name\n    candidate_paths = [\n        Path(f\"/Workspace/Users/{user_name}/.assistant/skills/maxgenie/scripts/runner.py\"),\n        Path(\"/Workspace/.assistant/skills/maxgenie/scripts/runner.py\"),\n    ]\n    for runner_path in candidate_paths:\n        if not runner_path.exists():\n            continue\n        spec = importlib.util.spec_from_file_location(\"maxgenie_skill_runner\", runner_path)\n        if spec is None or spec.loader is None:\n            continue\n        module = importlib.util.module_from_spec(spec)\n        spec.loader.exec_module(module)\n        return module.run_skill\n    raise FileNotFoundError(\"Could not find the synced MaxGenie runner.\")\n\n\nrun_skill = load_maxgenie_run_skill()\n\nrun_skill(\n    mode=\"full\",\n    space_url=\"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\",\n    strategy=\"serving\",\n    serving_endpoint=\"\u003ccandidate-serving-endpoint\u003e\",\n    workspace_root=\"/Workspace/Users/\u003cuser\u003e@\u003cdomain\u003e/.maxgenie/runs/full\",\n    use_latest_observed_benchmark=True,\n    plateau_rounds=4,\n    warehouse_id=\"\u003csql-warehouse-id\u003e\",\n)\n```\n\nIf the run should create clones in a shared folder, pass `clone_parent_path=\"/Shared/\u003cteam-folder\u003e\"`. Otherwise MaxGenie creates clones under the Databricks run-as user's home folder.\n\n## Multi-Hour Databricks Job Runs\n\nFor unattended optimization, submit the synced runner as a Databricks job. This is the recommended path for long runs because it is durable and does not depend on a foreground chat session:\n\n```python\nrun_skill(\n    mode=\"submit\",\n    space_url=\"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\",\n    strategy=\"serving\",\n    serving_endpoint=\"\u003ccandidate-serving-endpoint\u003e\",\n    workspace_root=\"/Workspace/Users/\u003cuser\u003e@\u003cdomain\u003e/.maxgenie/runs/job\",\n    use_latest_observed_benchmark=True,\n    plateau_rounds=4,\n    timeout_seconds=14400,\n    warehouse_id=\"\u003csql-warehouse-id\u003e\",\n)\n```\n\n`mode=\"submit\"` is non-blocking by default from the programmatic workspace entrypoint. It prints the run id and canonical Jobs run page URL immediately; call `run_skill(mode=\"status\", run_id=\"\u003crun_id\u003e\", status_format=\"chat\")` for later chat snapshots or final results. In a local shell, pass `wait=True` or use `maxgenie submit-run --wait` when you want the foreground process to poll and print streamed status updates.\n\nThe default job compute mode is serverless. If serverless jobs are unavailable, pass `compute_mode=\"existing_cluster\"` plus `existing_cluster_id=\"\u003ccluster-id\u003e\"`, or pass `compute_mode=\"new_cluster\"` plus `new_cluster_json=\"\u003cjson-or-file\u003e\"`.\n\nFirst-time users may need permissions for:\n\n- reading the source Genie space and its benchmarks\n- creating Genie spaces under the selected clone parent folder\n- running the selected Databricks job compute mode\n- calling the configured serving endpoint\n- using a SQL warehouse for result-based comparison\n\nQuery-history enrichment may additionally need access to `system.query`. That signal is optional; MaxGenie reports the blocker instead of failing the run.\n\n## Lite Smoke Test\n\nUse lite mode only to verify that export, clone creation, a tiny benchmark subset, candidate evaluation, report writing, and optional clone cleanup work end to end:\n\n```python\nrun_skill(\n    mode=\"lite\",\n    space_url=\"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\",\n    benchmark_limit=1,\n    max_candidates=1,\n    trash_clone=True,\n)\n```\n\nLite mode is not the optimizer; full or submit mode is the production path.\n\n## Local Operator Setup\n\nLocal installation is useful for syncing the skill, submitting jobs, running status checks, and running the same workflow from a shell:\n\n```bash\npython3 -m venv .venv\nsource .venv/bin/activate\npip install -e '.[dev]'\n```\n\nMaxGenie resolves Databricks credentials in this order:\n\n1. `--profile \u003cprofile\u003e`\n2. `DATABRICKS_CONFIG_PROFILE`\n3. `DATABRICKS_HOST` plus `DATABRICKS_TOKEN`\n\nWhen a space URL host is available and `--profile` is omitted, MaxGenie checks `~/.databrickscfg` and prefers a profile whose `host` matches the space URL.\n\nValidate access:\n\n```bash\nmaxgenie doctor --space-url \"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\"\n```\n\nFor repeated runs, set:\n\n```bash\nexport MAXGENIE_DEFAULT_SPACE_URL=\"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\"\nexport MAXGENIE_SERVING_ENDPOINT=\"\u003ccandidate-serving-endpoint\u003e\"\n```\n\n## Local Full Run\n\n```bash\nmaxgenie optimize \\\n  --space-url \"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\" \\\n  --profile \u003cprofile\u003e \\\n  --strategy serving \\\n  --serving-endpoint \u003ccandidate-serving-endpoint\u003e \\\n  --use-latest-observed-benchmark \\\n  --plateau-rounds 4\n```\n\n`--strategy serving` asks the configured Databricks serving endpoint to return a strict JSON operation plan. MaxGenie applies only supported structured operations, pushes them to the clone, runs benchmark gates, rejects regressions, and checkpoints accepted states.\n\nBy default, serving optimization uses strict `artifact_patch_v2` candidates: compact, hash-checked edits grounded only in visible train/validation evidence. MaxGenie rejects malformed, over-broad, hidden-holdout-leaking, or guard-regressing patches before they can be accepted. Controlled candidate modes are available for diagnostics and advanced validation; see [Candidate modes](docs/candidate-modes.md).\n\nUse `--warehouse-id \u003cwarehouse_id\u003e` when result-based comparison should execute generated SQL and compare result sets. Without a usable warehouse, MaxGenie falls back to SQL-string comparison and records that in the report as diagnostic, non-comparable evidence. When generated SQL is missing or a result-based comparison attempt fails for an individual question, MaxGenie scores that question as a failed result-based comparison instead of falling back to SQL-string similarity or dropping it from coverage.\n\nUseful flags:\n\n- `--bootstrap-only`: export, clone, split, and score the baseline without candidate generation.\n- `--use-latest-observed-benchmark`: seed the baseline from the latest completed Genie benchmark eval run.\n- `--fresh-lineage`: start a new MaxGenie-managed clone lineage from the input space.\n- `--clone-parent-path /Users/\u003cuser\u003e` or `/Shared/\u003cteam-folder\u003e`: choose where optimization clones are created.\n- `--plateau-rounds` and `--timeout-seconds`: control the unattended loop for normal serving runs. Omit `--max-iterations` and `--candidate-budget` unless a diagnostic lane intentionally needs a hard cap; plateau and the job timeout should normally determine when the lane stops.\n- `--serving-reasoning-effort none|low|medium|high|xhigh`: raise proposer reasoning effort (default `low`). Example: `--serving-reasoning-effort xhigh`; endpoints that do not support `xhigh` cap at `high`.\n- `--audit-checkpoint-path \u003cpath\u003e`: restore a checkpoint into the clone and run a final audit without candidate generation.\n- `--audit-patch-path \u003cpath\u003e`: replay an artifact patch sequence into the clone and run a final audit without candidate generation.\n\n## Local Job Submission\n\nThe same job run can be submitted from a local shell:\n\n```bash\nmaxgenie submit-run \\\n  --space-url \"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\" \\\n  --profile \u003cprofile\u003e \\\n  --strategy serving \\\n  --serving-endpoint \u003ccandidate-serving-endpoint\u003e \\\n  --workspace-root \"/Workspace/Users/\u003cuser\u003e@\u003cdomain\u003e/.maxgenie/runs/job\" \\\n  --use-latest-observed-benchmark \\\n  --max-iterations 50 \\\n  --plateau-rounds 4 \\\n  --timeout-seconds 14400\n```\n\nJob submission requires the production-comparable protocol by default. Runs must provide a SQL warehouse for\nresult-based comparison and use a fresh baseline unless they are explicitly exploratory. To submit a diagnostic\nrun that should not be treated as promotion evidence, pass `--allow-diagnostic`.\n\nThe default job compute mode is serverless. If serverless jobs are unavailable, use `--compute-mode existing_cluster --existing-cluster-id \u003ccluster-id\u003e` or `--compute-mode new_cluster --new-cluster-json '\u003cjson-or-file\u003e'`.\n\nFirst-time users may need permissions for:\n\n- reading the source Genie space and its benchmarks\n- creating Genie spaces under the selected clone parent folder\n- running the selected Databricks job compute mode\n- calling the configured serving endpoint\n- using a SQL warehouse for result-based comparison\n\nQuery-history enrichment may additionally need access to `system.query`. That signal is optional; MaxGenie reports the blocker instead of failing the run.\n\nUseful job commands:\n\n```bash\nmaxgenie submit-run ... --dry-run\nmaxgenie run-status \u003crun_id\u003e --space-url \"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\"\nmaxgenie cancel-run \u003crun_id\u003e --space-url \"https://\u003cworkspace\u003e/genie/rooms/\u003cspace_id\u003e\"\n```\n\nAdvanced audit, replay, compute, and diagnostic-mode submission examples are covered in the [Operator reference](docs/operator-reference.md).\n\n## Artifact Layout\n\nBy default, local runs write under `workspace/\u003cspace_id\u003e/` and workspace jobs write under the `--workspace-root` path.\n\n```text\nworkspace/\u003cspace_id\u003e/\n  space/\n    meta.yml\n    tables/\n    instructions/\n    sample_questions.yml\n  benchmarks/\n    train_set.json\n    validation_set.json\n    validation_folds.json\n    test_set.json\n    split_meta.json\n  results/\n    baseline_train.json\n    latest_train.json\n    latest_validation_summary.json\n    latest_test_score.txt\n    latest_full_summary.json\n    observed_benchmark_seed.json\n    optimization_summary.json\n    terminal_selection.json\n    generalization_audit.json\n    trace_memory.json\n    failure_context_pack.json\n    cross_run_memory.json\n    query_history_summary.json\n    query_history_recommendations.json\n  history/\n    iteration_log.jsonl\n    train_runs/\n    validation_runs/\n    test_runs/\n    full_runs/\n    serving_candidates/\n    recovery_checkpoints/\n  checkpoints/\n  clone_meta.json\n  workspace_meta.json\n  OPTIMIZATION_TASK.md\n  final_report.md\n```\n\nThe final holdout is used for audit and reporting, not for candidate proposal generation.\n\n## Manual Commands\n\nAfter bootstrap, run these from any directory by passing `--workspace-root` and `--space-id`, or from `workspace/\u003cspace_id\u003e/` with defaults:\n\n```bash\nmaxgenie push\nmaxgenie benchmark --train\nmaxgenie benchmark --validation\nmaxgenie benchmark --test\nmaxgenie benchmark --full\nmaxgenie query-history --warehouse-id \u003cwarehouse_id\u003e\nmaxgenie status\nmaxgenie report\n```\n\n`maxgenie curate-loop` runs a conservative metadata curation sweep. Each curation mode is pushed and benchmarked independently, and a mode is accepted only when the benchmark gate does not regress.\n\n## Operating Rules\n\n- Never modify the source Genie space directly.\n- Create optimization clones under the run-as user's home folder by default.\n- Preserve run artifacts, checkpoints, and iteration logs.\n- Accept a candidate only when benchmark gates do not regress.\n- Keep hidden holdout content out of candidate proposal context.\n- Report authentication, permission, package, endpoint, and warehouse blockers explicitly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-solutions%2Fmaxgenie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-solutions%2Fmaxgenie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-solutions%2Fmaxgenie/lists"}