{"id":50969322,"url":"https://github.com/databricks-solutions/databricks-vision","last_synced_at":"2026-06-19T00:30:34.832Z","repository":{"id":361172269,"uuid":"1242179079","full_name":"databricks-solutions/databricks-vision","owner":"databricks-solutions","description":"Full-stack Databricks App for single + batch image generation, editing, automated evaluation, and semantic search.","archived":false,"fork":false,"pushed_at":"2026-05-29T12:10:35.000Z","size":17935,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-29T14:07:02.704Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS.txt","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE.md","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-18T07:37:05.000Z","updated_at":"2026-05-29T12:23:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/databricks-solutions/databricks-vision","commit_stats":null,"previous_names":["databricks-solutions/databricks-vision"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/databricks-solutions/databricks-vision","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fdatabricks-vision","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fdatabricks-vision/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fdatabricks-vision/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fdatabricks-vision/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-solutions","download_url":"https://codeload.github.com/databricks-solutions/databricks-vision/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-solutions%2Fdatabricks-vision/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34513020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-18T02:00:06.871Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-19T00:30:33.819Z","updated_at":"2026-06-19T00:30:34.818Z","avatar_url":"https://github.com/databricks-solutions.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Databricks Vision\n\nProduction-ready image generation, editing, analysis, and semantic search on Databricks. One Python library (`image_gen.py`) drives a deployable FastAPI + React Databricks App that supports single-image generate/edit, batch jobs, gallery, semantic search, and bulk import.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"samples/collage.png\" alt=\"Sample outputs\" width=\"900\" /\u003e\n\u003c/p\u003e\n\n## What it does\n\n- **Generate** images with `gpt-image-2` (auto-routes to `gpt-image-1.5` for transparent backgrounds). Any WxH multiple of 16 up to 3840px, total pixels 655K–8.3M, edge ratio ≤ 3:1.\n- **Edit** existing images with prompt + reference. Edits land as new gallery rows.\n- **Analyze** every generated image with `databricks-gpt-5-5`: description, tags, evaluation, 5×0-5 metric sub-scores, missing-elements, safety flags, brand conflicts, an improved-prompt suggestion, and an optional `criteria_evaluation` against user-supplied style guidelines.\n- **Search** the corpus by text or by uploaded image. SigLIP-2 1152-dim embeddings, pgvector HNSW + cosine similarity, single-query join of metadata + similarity. FTS fallback available.\n- **Batch generate** via a Databricks Job: multi-image (one prompt template applied to N inputs) or variations (one source × N variation prompts).\n- **Import** local images in bulk; the app synthesizes a prompt then runs the same analyzer + embedder pipeline so imported images become searchable.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"docs/assets/demo.gif\" alt=\"Databricks Vision demo\" width=\"900\" /\u003e\n\u003c/p\u003e\n\n## Stack\n\n- **Backend** — Python, FastAPI, Pydantic\n- **Frontend** — React 19, TypeScript, TanStack Router / Query, shadcn/ui, Tailwind\n- **Storage** — Unity Catalog Volumes (image bytes) + Lakebase Autoscaling Postgres with pgvector (metadata, evals, embeddings)\n- **Inference** — Databricks Model Serving wrapping the Responses API with `gpt-image-2` / `gpt-image-1.5`; analyzer backed by `databricks-gpt-5-5`; image + text embeddings from a custom SigLIP-2 SO400M/14-384 endpoint\n- **Build / deploy** — Databricks Asset Bundles, [apx](https://github.com/databricks/apx) toolkit, `uv`, `bun`\n\n## Architecture\n\n```\n┌────────────────────────────────────┐     ┌────────────────────────────────────┐\n│  Databricks App                    │     │  Lakebase Autoscaling Postgres     │\n│  (FastAPI + React)                 │◄───►│  (metadata + pgvector embeddings)  │\n└─────────────┬──────────────────────┘     └────────────────────────────────────┘\n              │\n              ├──► Foundation Model serving\n              │     ├── gpt-image-2 / gpt-image-1.5   (generate, edit)\n              │     └── databricks-gpt-5-5            (analyzer, prompt rewrite)\n              │\n              ├──► Custom SigLIP-2 SO400M/14-384 endpoint\n              │     (text + image  →  1152-dim embeddings)\n              │\n              └──► Unity Catalog Volumes\n                    (PNG bytes, organised by batch / folder)\n                                ▲\n   Batch Generation Job ────────┘\n   (Databricks Job; ai_query()\n    against image-generator endpoint)\n```\n\n**Single-image flow:** UI streams partial-image events over SSE while the model generates, then the backend persists bytes to a UC Volume, writes the row to Lakebase, and runs the analyzer + embedder as background tasks that `UPDATE` the row when they finish.\n\n**Batch flow:** the app kicks off a Databricks Job that reads inputs from a UC Volume, runs `ai_query()` against the image-generator serving endpoint, writes outputs back to a Volume, and syncs metadata + embeddings to Lakebase. The gallery shows single-gen and batch images from the same table.\n\n## What's interesting\n\n- **Image evaluation** — every generated image is scored on five 0-5 dimensions (quality, prompt adherence, purpose fit, text legibility, safe content) plus categorical safety / brand flags, with a critique paragraph and a suggested improved prompt for off-spec results. See [`ImageAnalyzer` in image_gen.py](image_gen.py).\n- **Semantic search over the corpus** — every image is embedded with SigLIP-2 (1152-dim) at ingest time and stored in pgvector with an HNSW index on cosine similarity. A single SQL query joins metadata filters with similarity ranking, so the gallery can search by text or by an uploaded image without a separate vector store. See [`ImageSearch` in image_gen.py](image_gen.py).\n- **Two-phase persistence** — generate / edit / import endpoints insert the gallery row immediately with placeholder eval fields, then run the analyzer + embedder as background tasks that `UPDATE` the row when ready. The UI polls for a short window after a generate so eval fields appear without manual refresh.\n- **Lakebase Autoscaling with OAuth-rotating connections** — the psycopg pool re-fetches a Lakebase credential token on every new connection, with `max_lifetime=2700` so connections recycle before the 1-hour token expiry. See [`VisionWorkspace` in image_gen.py](image_gen.py).\n- **`ai_query()` for batch inference** — [`notebooks/02_BATCH_GENERATE.py`](notebooks/02_BATCH_GENERATE.py) calls the image-generator serving endpoint via Spark SQL `ai_query()`, getting per-row parallelism for free.\n\n## Quickstart\n\nFull setup — including Lakebase, UC, and the two bootstrap notebooks — is in **[DEPLOY.md](DEPLOY.md)**. Summary:\n\n1. Provision Lakebase Autoscaling Postgres + a UC catalog/schema/volumes in the target workspace.\n2. Run [`notebooks/00_SIGLIP_DEPLOY.py`](notebooks/00_SIGLIP_DEPLOY.py) once (~30 min, GPU endpoint).\n3. Run [`notebooks/01_MODEL_DEPLOY.py`](notebooks/01_MODEL_DEPLOY.py) once (~10 min, image-generator pyfunc endpoint).\n4. Fill in the `dev` target block in [`databricks.yml`](databricks.yml) with your workspace coordinates.\n5. `./scripts/deploy.sh dev \u003cyour-profile\u003e`.\n\nSubsequent redeploys take 2–3 minutes.\n\n## Repo layout\n\n```\nimage_gen.py             # the library — drives the app and any notebook usage\napp/                     # FastAPI + React Databricks App\nnotebooks/               # 00_SIGLIP_DEPLOY, 01_MODEL_DEPLOY, 02_BATCH_GENERATE\nscripts/                 # deploy.sh, post-deploy.sh, render-app-yml.sh\ndatabricks.yml           # DAB config; variables-driven, per-target overrides\nDEPLOY.md                # full deploy guide\nsamples/                 # example outputs\n```\n\n## Library usage\n\nThe same `image_gen.py` the app uses can be driven from a notebook or script:\n\n```python\nfrom image_gen import VisionWorkspace, ImageGen\n\nws = VisionWorkspace(\n    catalog=\"\u003cyour-catalog\u003e\",\n    schema=\"\u003cyour-schema\u003e\",\n    lakebase_endpoint=\"projects/\u003cproject\u003e/branches/\u003cbranch\u003e/endpoints/\u003cendpoint\u003e\",\n)\ngen = ImageGen(ws)\n\nimg = gen.generate(\n    prompt=\"A high-contrast studio photograph of a brushed-aluminium product\",\n    size=\"1024x1024\",\n    quality=\"high\",\n)\nimg.show()\n```\n\nSee [DEPLOY.md](DEPLOY.md#8-auth-model) for the auth model (service-principal-only today).\n\n## How to get help\n\nDatabricks support doesn't cover this content. For questions or bugs, please [open a GitHub issue](https://github.com/databricks-solutions/databricks-vision/issues) and the maintainers will help on a best-effort basis. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to contribute.\n\n## License\n\n\u0026copy; 2026 Databricks, Inc. All rights reserved. The source in this repository is provided subject to the Databricks License [https://databricks.com/db-license-source]. All included or referenced third-party libraries are subject to the licenses set forth below.\n\n| Library | Description | License | Source |\n|---|---|---|---|\n| FastAPI | ASGI web framework | MIT | https://github.com/fastapi/fastapi |\n| Uvicorn | ASGI server | BSD-3-Clause | https://github.com/encode/uvicorn |\n| Pydantic | Data validation | MIT | https://github.com/pydantic/pydantic |\n| pydantic-settings | Settings management | MIT | https://github.com/pydantic/pydantic-settings |\n| sse-starlette | SSE support for Starlette | BSD-3-Clause | https://github.com/sysid/sse-starlette |\n| python-multipart | Multipart parsing | Apache-2.0 | https://github.com/Kludex/python-multipart |\n| python-dotenv | .env loader | BSD-3-Clause | https://github.com/theskumar/python-dotenv |\n| httpx | HTTP client | BSD-3-Clause | https://github.com/encode/httpx |\n| OpenAI Python SDK | OpenAI client | Apache-2.0 | https://github.com/openai/openai-python |\n| Databricks SDK for Python | Databricks client | Apache-2.0 | https://github.com/databricks/databricks-sdk-py |\n| psycopg | PostgreSQL adapter | LGPL-3.0 | https://github.com/psycopg/psycopg |\n| pgvector-python | pgvector client | MIT | https://github.com/pgvector/pgvector-python |\n| Pillow | Imaging library | MIT-CMU | https://github.com/python-pillow/Pillow |\n| matplotlib | Plotting library | PSF-2.0 | https://github.com/matplotlib/matplotlib |\n| React | UI library | MIT | https://github.com/facebook/react |\n| Vite | Frontend tooling | MIT | https://github.com/vitejs/vite |\n| TanStack Router | Type-safe routing | MIT | https://github.com/TanStack/router |\n| TanStack Query | Data fetching | MIT | https://github.com/TanStack/query |\n| Tailwind CSS | Utility CSS framework | MIT | https://github.com/tailwindlabs/tailwindcss |\n| shadcn/ui | Component primitives | MIT | https://github.com/shadcn-ui/ui |\n| apx | Databricks Apps toolkit | Databricks License | https://github.com/databricks/apx |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-solutions%2Fdatabricks-vision","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-solutions%2Fdatabricks-vision","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-solutions%2Fdatabricks-vision/lists"}