{"id":29628114,"url":"https://github.com/metabase/dataset-generator","last_synced_at":"2025-07-21T08:35:06.901Z","repository":{"id":300101148,"uuid":"1001088314","full_name":"metabase/dataset-generator","owner":"metabase","description":"AI Dataset Generator – Create realistic datasets for demos, learning, and dashboards","archived":false,"fork":false,"pushed_at":"2025-07-03T14:03:50.000Z","size":423,"stargazers_count":467,"open_issues_count":1,"forks_count":17,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-03T15:23:59.809Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.metabase.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metabase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-12T19:58:12.000Z","updated_at":"2025-07-03T14:03:46.000Z","dependencies_parsed_at":"2025-07-03T15:38:17.276Z","dependency_job_id":null,"html_url":"https://github.com/metabase/dataset-generator","commit_stats":null,"previous_names":["matthewhefferon/dataset-generator","metabase/dataset-generator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/metabase/dataset-generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdataset-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdataset-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdataset-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdataset-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metabase","download_url":"https://codeload.github.com/metabase/dataset-generator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metabase%2Fdataset-generator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266267530,"owners_count":23902388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-21T08:35:06.179Z","updated_at":"2025-07-21T08:35:06.875Z","avatar_url":"https://github.com/metabase.png","language":"TypeScript","funding_links":[],"categories":["Projects using `@faker-js/faker`"],"sub_categories":[],"readme":"# AI Dataset Generator\n\n**Generate realistic datasets for demos, learning, and dashboards. Instantly preview data, export as CSV or SQL, and explore with Metabase.**\n\nFeatures:\n\n- Conversational prompt builder: choose business type, schema, row count, and more\n- Real-time data preview in the browser\n- Export as CSV (single file or multi-table ZIP) or as SQL inserts\n- One-click Metabase launch for data exploration ([see Using Metabase](#using-metabase) for details)\n\n## Usage Flow\n\n1. Select your business type, schema, and other parameters.\n2. Click \"Preview Data\" to generate a 10-row sample (incurs a small LLM cost, depending on provider).\n3. Download CSV/SQL for as many rows as you want—no extra cost, always uses the same schema/columns as the preview.\n\n## Prerequisites\n\n- [Docker](https://www.docker.com/get-started) (includes Docker Compose)\n- At least one API key for a supported LLM provider (OpenAI, Anthropic, or Google GenAI)\n\n## Getting Started\n\n1. **Clone the repo:**\n\n   ```bash\n   git clone \u003cyour-repo-url\u003e\n   cd dataset-generator\n   ```\n\n2. **Create your .env file:**\n\n   Copy the example file and fill in your LLM provider API keys (OpenAI, Anthropic, Google, etc.):\n\n   ```bash\n   cp .env.example .env\n   ```\n\n   Then edit `.env` and add your API keys as needed:\n\n   ```env\n   # For local development, you can use any value for these keys:\n   LITELLM_MASTER_KEY=sk-1234\n   LITELLM_SALT_KEY=sk-1234\n\n   # Add at least one provider key below:\n   OPENAI_API_KEY=sk-...\n   ANTHROPIC_API_KEY=...\n   GOOGLE_GENAI_API_KEY=...\n\n   # Set LLM_MODEL to match your provider:\n   LLM_MODEL=gpt-4o\n\n   # Examples values:\n   # For OpenAI:      LLM_MODEL=gpt-4o\n   # For Anthropic:   LLM_MODEL=claude-4-sonnet\n   # For Google:      LLM_MODEL=gemini-2.5-flash\n   ```\n\n3. **Start the Next.js app:**\n\n   ```bash\n   npm install\n   npm run dev\n   ```\n\n   - The app runs at [http://localhost:3000](http://localhost:3000)\n\n4. **Start LiteLLM (Required for LLM Features):**\n\n   This app uses [LiteLLM](https://github.com/BerriAI/litellm) as a gateway for all LLM requests (OpenAI, Anthropic, Google, etc.).\n\n   **You must start LiteLLM for dataset generation and preview features to work.**\n\n   From your project root, run:\n\n   ```sh\n   docker compose up litellm db_litellm\n   ```\n\n   - This starts the LiteLLM gateway and its dedicated Postgres database.\n   - LiteLLM will listen on `http://localhost:4000` by default.\n\n5. **Generate a dataset:**\n\n   - Use the prompt builder to define your dataset.\n   - Click \"Preview Data\" to see a sample.\n\n6. **Export or Explore:**\n   - Download your dataset as CSV or SQL Inserts.\n   - Click \"Start Metabase\" to spin up Metabase in Docker.\n   - Once Metabase is ready, click \"Open Metabase\" to explore your data.\n   - When done, click \"Stop Metabase\" to shut down and clean up Docker containers.\n\n## How It Works\n\n- When you preview a dataset, the app uses LiteLLM (which can route to OpenAI, Anthropic, Google, etc.) to generate a detailed data spec (schema, business rules, event logic) for your chosen business type and parameters.\n- All actual data rows are generated locally using Faker, based on the LLM-generated spec.\n- Downloading or exporting data never calls an LLM again—it's instant and free.\n\n### Cost \u0026 Data Generation Summary\n\n| Action       | Calls LLM? | Cost?  | Uses LLM? | Uses Faker? | Row Count |\n| ------------ | :--------: | :----: | :-------: | :---------: | :-------: |\n| Preview      |    Yes     | ~$0.05 |    Yes    |     Yes     |    10     |\n| Download CSV |     No     |   $0   |    No     |     Yes     |   100+    |\n| Download SQL |     No     |   $0   |    No     |     Yes     |   100+    |\n\n_The above costs and behavior are based on testing with the OpenAI GPT-4o model. Costs and token usage may vary with other providers/models._\n\n- **You only pay for the preview/spec generation** (e.g., ~$0.05 per preview with OpenAI GPT-4o)\n- **All downloads use the same columns/spec, just with more rows, and are free**\n\n## Using Metabase\n\nWhen you click \"Start Metabase\", it will launch Metabase in a Docker container. Once ready:\n\n1. Click \"Open Metabase\" to access the Metabase interface\n2. Follow Metabase's setup process\n3. To analyze your generated data:\n   - Use the CSV export feature to download your dataset\n   - In Metabase, use the [\"Upload Data\" feature](https://www.metabase.com/docs/latest/exploration-and-organization/uploads) to analyze your CSV files\n   - Or [connect to your own database](https://www.metabase.com/docs/latest/databases/connecting) where you've loaded the data\n\n## Project Structure\n\n- `/app/page.tsx` – Main UI and prompt builder\n- `/app/api/generate/route.ts` – Synthetic data generator (via LiteLLM: OpenAI, Anthropic, Google, etc.)\n- `/app/api/metabase/start|stop|status/route.ts` – Docker orchestration for Metabase\n- `/lib/export/` – CSV/SQL export logic\n- `/docker-compose.yml` – Used for Metabase and LiteLLM services\n\n## Stack\n\n- **Next.js** (App Router, TypeScript)\n- **Tailwind CSS + ShadCN UI** (modern, dark-themed UI)\n- **LiteLLM** (multi-provider LLM gateway: OpenAI, Anthropic, Google, etc.)\n- **Metabase** (Dockerized, launched on demand)\n\n## Extending/Contributing\n\n- To add new business types, edit `lib/spec-prompts.ts` and add entries to the `businessTypeInstructions` object\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetabase%2Fdataset-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetabase%2Fdataset-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetabase%2Fdataset-generator/lists"}