{"id":24480579,"url":"https://github.com/mrsimonemms/gobblr","last_synced_at":"2025-06-13T17:06:46.370Z","repository":{"id":64175150,"uuid":"570862737","full_name":"mrsimonemms/gobblr","owner":"mrsimonemms","description":"Make your development databases gobble up known data","archived":false,"fork":false,"pushed_at":"2025-05-22T05:11:16.000Z","size":141,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-08T01:44:40.448Z","etag":null,"topics":["data-import","data-ingestion","database","developer-tools","development","gitpod-compatible","ingestion","mongo","mongodb","mysql","postgresql","sql","sqlite","testing"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrsimonemms.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["mrsimonemms"]}},"created_at":"2022-11-26T11:26:57.000Z","updated_at":"2025-05-22T05:11:20.000Z","dependencies_parsed_at":"2025-03-14T11:26:14.422Z","dependency_job_id":"e3720a18-3891-4fdf-b1df-d98498f4ca9a","html_url":"https://github.com/mrsimonemms/gobblr","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":"mrsimonemms/golang-cobra-starter","purl":"pkg:github/mrsimonemms/gobblr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsimonemms%2Fgobblr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsimonemms%2Fgobblr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsimonemms%2Fgobblr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsimonemms%2Fgobblr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrsimonemms","download_url":"https://codeload.github.com/mrsimonemms/gobblr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsimonemms%2Fgobblr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259686085,"owners_count":22896049,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-import","data-ingestion","database","developer-tools","development","gitpod-compatible","ingestion","mongo","mongodb","mysql","postgresql","sql","sqlite","testing"],"created_at":"2025-01-21T11:16:29.215Z","updated_at":"2025-06-13T17:06:46.296Z","avatar_url":"https://github.com/mrsimonemms.png","language":"Go","funding_links":["https://github.com/sponsors/mrsimonemms"],"categories":[],"sub_categories":[],"readme":"# Gobblr\n\n[![made-with-Go](https://img.shields.io/badge/Made%20with-Go-1f425f.svg)](https://go.dev/)\n[![Maintainer](https://img.shields.io/badge/maintainer-mrsimonemms-blue)](https://github.com/MrSimonEmms)\n[![Go Report Card](https://goreportcard.com/badge/github.com/mrsimonemms/gobblr)](https://goreportcard.com/report/github.com/mrsimonemms/gobblr)\n[![GitHub go.mod Go version of a Go module](https://img.shields.io/github/go-mod/go-version/mrsimonemms/gobblr.svg)](https://github.com/mrsimonemms/gobblr)\n[![GoDoc reference example](https://img.shields.io/badge/godoc-reference-blue.svg)](https://godoc.org/github.com/mrsimonemms/gobblr)\n\nMake your development databases gobble up known data\n\n\u003c!-- toc --\u003e\n\n* [Compatability](#compatability)\n* [Data](#data)\n  * [Tips](#tips)\n  * [Supported formats](#supported-formats)\n    * [JSON](#json)\n    * [JavaScript](#javascript)\n  * [Meta data](#meta-data)\n* [Running Gobblr](#running-gobblr)\n  * [CLI](#cli)\n  * [Docker Compose](#docker-compose)\n  * [Web server](#web-server)\n  * [Docker](#docker)\n* [Resetting data](#resetting-data)\n  * [Resetting with the CLI](#resetting-with-the-cli)\n  * [Resetting with the web server](#resetting-with-the-web-server)\n  * [Use in integration/end-to-end tests](#use-in-integrationend-to-end-tests)\n* [Why?](#why)\n  * [Why might you need test data?](#why-might-you-need-test-data)\n  * [Why shouldn't I use my ORM's migration in development?](#why-shouldnt-i-use-my-orms-migration-in-development)\n* [Contributing](#contributing)\n  * [Open in a container](#open-in-a-container)\n  * [Commit style](#commit-style)\n\n\u003c!-- Regenerate with \"pre-commit run -a markdown-toc\" --\u003e\n\n\u003c!-- tocstop --\u003e\n\n## Compatability\n\nThis supports the following databases:\n\n* [MongoDB](https://www.mongodb.com)\n* [MySQL](https://www.mysql.com)\n* [PostgreSQL](https://www.postgresql.org/)\n* [SQLite](https://www.sqlite.org)\n* [SQL Server](https://www.microsoft.com/sql-server)\n\nTo request additional drivers, please raise a PR.\n\n## Data\n\nYour test data will live in your repo, usually in a folder called `/data`.\n\n### Tips\n\n* if you need to link data between datasets, use a hard-coded identifying key\n* any data that matches an [RFC3339](https://pkg.go.dev/time) or YYYY-MM-DD\n  eg, `2022-10-23`) format is sent as a date object\n\n### Supported formats\n\n#### JSON\n\nThis would be used to add static data.\n\n```json\n[\n  {\n    \"registered\": true,\n    \"confirmed\": true,\n    \"username\": \"test1\",\n    \"email_address\": \"test@test.com\",\n    \"name\": \"Test Testington\",\n    \"password\": \"13905934cb5503269ca5613b72a8aa8a:57a1846d12716308be489ec25f2a017e22aba88d591f5c5184e046cb42ec921f1ac5f5249b212efe5c8a644d45d7c63d3ee075cce79ba3a663c731971a56f4c3\"\n  }\n]\n```\n\n#### JavaScript\n\nThis would be used to add dynamic data and must be returned from a `data` function.\n\n```js\nfunction data() {\n  return [\n    {\n      item: 2,\n      some_date: new Date(),\n    },\n    {\n      item: 3,\n      some_date: new Date(),\n    },\n  ]\n}\n\n```\n\n### Meta data\n\nBy default, each data item will add a `createdAt` and `updatedAt` entry with the\ncurrent datetime.\n\nIf you need to amend this, either to remove either or both keys or to rename them,\nyou can set the data file to include a `meta` object and `data` array. The `data`\narray will remain the same as per the above examples and the `meta` key controls\nhow the data is added.\n\n```json\n{\n  \"meta\": {\n    \"created\": true,\n    \"createdKey\": \"createdAt\",\n    \"updated\": true,\n    \"updatedKey\": \"updatedAt\"\n  },\n  \"data\": []\n}\n```\n\n| Key | Type | Default Value | |\n| --- | --- | --- | --- |\n| `created` | `boolean` | `true` | Controls whether the `createdKey` appears in the dataset |\n| `createdKey` | `strng` | `createdAt` | The name of the key |\n| `updated` | `boolean` | `true` | Controls whether the `updatedKey` appears in the dataset |\n| `updatedKey` | `strng` | `updatedAt` | The name of the key |\n\n## Running Gobblr\n\nFor ease, all these example use MongoDB. The principles are the same for all database\ntypes - please run `gobblr db --help` to see the documentation for the individual\ndatabases.\n\nYou will need one instance of Gobblr running for each database you have in your\ndevelopment stack.\n\nFor the connection URI, see the [MongoDB docs](https://www.mongodb.com/docs/manual/reference/connection-string).\n\n### CLI\n\n```bash\ngobblr db mongodb -u \u003cconnection-uri\u003e -d \u003cdatabase\u003e --path /path/to/data\n```\n\n### Docker Compose\n\n```yaml\nservices:\n  gobblr-mongodb:\n    image: ghcr.io/mrsimonemms/gobblr\n    ports:\n      - 5670:5670\n    environment:\n      GOBBLR_CONNECTION_URI: mongodb://mongodb:27017/gobblr\n      GOBBLR_DATABASE: gobblr\n      GOBBLR_PATH: /data\n    links:\n      - mongodb\n    volumes:\n      - ./data:/data # Path to your data files\n    restart: on-failure\n    command: db mongodb --run\n\n  mongodb:\n    image: library/mongo\n    ports:\n      - 27017:27017\n```\n\n### Web server\n\n```bash\ngobblr db mongodb -u \u003cconnection-uri\u003e -d \u003cdatabase\u003e --path /path/to/data --run\n```\n\n### Docker\n\n```bash\ndocker run -it --rm ghcr.io/mrsimonemms/gobblr \\\n  db mongodb -u \u003cconnection-uri\u003e -d \u003cdatabase\u003e --path /path/to/data --run\n```\n\n## Resetting data\n\nThe key advantage of Gobblr is that the data can be reset to the known state very\nquickly. This can be done in two ways:\n\n### Resetting with the CLI\n\nEvery time a `gobblr db` command is run, the data is cleared down before the data\nis ingested again. Just run the command again\n\n### Resetting with the web server\n\nGobblr can be run as a web server to make resetting the data possible via an HTTP\ncall. The call is `POST:/data/reset`. This call accepts no arguments and returns\nthe tables affected and gives a count of the data inserted. For example:\n\n```json\n[\n  {\n    \"table\": \"users\",\n    \"count\": 1\n  },\n  {\n    \"table\": \"items\",\n    \"count\": 2\n  }\n]\n```\n\n### Use in integration/end-to-end tests\n\nThis can be used in integration/end-to-end tests to check that your application\ncorrectly interacts with the database(s). Most test frameworks have a `beforeEach`\ntype method that runs an arbitrary function prior to each test being executed -\nyou will need to make a call to Gobblr in one of these test hooks to reset the\ndata to ensure the data is consistent.\n\nHere is an example using [Jest](https://jestjs.io). In the your tests, create a\nfile similar to this:\n\n```js\nimport axios from 'axios';\n\n// Reset the dataset before every test is run\nbeforeEach(() =\u003e\n  axios({\n    // GOBBLR_URL envvar will typically be http://localhost:5670 (local running) or http://gobblr:5670 (Docker Compose)\n    url: `${process.env.GOBBLR_URL}/data/reset`,\n    method: 'post',\n    headers: {\n      'content-type': 'application/json',\n    },\n    data: {},\n  }),\n);\n```\n\n## Why?\n\nI've worked on so many projects where there's a need to have a known data set for\ndevelopment and testing purposes. Most database ORMs provide a way of migrating\nthe database structure and seed data, but they can be a bit difficult to target\ndata specifically for testing purposes.\n\nI've used this basic approach for years where the data is driven from JSON files\nor JS scripts, but this is a way of making this cross-platform.\n\n**NB**. This should not be used for populating production databases. If you need\nto seed your production databases with data, use an ORM for the language you're\nusing.\n\n### Why might you need test data?\n\nHaving a known data set allows developers to get started fast. If you want to see\nhow a `GET` endpoint works, you need data in there. If you want to update or delete\na record, you need pre-existing data.\n\nIf you're writing integration/end-to-end tests (and you probably should be for most\nweb apps) then you need a known data set so your tests pass. And of course, when\nwriting these types of tests, you want to ensure that your data set is reset before\neach run - this ensures truly isolated tests and avoids the problem of having a\nsuite of 10,000 tests failing, but passing when run individually.\n\n### Why shouldn't I use my ORM's migration in development?\n\nIf your current setup works for you, keep using it.\n\nTypically, ORM migrations require your dependency tree installing before they can\nrun. This will take time that you can avoid with a zero-dependency binary.\n\nIn addition to that, you typically don't want to seed your dev/test data into your\nproduction seed data and not all ORM migration tools allow you to segregate data\nby deployment type.\n\nLastly, ORM migrations are there to migrate data, not set it to a known state.\nWhen running an integration test against known data in a database, you should set\nyour databases to a known state before each test it run. With a migration tool,\nthere are usually no guarantees that any changed data has been put back to the known\nstate - with Gobblr, the tables are wiped clean before the data is ingested making\ncertain that your data is in a known state.\n\n## Contributing\n\n### Open in a container\n\n* [Open in a container](https://code.visualstudio.com/docs/devcontainers/containers)\n\n### Commit style\n\nAll commits must be done in the [Conventional Commit](https://www.conventionalcommits.org)\nformat.\n\n```git\n\u003ctype\u003e[optional scope]: \u003cdescription\u003e\n\n[optional body]\n\n[optional footer(s)]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrsimonemms%2Fgobblr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrsimonemms%2Fgobblr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrsimonemms%2Fgobblr/lists"}