{"id":13514191,"url":"https://github.com/cockroachdb/copyist","last_synced_at":"2025-06-17T09:10:40.569Z","repository":{"id":39904582,"uuid":"276266698","full_name":"cockroachdb/copyist","owner":"cockroachdb","description":"Mocking your SQL database in Go tests has never been easier.","archived":false,"fork":false,"pushed_at":"2025-02-19T00:38:16.000Z","size":1721,"stargazers_count":849,"open_issues_count":6,"forks_count":20,"subscribers_count":52,"default_branch":"master","last_synced_at":"2025-06-13T14:02:31.149Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cockroachdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-01T03:24:25.000Z","updated_at":"2025-06-10T15:25:35.000Z","dependencies_parsed_at":"2024-04-10T17:57:08.348Z","dependency_job_id":"06de19e6-1206-4e1c-81df-f07f9c782e0b","html_url":"https://github.com/cockroachdb/copyist","commit_stats":{"total_commits":56,"total_committers":9,"mean_commits":6.222222222222222,"dds":0.3214285714285714,"last_synced_commit":"de1f16c43c224f173a75e80f791b3bcfd18a0554"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/cockroachdb/copyist","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fcopyist","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fcopyist/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fcopyist/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fcopyist/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cockroachdb","download_url":"https://codeload.github.com/cockroachdb/copyist/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fcopyist/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260326793,"owners_count":22992388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T05:00:49.259Z","updated_at":"2025-06-17T09:10:40.540Z","avatar_url":"https://github.com/cockroachdb.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# copyist\n\n[![Go Reference](https://pkg.go.dev/badge/github.com/cockroachdb/copyist.svg)](https://pkg.go.dev/github.com/cockroachdb/copyist) [![Latest Release](https://img.shields.io/github/release/cockroachdb/copyist.svg?style=flat-square)](https://github.com/cockroachdb/copyist/releases/latest) [![License](https://img.shields.io/badge/License-Apache_2.0-yellowgreen.svg)](https://opensource.org/licenses/Apache-2.0)\n\nMocking your SQL database in Go tests has never been easier. The copyist library\nautomatically records low-level SQL calls made during your tests. It then\ngenerates recording files that can be used to play back those calls without\nconnecting to the real SQL database. Run your tests again. This time, they'll\nrun much faster, because now they do not require a database connection.\n\nBest of all, your tests will run as if your test database was reset to a clean,\nwell-known state between every test case. Gone are the frustrating problems\nwhere a test runs fine in isolation, but fails when run in concert with other\ntests that modify the database. In fact, during playback you can run different\ntest packages in parallel, since they will not conflict with one another at the\ndatabase level.\n\ncopyist imposes no overhead on production code, and it requires almost no\nchanges to your application or testing code, as long as that code directly or\nindirectly uses Go's `sql` package (e.g. Go ORM's and the widely used `sqlx`\npackage). This is because copyist runs at the driver level of Go's `sql`\npackage.\n\n## What problems does copyist solve?\n\nImagine you have some application code that opens a connection to a Postgres\ndatabase and queries some customer data:\n\n```go\nfunc QueryName(db *sql.DB) string {\n\trows, _ := db.Query(\"SELECT name FROM customers WHERE id=$1\", 100)\n\tdefer rows.Close()\n\n\tfor rows.Next() {\n\t\tvar name string\n\t\trows.Scan(\u0026name)\n\t\treturn name\n\t}\n\treturn \"\"\n}\n```\n\nThe customary way to test this code would be to create a test database and\npopulate it with test customer data. However, what if application code modifies\nrows in the database, like removing customers? If the above code runs on a\nmodified database, it may not return the expected customer. Therefore, it's\nimportant to reset the state of the database between test cases so that tests\nbehave predictably. But connecting to a database is slow. Running queries is\nslow. And resetting the state of an entire database between every test is\n_really_ slow.\n\nVarious mocking libraries are another alternative to using a test database.\nThese libraries intercept calls at some layer of the application or data access\nstack, and return canned responses without needing to touch the database. The\nproblem with many of these libraries is that they require the developer to\nmanually construct the canned responses, which is time-consuming and fragile\nwhen application changes occur.\n\n## How does copyist solve these problems?\n\ncopyist includes a Go `sql` package driver that records the low-level SQL calls\nmade by application and test code. When a Go test using copyist is invoked with\nthe \"-record\" command-line flag, then the copyist driver will record all SQL\ncalls. When the test completes, copyist will generate a custom text file that\ncontains the recorded SQL calls. The Go test can then be run again without the\n\"-record\" flag. This time the copyist driver will play back the recorded calls,\nwithout needing to access the database. The Go test is none the wiser, and runs\nas if it was using the database.\n\n## How do I use copyist?\n\nBelow is the recommended test pattern for using copyist. The example shows how\nto unit test the `QueryName` function shown above.\n\n```go\nfunc init() {\n\tcopyist.Register(\"postgres\")\n}\n\nfunc TestQueryName(t *testing.T) {\n\tdefer copyist.Open(t).Close()\n\n\tdb, _ := sql.Open(\"copyist_postgres\", \"postgresql://root@localhost\")\n\tdefer db.Close()\n\n\tname := QueryName(db)\n\tif name != \"Andy\" {\n\t\tt.Error(\"failed test\")\n\t}\n}\n```\n\nIn your `init` or `TestMain` function (or any other place that gets called\nbefore any of the tests), call the `copyist.Register` function. This function\nregisters a new driver with Go's `sql` package with the name\n`copyist_\u003cdriverName\u003e`. In any tests you'd like to record, add a\n`defer copyist.Open(t).Close()` statement. This statement begins a new recording\nsession, and then generates a playback file when `Close` is called at the end of\nthe test.\n\ncopyist does need to know whether to run in \"recording\" mode or \"playback\" mode.\nTo make copyist run in \"recording\" mode, invoke the test with the `record` flag:\n\n```\ngo test -run TestQueryName -record\n```\n\nThis will generate a new recording file in a `testdata` subdirectory, with the\nsame name as the test file, but with a `.copyist` extension. For example, if the\ntest file is called `app_test.go`, then copyist will generate a\n`testdata/app_test.copyist` file containing the recording for the\n`TestQueryName` test. Now try running the test again without the `record` flag:\n\n```\ngo test -run TestQueryName\n```\n\nIt should now run significantly faster. You can also define the COPYIST_RECORD\nenvironment variable (to any value) to make copyist run in recording mode:\n\n```\nCOPYIST_RECORD=1 go test ./...\n```\n\nThis is useful when running many test packages, some of which may not link to\nthe copyist library, and therefore do not define the `record` flag.\n\n## How do I reset the database between tests?\n\nYou can call `SetSessionInit` to register a function that will clean your\ndatabase:\n\n```go\nfunc init() {\n    copyist.Register(\"postgres\")\n    copyist.SetSessionInit(resetDB)\n}\n```\n\nThe resetDB function will be called by copyist each time you call `copyist.Open`\nin your tests, as long as copyist is running in \"recording\" mode. The session\ninitialization function can do anything it likes, but usually it will run a SQL\nscript against the database in order to reset it to a clean state, by\ndropping/creating tables, deleting data from tables, and/or inserting \"fixture\"\ndata into tables that makes testing more convenient.\n\n## Troubleshooting\n\n#### I'm seeing \"unexpected call\" panics telling me to \"regenerate recording\"\n\nThis just means that you need to re-run your tests with the \"-record\" command\nline flag, in order to generate new recordings. Most likely, you changed either\nyour application or your test code so that they call the database differently,\nusing a different sequence or content of calls.\n\nHowever, there are rarer cases where you've regenerated recordings, have made no\ntest or application changes, and yet are still seeing this error when you run\nyour tests in different orders. This is caused by non-determinism in either your\napplication or in the ORM you're using.\n\nAs an example of non-determinism, some ORMs send a setup query to the database\nwhen the first connection is opened in order to determine the database version.\nSo whichever test happens to run first records an extra Query call. If you run\na different test first, you'll see the \"unexpected call\" error, since other\ntests aren't expecting the extra call.\n\nThe solution to these problems is to eliminate the non-determinism. For example,\nin the case of an ORM sending a setup query, you might initialize it from your\n`TestMain` method:\n\n```go\nfunc TestMain(m *testing.M) {\n\tflag.Parse()\n\tcopyist.Register(\"postgres\")\n\tcopyist.SetSessionInit(resetDB)\n\tcloser := copyist.OpenNamed(\"test.copyist\", \"OpenCopyist\")\n\tpop.Connect(\"copyist-test\")\n\tcloser.Close()\n\tos.Exit(m.Run())\n}\n```\n\nThis triggers the first query in TestMain, which is always run before tests.\n\n#### The generated copyist recording files are too big\n\nThe size of the recording files is directly related to the number of accesses\nyour tests make to the database, as well as the amount of data that they\nrequest. While copyist takes pains to generate efficient recording files that\neliminate as much redundancy as possible, there's only so much it can do. Try\nto write tests that operate over smaller amounts of interesting data. For tests\nthat require large numbers of database calls, or large amounts of data, use a\ndifferent form of verification. One nice thing about copyist is that you can\npick and choose which tests will use it. The right tool for the right job, and\nall that.\n\n## Limitations\n\n- Because of the way copyist works, it cannot be used with test and application\n  code that accesses the database concurrently on multiple threads. This\n  includes tests running with the \"-parallel\" testing flag, which enables tests\n  in the same package to run in parallel. Multiple threads are problematic\n  because the copyist driver code has no way to know which threads are\n  associated with which tests. However, this limitation does not apply to\n  running different test packages in parallel; in playback mode, this is both\n  possible and highly encouraged! However, in recording mode, there may be\n  problems if your tests conflict with one another at the database layer (i.e.\n  by reading/modifying the same rows). The recommended pattern is to run test\n  packages serially in recording mode, and then in parallel in playback mode.\n\n- copyist currently supports only the Postgres `pq` and `pgx stdlib` drivers. If\n  you'd like to extend copyist to support other drivers, like MySql or SQLite,\n  you're invited to submit a pull request.\n\n- copyist does not implement every `sql` package driver interface and method.\n  This may mean that copyist may not fully work with some drivers with more\n  advanced features. Contributions in this area are welcome.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcockroachdb%2Fcopyist","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcockroachdb%2Fcopyist","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcockroachdb%2Fcopyist/lists"}