{"id":19914722,"url":"https://github.com/fullstorydev/emulators","last_synced_at":"2025-05-03T05:31:44.473Z","repository":{"id":46818355,"uuid":"375720050","full_name":"fullstorydev/emulators","owner":"fullstorydev","description":"High quality cloud service emulators for local development stacks","archived":false,"fork":false,"pushed_at":"2024-04-09T20:19:17.000Z","size":468,"stargazers_count":112,"open_issues_count":4,"forks_count":18,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-04-10T00:32:31.726Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fullstorydev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-06-10T14:09:03.000Z","updated_at":"2024-04-15T15:56:20.512Z","dependencies_parsed_at":"2023-01-23T20:01:33.577Z","dependency_job_id":"6c5e63e7-fa10-4371-84a9-7370129aff71","html_url":"https://github.com/fullstorydev/emulators","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Femulators","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Femulators/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Femulators/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fullstorydev%2Femulators/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fullstorydev","download_url":"https://codeload.github.com/fullstorydev/emulators/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224354150,"owners_count":17297401,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T21:36:52.965Z","updated_at":"2024-11-12T21:36:53.493Z","avatar_url":"https://github.com/fullstorydev.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# emulators\nHigh quality Google Cloud service emulators for local development stacks\n\n## Why?\n\nAt Fullstory, our entire product and backend software stack runs in each engineer's local workstation. This high-quality local development experience keeps our engineers happy and productive, because they are able to:\n\n- build and test features locally\n- reproduce and fix bugs quickly and easily\n- run high-quality services in unit and integration tests\n\nOur local development and testing story is simpler when our live code can rely on expected services to exist, and we don't have to write as many mocks.\n\nSome of the ways we achieve this:\n\n- Our own backend services operate in a reasonable manner in a local environment.\n- Open source, third party services (such as Redis, Zookeeper, or Solr) run locally.\n- We emulate Google Cloud infrastructure.\n\n## What Google Cloud services do we emulate?\n\n| Service                    | Persistence? | Status                  | Notes                                                                                                                         |\n|----------------------------|--------------|-------------------------|-------------------------------------------------------------------------------------------------------------------------------|\n| Google Bigtable            | Yes          | Shipped, see below      | Fork of [bigtable/bttest](https://github.com/googleapis/google-cloud-go/tree/master/bigtable/bttest)                          |\n| Google Cloud Storage (GCS) | Yes          | Shipped, see below      | Written from scratch                                                                                                          |\n| Google Pubsub              | No           | Considering persistence | Vanilla [pubsub/pstest](https://github.com/googleapis/google-cloud-go/tree/master/pubsub/pstest)                              |\n| Google Cloud Functions     | n/a          | In consideration        | Thin wrapper that manages `node` processes.                                                                                   |\n| Google Datastore           | Yes          | -                       | Google's [Datastore emulator](https://cloud.google.com/datastore/docs/tools/datastore-emulator) (written in Java) works great |\n\n## Google Bigtable Emulator\n\nOur bigtable emulator is a fork of [bigtable/bttest](https://github.com/googleapis/google-cloud-go/tree/master/bigtable/bttest).  A summary of the changes we made:\n- The core operates directly on Bigtable protobuf types, such as Table and Row, instead of bespoke types.\n- The storage layer is pluggable and operates on protos.\n- Leveldb is the default storage implementation, and runs either in-memory (transient for unit tests) or on disk (long running, persistence).\n\n### Installing\n\n```sh\ngo install github.com/fullstorydev/emulators/bigtable/...@latest\n```\n\n### Running, out of process\n\nExample, running on a specific port, with persistence:\n```sh\n\u003e cbtemulator -port 8888 -dir var/bigtable\nWriting to: var/bigtable\nCloud Bigtable emulator running on 127.0.0.1:8888\n```\n\nUsage:\n```\n  -dir string\n    \tif set, use persistence in the given directory\n  -host string\n    \tthe address to bind to on the local machine (default \"localhost\")\n  -port int\n    \tthe port number to bind to on the local machine (default 9000)\n```\n\n### Running, in process\n\nYou can link bigtable emulator into existing Go binaries as a drop-in replacement for `bigtable/bttest`.\n\nFor unit tests:\n```go\n\t// start an in-memory leveldb BigTable test server (for unit tests)\n\tsrv, err := bttest.NewServer(\"127.0.0.1:0\", grpc.MaxRecvMsgSize(math.MaxInt32))\n\tif err != nil { \n\t\t// ...\n\t}\n\tdefer srv.Close()\n\t// bigtable.NewClient (via DefaultClientOptions) will look at this env var to figure out what host to talk to\n\tos.Setenv(\"BIGTABLE_EMULATOR_HOST\", svr.Addr)\n```\n\nFor on-disk persistence:\n```go\n\t// start an leveldb-backed BigTable test server\n\tsrv, err := bttest.NewServerWithOptions(fmt.Sprintf(\"127.0.0.1:%d\", *btport), bttest.Options{\n\t\tStorage: bttest.LeveldbDiskStorage{\n\t\t\tRoot: bigtableStorageDir,\n\t\t\tErrLog: func(err error, msg string) {\n\t\t\t\t// wire into logging\n\t\t\t},\n\t\t},\n\t\tGrpcOpts: []grpc.ServerOption{grpc.MaxRecvMsgSize(maxGrpcMessageSize)},\n\t})\n```\n\n### Connecting to the Bigtable emulator from Go\n\n```go\n\t// assuming BIGTABLE_EMULATOR_HOST is already set...\n\tconn, err := grpc.Dial(os.Getenv(\"BIGTABLE_EMULATOR_HOST\"), grpc.WithInsecure())\n\tif err != nil {\n\t\t// ...\n\t}\n\tdefer conn.Close() // only if the life cycle is scoped to this call\n\n\tclient, err := bigtable.NewClient(ctx, project, instance, option.WithGRPCConn(conn))\n\tif err != nil {\n\t\t// ...\n\t}\n\ttbl := client.Open(\"example\")\n```\n\n## Google Cloud Storage Emulator\n\nOur storage emulator was written in house.\n- Supports basic file operations, iteration, attributes, copying, and some conditionals.\n- The storage layer is pluggable.\n- In memory btree (transient for unit tests) or disk-based storage (long running, persistence).\n\n### Installing\n\n```sh\ngo install github.com/fullstorydev/emulators/storage/...@latest\n```\n\n### Running, out of process\n\nExample, running on a specific port, with persistence:\n```sh\n\u003e gcsemulator -port 8888 -dir var/storage\nWriting to: var/storage\nCloud Storage emulator running on http://127.0.0.1:8888\n```\n\nUsage:\n```\n  -dir string\n    \tif set, use persistence in the given directory\n  -host string\n    \tthe address to bind to on the local machine (default \"localhost\")\n  -port int\n    \tthe port number to bind to on the local machine (default 9000)\n```\n\nFor unit tests:\n```go\n\t// start an in-memory Storage test server (for unit tests)\n\tsvr, err := gcsemu.NewServer(\"127.0.0.1:0\", gcsemu.Options{})\n\tif err != nil {\n\t\t// ...\n\t}\n\tdefer svr.Close()\n\t// gcsemu.NewClient will look at this env var to figure out what host/port to talk to\n\tos.Setenv(\"GCS_EMULATOR_HOST\", svr.Addr)\n```\n\nFor on-disk persistence:\n```go\n\t// start an on-disk Storage test server\n\tsvr, err := gcsemu.NewServer(fmt.Sprintf(\"127.0.0.1:%d\", *port), gcsemu.Options{\n\t\tStore: gcsemu.NewFileStore(*gcsDir),\n\t})\n```\n\n### Connecting to the GCS emulator from Go\n\n```go\n\t// assuming GCS_EMULATOR_HOST is already set...\n\tclient, err := gcsemu.NewClient(ctx)\n\tif err != nil {\n\t\t// ...\n\t}\n\tdefer client.Close() // only if the life cycle is scoped to this call\n```\n\n#### NOTE ####\n\nDo NOT use `STORAGE_EMULATOR_HOST`, as defined in `cloud.google.com/go/storage`.  There are unresolved issues in the Go\nclient implementation.  `STORAGE_EMULATOR_HOST` is supported inconsistently, and even has some bugs that can cause\ndata races when using the same `*storage.Client` for different types of access.\n\nSee:\n- [storage: when using an emulator, it is not possible to use the same Client object for both uploading and other operations #2476](https://github.com/googleapis/google-cloud-go/issues/2476)\n- [Storage: empty readHost when STORAGE_EMULATOR_HOST is set to host:port #4444](https://github.com/googleapis/google-cloud-go/issues/4444)\n\nInstead, use our `gcsemu.NewClient(ctx)` method which swaps out the entire HTTP transport.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffullstorydev%2Femulators","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffullstorydev%2Femulators","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffullstorydev%2Femulators/lists"}