{"id":37063824,"url":"https://github.com/aylien/emulator-tools","last_synced_at":"2026-01-14T07:18:32.061Z","repository":{"id":57426339,"uuid":"241342171","full_name":"AYLIEN/emulator-tools","owner":"AYLIEN","description":"Google Cloud BigTable and PubSub emulator tools to make development a breeze","archived":false,"fork":false,"pushed_at":"2020-08-12T09:11:56.000Z","size":500,"stargazers_count":17,"open_issues_count":0,"forks_count":4,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-09-22T14:42:41.084Z","etag":null,"topics":["bigtable","emulator","google-bigtable","google-cloud","google-cloud-platform","google-pubsub","pubsub"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AYLIEN.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-18T11:18:15.000Z","updated_at":"2023-10-15T18:49:22.000Z","dependencies_parsed_at":"2022-09-11T05:01:58.798Z","dependency_job_id":null,"html_url":"https://github.com/AYLIEN/emulator-tools","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AYLIEN/emulator-tools","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AYLIEN%2Femulator-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AYLIEN%2Femulator-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AYLIEN%2Femulator-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AYLIEN%2Femulator-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AYLIEN","download_url":"https://codeload.github.com/AYLIEN/emulator-tools/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AYLIEN%2Femulator-tools/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28217665,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2026-01-05T02:00:06.358Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigtable","emulator","google-bigtable","google-cloud","google-cloud-platform","google-pubsub","pubsub"],"created_at":"2026-01-14T07:18:31.386Z","updated_at":"2026-01-14T07:18:32.056Z","avatar_url":"https://github.com/AYLIEN.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"emulator-tools\n===============\n\nGoogle Cloud provides [emulators](https://cloud.google.com/sdk/gcloud/reference/beta/emulators/) for testing cloud components such as PubSub and BigTable locally without having to connect to a remote instance. Here you will find tools for interacting with emulators easily, used for setting up your local environment for development.\n\nInstall using pip (Python 3):\n\n```\npip install emulator-tools\n```\n\n# Emulators\n\n## Basic Setup\n\nEmulators come with the [`gcloud`](https://cloud.google.com/sdk/install) SDK. The basic setup for all emulators is very similar. You have to first start up the emulator and then make sure to set the environment variables for the project that's going to use these emulators as well as for the emulator tools to connect to the emulator rather than your remote project.\n\n### Start\n\nStart an emulator (set the project according to what your code uses)\n\n```\ngcloud beta emulators pubsub start --log-http --project your-project-id\ngcloud beta emulators bigtable start --log-http --host-port=localhost:8428 --project your-project-id\n```\n\n### Set Environment\n\nThen set the environment variables for your program before running it. You also need to set the environment for using emulator tools.\n\n```\n$(gcloud beta emulators pubsub env-init)\necho $PUBSUB_EMULATOR_HOST # verify the environment variable is set\n$(gcloud beta emulators bigtable env-init)\necho $BIGTABLE_EMULATOR_HOST\n```\n\n## PubSub\n\nAfter [starting the emulator](#start), [set the environment](#set-environment) and get started with setting up pubsub.\n\nFirst off, check to make sure you are connected to your emulator and not the production instance.\n\n```\nemulator-tools pubsub \u003cproject-id\u003e list-topics\n```\n\nThe result should be empty, otherwise if you see a list of topics, you are connected to production! Make sure your environment variables are set properly.\nAfter you have checked the connection, you can create topics as needed by the component (you can usually find these topic names in configuration files of the app):\n\n```\nemulator-tools pubsub \u003cproject-id\u003e create-topic \u003ctopic-name\u003e\n```\n\nThen create subscriptions:\n```\nemulator-tools pubsub \u003cproject-id\u003e create-subscription \u003ctopic-name\u003e \u003csubscription-name\u003e\n```\n\nYou can check to validate your topic and subscription:\n```\nemulator-tools pubsub \u003cproject-id\u003e list-topics\n\nemulator-tools pubsub \u003cproject-id\u003e list-subscriptions-in-topic \u003ctopic-name\u003e\n\nemulator-tools pubsub \u003cproject-id\u003e list-subscriptions-in-project\n```\n\nAfterwards, you can test publishing a message and receiving it:\n```\ncat sample-message.json | emulator-tools pubsub \u003cproject-id\u003e publish \u003ctopic-name\u003e\n\nemulator-tools pubsub \u003cproject-id\u003e receive-messages \u003csubscription-name\u003e\n```\n\nIf you see \"Received message: Message {\" including the content of the file `sample.json`, you are all set!\n\nYou can now connect your project to this emulator and use these scripts to send envelopes to it.\n\nIf you want a sample envelope for your project, you can get on by going to the production pubsub subscription for the app and use \"View Messages\" and then \"Pull\" to get a sample message. Sometimes you might have to click \"Pull\" multiple times to get some messages. Just take care NOT to \"Ack\" the message.\n\n![view-messages](pics/view-messages.png)\n![pull](pics/pull.png)\n\n## BigTable\n\nAfter [starting the emulator](#start), [set the environment](#set-environment) and get started with setting up BigTable.\n\nFirst off, check to make sure you are connected to your emulator and not the production instance.\n\n```\nemulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e list-tables\n```\n\nThe result should be empty, otherwise if you see a list of tables, you are connected to production! Make sure your environment variables are set properly.\nAfter you have checked the connection, you can create tables as needed by the component (you can usually find these table names in configuration files of the app). For creating the table you need to provide the list of column families and column types in a JSON format:\n\n```\n{\n  \"column_families\": [{\n    \"name\": \"i\",\n    \"columns\": [{\n      \"key\": \"id\",\n      \"type\": \"long\"\n    }, {\n      \"key\": \"score\",\n      \"type\": \"double\"\n    }, {\n      \"key\": \"title\",\n      \"type\": \"string\"\n    }]\n  }, {\n    \"name\": \"m\",\n    \"columns\": [{\n      \"key\": \"namespace\",\n      \"type\": \"string\"\n    }]\n  }]\n}\n```\n\n```\ncat sample-table.json | emulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e create-table \u003ctable-name\u003e\n```\n\nYou can now insert a new row using json files of this format (see `sample-row.json`):\n```\n{\n  \"rows\": [\n    {\n      \"rowkey\": \"testing#12345\",\n      \"columns\": [\n        {\n          \"key\": \"i:id\",\n          \"value\": 1234,\n          \"timestamp\": 1579607960.80492\n        },\n        {\n          \"key\": \"i:title\",\n          \"value\": \"Here's a test title for you\"\n        },\n        {\n          \"key\": \"i:score\",\n          \"value\": 1.5\n        },\n        {\n          \"key\": \"m:namespace\",\n          \"value\": \"core\"\n        }\n      ]\n    }\n  ]\n}\n```\n\n```\ncat sample-row.json | emulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e write \u003ctable-name\u003e sample-table.json\n```\n\nTo read the rows:\n```\nemulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e read \u003ctable-name\u003e sample-table.json\n\n# for specific rowkeys\nemulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e read \u003ctable-name\u003e sample-table.json --rowkey testing:.*\n```\n\nYou can also use [cbt](https://cloud.google.com/bigtable/docs/cbt-reference), the official tool for interacting with BigTable to interact with your emulated environment. Just be careful to set the environment properly and not to interact with the production instance.\n\n```\n# list tables\ncbt -project \u003cproject-id\u003e -instance \u003cinstance-id\u003e ls\n\n# list column families in table\ncbt -project \u003cproject-id\u003e -instance \u003cinstance-id\u003e ls \u003ctable-name\u003e\n\n# read rows\ncbt -project \u003cproject-id\u003e -instance \u003cinstance-id\u003e read \u003ctable-name\u003e\n```\n\nIf you want some sample data for your project, you can import this data from the production instance to your local emulator. To do that, read from the remote instance and write it down to a file, then write it to your local emulator in another terminal with the correct environment variables set.\n\n```\n# in a terminal with no emulator env variables\nemulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e read \u003ctable-name\u003e sample-table.json --limit 5 --json \u003e data.json\n\n# in a terminal with emulator environment set\ncat data.json | emulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e write \u003ctable-name\u003e sample-table.json\n```\n\nYou should then be able to inspect this data in your emulated environment:\n```\nemulator-tools bigtable \u003cproject-id\u003e \u003cinstance-id\u003e read \u003ctable-name\u003e sample-table.json\n```\n\n# Services Model\n\nThe way we use these scripts at AYLIEN is that we have a `setup-emulators.sh` script for each of our services interacting with BigTable and PubSub, and these scripts are responsible for setting up the necessary tables, topics, subscriptions and sending a sample message upon running. This means getting started is as easy as running the app with emulator environment variables and running `setup-emulators.sh`.\n\nA sample script might look like this:\n\n```\n$(gcloud beta emulators bigtable env-init)\n$(gcloud beta emulators pubsub env-init)\n\nproject=\"awesome-project-staging\"\n\nemulator-tools pubsub $project create-topic test\nemulator-tools pubsub $project create-subscription test test\n\ncat ./messages/love-letter.json | emulator-tools pubsub $project publish test\n```\n\n# Development\n\nTo develop the tools, you need to install python and the requirements, if using `pyenv`:\n\n```\npyenv install $(cat .python-version)\npyenv local\npip install -r requirements.txt\n```\n\n# Known Issues\n\n## Alpakka doesn't connect to my PubSub emulator!\n\nIt seems Alpakka PubSub GRPC connector does not support the emulator environment variables yet (but the [http version does](https://github.com/akka/alpakka/pull/299)). If you want to connect to the emulator while using the GRPC you may have to manually configure alpakka to point to your emulator in. To do so, add these configurations in your `application.conf` file:\n```\n// Overriding configuration for PubSub Emulator\nalpakka.google.cloud.pubsub.grpc {\n  host = \"localhost\"\n  port = \"8085\"\n  rootCa = \"none\"\n  callCredentials = \"none\"\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faylien%2Femulator-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faylien%2Femulator-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faylien%2Femulator-tools/lists"}