{"id":43903268,"url":"https://github.com/singlestore-labs/singlestore-logistics-sim","last_synced_at":"2026-02-06T19:20:38.789Z","repository":{"id":49184187,"uuid":"371157749","full_name":"singlestore-labs/singlestore-logistics-sim","owner":"singlestore-labs","description":"Scalable package delivery logistics simulator built using SingleStore and Vectorized Redpanda","archived":false,"fork":false,"pushed_at":"2024-03-08T16:31:51.000Z","size":1829,"stargazers_count":37,"open_issues_count":0,"forks_count":10,"subscribers_count":3,"default_branch":"main","last_synced_at":"2026-01-18T02:14:36.239Z","etag":null,"topics":["avro","demo","kafka","logistics","redpanda","simulation","singlestore","vectorized"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/singlestore-labs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-26T20:12:56.000Z","updated_at":"2025-11-30T17:16:07.000Z","dependencies_parsed_at":"2022-09-19T02:50:47.962Z","dependency_job_id":null,"html_url":"https://github.com/singlestore-labs/singlestore-logistics-sim","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/singlestore-labs/singlestore-logistics-sim","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-logistics-sim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-logistics-sim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-logistics-sim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-logistics-sim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/singlestore-labs","download_url":"https://codeload.github.com/singlestore-labs/singlestore-logistics-sim/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/singlestore-labs%2Fsinglestore-logistics-sim/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29173490,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T16:33:35.550Z","status":"ssl_error","status_checked_at":"2026-02-06T16:33:30.716Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","demo","kafka","logistics","redpanda","simulation","singlestore","vectorized"],"created_at":"2026-02-06T19:20:38.141Z","updated_at":"2026-02-06T19:20:38.781Z","avatar_url":"https://github.com/singlestore-labs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reference Architecture using SingleStore and Redpanda for global logistics\n\n**Attention**: The code in this repository is intended for experimental use only and is not fully tested, documented, or supported by SingleStore. Visit the [SingleStore Forums](https://www.singlestore.com/forum/) to ask questions about this repository.\n\n\u003e **📯 INFO:** For the story behind this code (and epic dashboards), [check out the blog post](https://www.singlestore.com/blog/scaling-worldwide-parcel-logistics-with-singlestore-and-vectorized/)!\n\nThis repository provides a reference architecture which simulates the parcel\ntracking system of a hypothetical global logistics company using\n[SingleStore](https://singlestore.com) and [Redpanda](https://vectorized.io/).\n\n## Architecture overview\n\n![architecture diagram](images/architecture.png)\n\n## Data model\n\n![data model](images/data_model.png)\n\n## Running locally\n\n1. [Sign up](https://www.singlestore.com/try-free/) for a free SingleStore license.  This allows you to run up to 4 nodes up to 32 gigs each for free. Grab your license key from [SingleStore portal](https://portal.singlestore.com/?utm_medium=osm\u0026utm_source=github) and set it as an environment variable.\n\n```bash\nexport SINGLESTORE_LICENSE=\"\u003c\u003csinglestore license\u003e\u003e\"\n```\n\n2. Run the simulation locally using `make` and then check that everything is running correctly using `docker-compose ps`\n\n```bash\n$ make\n...SNIP output...\nSuccessfully built fad3bf30af1d\nSuccessfully tagged singlestore-logistics-simulator:latest\nCreating simulator ... done\n\n$ docker-compose ps\n   Name                 Command               State\n---------------------------------------------------\ngrafana      /run.sh                          Up\nprometheus   /bin/prometheus --config.f ...   Up\nrp-node-0    /usr/bin/rpk redpanda star ...   Up\nrp-setup     /bin/bash -c rpk --brokers ...   Exit 0\ns2-agg-0     /startup                         Up\nsimulator    ./simulator --config confi ...   Up\n```\n\n\u003e **NOTE:** If you are running the above on a mac you may need to increase the docker system resources. We have found that 6 GB of RAM and 2 CPUs is sufficient to run this demo - but the more you provide to docker the faster things will go. :)\n\n3. You can also check the logs using\n\n```bash\nmake logs\n\n# or specify a specific service like so:\nmake logs SERVICE=simulator\n```\n\nAt this point you can open up any of the services below to view dashboards, cluster information or run queries against the dataset:\n\n| service     | url                   | user | password |\n|-------------|-----------------------|------|----------|\n| Grafana     | http://localhost:3000 | root | root     |\n| SingleStore | http://localhost:8080 | root | root     |\n| Prometheus  | http://localhost:9090 |      |          |\n\n\u003e **NOTE**: The node exporter is not run locally, so the node exporter dashboard will not work out of the box.\n\n## Deploying into Google Cloud\n\nThis repo contains a full Terraform based deployment solution which will result in this simulator running at nearly unlimited scale in the cloud.\n\n\u003e **WARNING**: Before running this in your own environment be cautious to read the variables carefully and tune it to your own appetite. **It's easy to burn a lot of money very quickly.** SingleStore and Vectorized do not take responsibility for any costs or issues you may have while using this repository. Please be careful.\n\nWithout going into too much detail (if you want to do this, you need to understand how Terraform works), follow these steps to run everything in Google Cloud:\n\n1. Create `deploy/terraform-gcp/terraform.tfvars` to set variables. You will need at least these variables set to continue:\n\n```\nproject_name = \"YOUR PROJECT NAME\"\nregion = \"THE REGION YOU WANT TO USE\"\nzone = \"THE ZONE YOU WANT TO USE: i.e. a, b, c\"\n\ns2_license = \"YOUR SINGLESTORE LICENSE KEY - GET THIS FROM THE PORTAL\"\n```\n\n2. Deploy! (make sure to read everything carefully)\n\n```bash\ncd deploy/terraform-gcp\nterraform apply\n```\n\n3. Grafana, Studio, and Prometheus are running locally on the logistics-dashboard which you can access via the following port forwarding options:\n\n```bash\ngcloud compute ssh logistics-dashboard -- -L 9090:localhost:9090 -L 3000:localhost:3000 -L 8080:localhost:8080\n```\n\n## Redpanda topic schemas\n\nThe [simulator](simulator) is a go program which generates package histories and writes them into Redpanda topics.\n\nThere are two topics:\n - packages\n - transitions\n\n## Packages topic\n\nThe packages topic contains a record per package. The record is written when we receive the package in question.\n\n**Avro schema**:\n\n```json\n{\n    \"type\": \"record\",\n    \"name\": \"Package\",\n    \"fields\": [\n        { \"name\": \"PackageID\", \"type\": { \"type\": \"string\", \"logicalType\": \"uuid\" } },\n        { \"name\": \"SimulatorID\", \"type\": \"string\" },\n        { \"name\": \"Received\", \"type\": { \"type\": \"long\", \"logicalType\": \"timestamp-millis\" } },\n        { \"name\": \"DeliveryEstimate\", \"type\": { \"type\": \"long\", \"logicalType\": \"timestamp-millis\" } },\n        { \"name\": \"OriginLocationID\", \"type\": \"long\" },\n        { \"name\": \"DestinationLocationID\", \"type\": \"long\" },\n        { \"name\": \"Method\", \"type\": { \"name\": \"Method\", \"type\": \"enum\", \"symbols\": [\n            \"standard\", \"express\"\n        ] } }\n    ]\n}\n```\n\n## Transitions topic\n\nThe transitions topic is written to whenever a package changes states. A normal package goes through the following transitions during it's lifetime:\n\n1. arrival scan - the package has been received\n2. departure scan - the package has been scanned and put in transit to another location\n    * arrival scan and departure scan can occur multiple times as the package moves through our global logistics network\n3. delivered - the package has been delivered\n\n\u003e **NOTE**: We don't currently model last-mile delivery, but it's an interesting problem space for a future iteration on this project.\n\n**Avro schema**:\n\n```json\n{\n    \"type\": \"record\",\n    \"name\": \"PackageTransition\",\n    \"fields\": [\n        { \"name\": \"PackageID\", \"type\": { \"type\": \"string\", \"logicalType\": \"uuid\" } },\n        { \"name\": \"Seq\", \"type\": \"int\" },\n        { \"name\": \"LocationID\", \"type\": \"long\" },\n        { \"name\": \"NextLocationID\", \"type\": [\"null\", \"long\"] },\n        { \"name\": \"Recorded\", \"type\": { \"type\": \"long\", \"logicalType\": \"timestamp-millis\" } },\n        { \"name\": \"Kind\", \"type\": { \"name\": \"Kind\", \"type\": \"enum\", \"symbols\": [\n            \"arrival_scan\", \"departure_scan\", \"delivered\"\n        ] } }\n    ]\n}\n```\n\n## Interesting queries\n\nPlease contribute interesting queries on the dataset as you find them!\n\n### Show the full history of a single package\n\nReplace the package id with a package you want to track.\n\n```sql\nSELECT\n    pt.seq,\n    pt.kind,\n    current_loc.city,\n    current_loc.country,\n    GEOGRAPHY_DISTANCE(current_loc.lonlat, destination.lonlat) / 1000 AS distance_to_destination,\n    pt.recorded\nFROM package_transitions pt\nINNER JOIN locations current_loc ON pt.locationid = current_loc.locationid\nINNER JOIN packages p ON pt.packageid = p.packageid\nINNER JOIN locations destination ON p.destination_locationid = destination.locationid\nWHERE pt.packageid = '516aa045-d8df-4363-b250-da335df82269'\nORDER BY seq DESC;\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinglestore-labs%2Fsinglestore-logistics-sim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsinglestore-labs%2Fsinglestore-logistics-sim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsinglestore-labs%2Fsinglestore-logistics-sim/lists"}