{"id":28545591,"url":"https://github.com/crate/devrel-mongo-cdc-demo","last_synced_at":"2026-05-18T07:32:22.754Z","repository":{"id":278583093,"uuid":"936106944","full_name":"crate/devrel-mongo-cdc-demo","owner":"crate","description":"Simple hotel front desk jobs demo code to show CDC syncronization of a MongoDB collection to CrateDB Cloud","archived":false,"fork":false,"pushed_at":"2025-03-12T11:44:59.000Z","size":48,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2026-04-26T20:39:08.812Z","etag":null,"topics":["cratedb","cratedb-examples","grafana-dashboard","mongodb","python"],"latest_commit_sha":null,"homepage":"https://cratedb.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/crate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-02-20T14:40:34.000Z","updated_at":"2025-11-12T18:14:20.000Z","dependencies_parsed_at":"2025-02-20T15:38:11.088Z","dependency_job_id":"2e6911a2-4267-4daa-9005-a4cf5a284b6d","html_url":"https://github.com/crate/devrel-mongo-cdc-demo","commit_stats":null,"previous_names":["simonprickett/mongodb-hotel-jobs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/crate/devrel-mongo-cdc-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crate%2Fdevrel-mongo-cdc-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crate%2Fdevrel-mongo-cdc-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crate%2Fdevrel-mongo-cdc-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crate%2Fdevrel-mongo-cdc-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/crate","download_url":"https://codeload.github.com/crate/devrel-mongo-cdc-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/crate%2Fdevrel-mongo-cdc-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33169209,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-18T05:43:36.989Z","status":"ssl_error","status_checked_at":"2026-05-18T05:43:19.133Z","response_time":71,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cratedb","cratedb-examples","grafana-dashboard","mongodb","python"],"created_at":"2025-06-09T23:08:10.892Z","updated_at":"2026-05-18T07:32:22.721Z","avatar_url":"https://github.com/crate.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MongoDB/CrateDB/Grafana CDC Demonstration\n\n## Introduction\n\nThis is a small Python project that demonstrates how a CrateDB database can be populated and kept in sync with a collection in MongoDB using Change Data Capture (CDC).  Before configuring and running this project, you should watch our [video walkthrough](https://www.youtube.com/watch?v=N8n-yg3ru8I).\n\nThis project uses a hotel's front desk for its sample data.  Imagine that guests can visit the front desk, or use the phone or the hotel's app to request different service items.  Each request raises a job (for example \"provide fresh towels\", \"call a taxi\", \"order room service food\") that is associated with a room number.\n\nThese jobs are stored as documents in a collection in MongoDB, and are updated periodically as staff members complete them - adding a completion time and the ID of the staff member who did the work.\n\nThis data is then replicated to CrateDB for analysis and visualization in Grafana.\n\nThe project includes Python scripts to create and update jobs in MongoDB, as well as an (optional) Grafana dashboard exported as a JSON file.\n\n![Screenshot of the Grafana Dashboard](grafana_dashboard_screenshot.png)\n\n## Prerequisites\n\nTo run this project you'll need to install the following software:\n\n* Python 3 ([download](https://www.python.org/downloads/)) - we've tested this project with Python 3.12 on macOS Sequoia.\n* Git command line tools ([download](https://git-scm.com/downloads)).\n* Your favorite code editor, to edit configuration files and browse/edit the code if you wish.  [Visual Studio Code](https://code.visualstudio.com/) is great for this.\n* Access to an instance of [CrateDB](https://console.cratedb.cloud) in the cloud.\n* Access to an instance of [MongoDB](https://www.mongodb.com/cloud/atlas/register) in the cloud.\n* Optional: Access to an instance of [Grafana](https://grafana.com/get/) in the cloud.\n\n## Getting the Code\n\nGrab a copy of the code from GitHub by cloning the repository.  Open up your terminal and change directory to wherever you store coding projects, then enter the following commands:\n\n```bash\ngit clone https://github.com/crate/devrel-mongo-cdc-demo.git\ncd devrel-mongo-cdc-demo\n```\n\n## Getting a CrateDB Database in the Cloud\n\nCreate a database in the cloud by first pointing your browser at [`console.cratedb.cloud`](https://console.cratedb.cloud/).\n\nLogin or create an account, then follow the prompts to create a \"CRFREE\" database on shared infrastructure in the cloud of your choice (choose from Amazon AWS, Microsoft Azure and Google Cloud).  \n\nOnce you've created your cluster, you'll see a \"Download\" button.  This downloads a text file containing a copy of your database hostname, port, username and password.  Make sure to download these as you'll need them later and won't see them again.  Your credentials will look something like this example (exact values will vary based on your choice of AWS/Google Cloud/Azure etc):\n\n```\nHost:              some-host-name.gke1.us-central1.gcp.cratedb.net\nPort (PostgreSQL): 5432\nPort (HTTPS):      4200\nDatabase:          crate\nUsername:          admin\nPassword:          the-password-will-be-here\n```\n\nWait until the cluster status shows a green status icon and \"Healthy\" status before continuing.  Note that it may take a few moments to provision your database.\n\n### Database Schema Setup (CrateDB)\n\nThis demo uses two tables, `jobs` and `staff`.  The `jobs` table is created by the CDC data synchronization process.  You'll need to create the `staff` table yourself by running the following SQL statements at your CrateDB console:\n\n```sql\nCREATE TABLE staff (\n  id INTEGER,\n  name TEXT\n);\n```\n\n```sql\nINSERT INTO\n  staff (id, name)\nVALUES\n  (1, 'Simon'),\n  (2, 'Alice'),\n  (3, 'Michael'),\n  (4, 'Stefan'),\n  (5, 'Alea');\n```\n\n## Getting a MongoDB Database in the Cloud\n\nYou'll need to create a MongoDB database in the cloud - do this for free with [MongoDB Atlas](https://www.mongodb.com/cloud/atlas/register).\n\nCreate an empty collection called `jobs` in your new MongoDB instance.\n\nYou should also create a role, a user, and configure IP access for CrateDB's CDC process.  Instructions for these steps can be found [here](https://cratedb.com/docs/cloud/en/latest/cluster/integrations/mongo-cdc.html#set-up-mongodb-atlas-authentication).\n\n## Setting up CDC from CrateDB Cloud\n\nThe next step is to set up a CDC integration in your CrateDB Cloud cluster.  Follow our instructions [here](https://cratedb.com/docs/cloud/en/latest/cluster/integrations/mongo-cdc.html#set-up-integration-in-cratedb-cloud) to do this.\n\nWhen setting up the target table, be sure to name it `jobs`, and to select `dynamic` as the object type in the dropdown.\n\n## Editing the Project Configuration File\n\nThe Python scripts use a `.env` file to store the MongoDB connection details in.  We've provided a template file that you can copy like this:\n\n```bash\ncp env.example .env\n```\n\nNow, edit `.env` as follows:\n\nSet the value of `MONGODB_URI` to be:\n\n`mongodb+srv://\u003cusername\u003e:\u003cpassword\u003e\u003e@\u003chostname\u003e/?retryWrites=true\u0026w=majority`\n\nreplacing `\u003cusername\u003e`, `\u003cpassword\u003e` and `\u003chostname\u003e` with the values for your MongoDB Atlas cluster.\n\nSet the value of `MONGODB_DATABASE` to be the name of the database in your MongoDB Atlas cluster.\n\nSet the value of `MONGODB_COLLECTION` to be `jobs`.\n\nSave your changes.\n\nRemember, this file contains secrets... don't commit it to source control!\n\n## Setting up a Python Environment\n\nYou should create and activate a Python Virtual Environment to install this project's dependencies into.  To do this, run the following commands:\n\n```bash\npython -m venv venv\n. ./venv/bin/activate\n```\n\nNow install the dependencies that this project requires:\n\n```bash\npip install -r requirements.txt\n```\n\n## Running the Python Code\n\n### Job Creator Component\n\nRun the job creator component to create new job documents in the `jobs` collection in MongoDB.  It will generate a random job, add it to the collection, then sleep for a random time before repeating the process.\n\n```bash\npython job_creator.py\n```\n\nYou can stop the job creator with `Ctrl-C`.\n\n### Job Completer Component\n\nRun the job completer component to update existing job documents in the `jobs` collection.  This component picks the oldest outstanding job in the collection, sleeps for a while to pretend to perform the work required, then updates the job with a completion time and a randomly chosen staff ID for the staff member that completed the job.\n\n```bash\npython job_completer.py\n```\n\nYou can stop the job completer with `Ctrl-C`.\n\nTo simulate constant workflow through the databases, run both the job creator and the job completer at the same time.\n\n## Some Example SQL Queries\n\nOnce you have data flowing from MongoDB to your CrateDB cluster, you can start to run some SQL queries.  At the CrateDB console, try some of these example queries.\n\n### How many jobs are outstanding?\n\n```sql\nselect\n  count(*)\nfrom\n  jobs as backlog\nwhere\n  document['completedAt'] is null;\n```\n\n### How many jobs have been completed?\n\n```sql\nselect\n  count(*)\nfrom\n  jobs as completed\nwhere\n  document['completedAt'] is not null;\n```\n\n### Outstanding jobs by type:\n\n```sql\nselect\n  document['job'] as job_type,\n  count(*) as backlog\nfrom\n  jobs\nwhere\n  document['completedAt'] is null\ngroup by\n  job_type\norder by\n  backlog desc\n```\n\n### Average time to complete a job:\n\n```sql\nselect\n  round(avg(document['completedAt'] - document['requestedAt']) / 1000) as job_avg_time\nfrom\n  jobs \nwhere\n  document['completedAt'] is not null\n```\n\n### League table of who has completed the most jobs:\n\n```sql\nselect\n  s.id,\n  s.name,\n  count(j.document) as jobs_completed\nfrom\n  staff s join jobs j on s.id = j.document['completedBy']\nwhere\n  document['completedAt'] is not null\ngroup by\n  s.id, s.name\norder by\n  jobs_completed desc\n```\n\n## Optional: Grafana Dashboard\n\nThe file `grafana_dashboard.json` contains an export of a [Grafana](https://grafana.com/get/) dashboard that visualizes some of the above queries. \n\nTo use this, you'll need to sign up for a free Grafana cloud instance and connect it to your CreateDB cloud cluster using a Posgres data source (see [Grafana documentation](https://grafana.com/docs/grafana/latest/datasources/postgres/)).\n\n## CrateDB Academy\n\nWant to learn more about CrateDB?  Take our free online \"CrateDB Fundamentals\" course, available now at the [CrateDB Academy](https://cratedb.com/academy/fundamentals/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrate%2Fdevrel-mongo-cdc-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcrate%2Fdevrel-mongo-cdc-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcrate%2Fdevrel-mongo-cdc-demo/lists"}