{"id":26267307,"url":"https://github.com/oslabs-beta/turnstyl","last_synced_at":"2025-10-06T15:39:44.624Z","repository":{"id":44735114,"uuid":"396979203","full_name":"oslabs-beta/turnstyl","owner":"oslabs-beta","description":"turnstyl is an open-source ETL data integrity monitoring node package built to work with event and messaging brokers like Kafka","archived":false,"fork":false,"pushed_at":"2022-01-27T21:58:02.000Z","size":2033,"stargazers_count":83,"open_issues_count":1,"forks_count":14,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-31T20:36:20.618Z","etag":null,"topics":["kafka","nodejs"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oslabs-beta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-16T21:29:00.000Z","updated_at":"2024-07-30T08:45:15.000Z","dependencies_parsed_at":"2022-08-27T17:01:56.625Z","dependency_job_id":null,"html_url":"https://github.com/oslabs-beta/turnstyl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oslabs-beta/turnstyl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2Fturnstyl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2Fturnstyl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2Fturnstyl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2Fturnstyl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oslabs-beta","download_url":"https://codeload.github.com/oslabs-beta/turnstyl/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oslabs-beta%2Fturnstyl/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278328240,"owners_count":25968917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","nodejs"],"created_at":"2025-03-14T04:16:26.796Z","updated_at":"2025-10-06T15:39:44.589Z","avatar_url":"https://github.com/oslabs-beta.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003cimg align=\"center\" src=\"https://raw.githubusercontent.com/oslabs-beta/turnstyl/main/assets/turnstile-logo-softwhite-v3-a.png\"\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg alt=\"npm version\" src=\"https://img.shields.io/npm/v/turnstyl\" /\u003e\n  \u003cimg alt=\"Last commit\" src=\"https://img.shields.io/github/last-commit/oslabs-beta/turnstyl\" /\u003e\n  \u003cimg alt=\"Github stars\" src=\"https://shields.io/github/stars/oslabs-beta/turnstyl\" /\u003e\n  \u003cimg alt=\"Snyk\" src=\"https://snyk.io/advisor/npm-package/turnstyl/badge.svg\" href='https://snyk.io/advisor/npm-package/turnstyl' /\u003e\n\u003c/p\u003e\n\nTurnstyl is an NPM package built to work with Kafka and Google Big Query to allow you to compare the data schema input at the producer level to what arrives into your data warehouse.\n\nThe aim of Turnstyl is to monitor and notify the user of data validity issues in your data pipeline, specifically at the fringe between backend and ETL systems. Turnstyl was primarily designed for use in micro-service architecture where events are dumped into a data warehouse in a generic and schema-agnostic format.\n\n## ✅ Pre-requisites\n\nTo run this package you'll need to have:\n\n- A working instance of Kafka or equivalent message broker\n- Be setup and using Google BigQuery as your data warehouse\n- Have activated the Google API and setup a [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts)\n\n## 🚀 QuickStart\n\nGetting started with Turntstyl is easy:\n\n1. In your terminal run `npm install turnstyl` in your target project directory\n2. Create a `turnstyl.config.yaml` in your project root directory, using the template below to input your Google Big Query project and dataset names, and the path to your Google API service account credentials JSON.\n\nExample `turnstyl.config.yaml` file:\n\n```yaml\nversion: '1.0'\n\ngoogle_service_credentials: './GOOGLE_BIG_QUERY_SERVICE_ACCOUNT_CREDENTIALS.json'\nbig_query_project_name: 'BIG_QUERY_PROJECT_NAME'\nbig_query_dataset_name: 'BIG_QUERY_DATASET_NAME'\n```\n\n3. Import the package into the service where data is being sent to your producer\n\n`const { Turnstyl } = require(\"turnstyl\");`\n\n4. Instantiate a new `Turnstyl` object and invoke the following two methods:\n\n`const newTurnstyl = new Turnstyl();`\n\n- `newTurnstyl.cacheProducerEvent(topic, message);` - Caches your events where they are produced (or in the producer) and with reference to the topic that they will be sent through, parsing the object for it's data types.\n- `newTurnstyl.compareProducerToDBSchema(topic);` - Compares the events that have arrived into Big Query (assuming the topic name matches your target table) flagging if there is a discrepancy between the two.\n\n## ✨ Features\n\n- Compare data object processed by the producer to the JSON deposited in the data warehouse\n- Integration with Google Cloud Platform (Big Query)\n\n## 🛠️ Example usage\n\n```js\nconst { Kafka } = require('kafkajs');\nconst { Turnstyl } = require('turnstyl');\n\nconst newTurnstyl = new Turnstyl();\n\n/**\n * @function producer function that connects to Kafka sends a message then disconnects\n * @param producerName STRING name of producer\n * @param message STRING message that will be sent to Kafka\n * @param topic STRINg name of the topic that message will be posed to on Kafka\n */\nconst producer = async (\n  producerName\n  message\n  topic\n) =\u003e {\n  //Declare a variable kafka assigned to an instance of kafka (door into the kafka brokerage)\n  const kafka = new Kafka({\n    clientId: producerName,\n    brokers: ['kafka:9092'],\n  });\n  const producer = kafka.producer();\n\n  newTurnstyl.cacheProducerEvent(topic, message);\n  newTurnstyl.compareProducerToDBSchema(topic);\n\n  try {\n    // Connect to the producer\n    await producer.connect();\n  } catch (error) {\n    console.log('Producer Connection error: ', error);\n  }\n\n  try {\n    // Send message\n    await producer.send({\n      topic: topic,\n      messages: [{ value: JSON.stringify(message) }],\n    });\n    console.log('message is:', message);\n  } catch (error) {\n    console.log('error in message send', error);\n  }\n  console.log('Data sent by producer');\n  // Close connection to the broker\n  producer.disconnect();\n};\n\nexport { producer };\n```\n\n## 🏭 Supported Message brokers\n\n- Apache Kafka\n\n## 🏭 Supported Data Warehouses\n\n- Google Big Query\n\n## 📸 Screenshots\n\n\u003cimg align=\"center\" src=\"https://raw.githubusercontent.com/oslabs-beta/turnstyl/main/assets/screenshot_error_flags.png\"\u003e\n\n## 🚗 Roadmap\n\n[Head to our roadmap](https://homespot.notion.site/330ba936328b488e86e6d7395003518b?v=0ae19b46612b43f69630e4cd60a0b0a0) to see our upcoming planned features\n\n## 👍 Contribute\n\nWant to contribute to Turnstyl? Head over to our [contribution page](https://github.com/oslabs-beta/turnstyl/blob/main/CONTRIBUTING.md)\n\n### Contributors:\n\n[Jae Kim](https://github.com/jaeklm) - [LinkedIn](https://www.linkedin.com/in/jaeklm/)\n\n[Yolan Arnett](https://github.com/yarnett) - [LinkedIn](https://www.linkedin.com/in/yolanarnett/)\n\n[Dillon Schriver](https://github.com/Dillon-Schriver) - [LinkedIn](https://www.linkedin.com/in/dillon-schriver-236a16bb/)\n\n[Emeric David](https://github.com/emeric-gh) - [LinkedIn](https://www.linkedin.com/in/emericbdavid/)\n\n[Michael Colley](https://github.com/michaelecolley) - [LinkedIn](https://www.linkedin.com/in/michaelecolley/) - [Twitter](https://twitter.com/michaelecolley)\n\nIf you'd like to support the active development of Turnstyl:\n\n- Add a GitHub Star to the project\n- Tweet about the project on your Twitter.\n- Write a review or tutorial on Medium, Dev.to or personal blog.\n\n## 🎟️ License\n\n`Turnstyl` is free and open-source software licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslabs-beta%2Fturnstyl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foslabs-beta%2Fturnstyl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foslabs-beta%2Fturnstyl/lists"}