{"id":17932747,"url":"https://github.com/lapetitesouris/kuronososhiki","last_synced_at":"2025-04-03T11:17:20.841Z","repository":{"id":96859013,"uuid":"264481675","full_name":"LaPetiteSouris/KuronoSoshiki","owner":"LaPetiteSouris","description":"Data Stream Quality Control with Apache Kafka","archived":false,"fork":false,"pushed_at":"2020-05-17T08:41:39.000Z","size":442,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-09T00:41:20.106Z","etag":null,"topics":["dataquality","faust","kafka","kafka-streams"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LaPetiteSouris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-05-16T16:44:28.000Z","updated_at":"2020-05-19T10:31:27.000Z","dependencies_parsed_at":null,"dependency_job_id":"ef04c047-97f9-44c7-a4c7-3477b8852f84","html_url":"https://github.com/LaPetiteSouris/KuronoSoshiki","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaPetiteSouris%2FKuronoSoshiki","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaPetiteSouris%2FKuronoSoshiki/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaPetiteSouris%2FKuronoSoshiki/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LaPetiteSouris%2FKuronoSoshiki/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LaPetiteSouris","download_url":"https://codeload.github.com/LaPetiteSouris/KuronoSoshiki/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246989753,"owners_count":20865331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataquality","faust","kafka","kafka-streams"],"created_at":"2024-10-28T21:30:15.416Z","updated_at":"2025-04-03T11:17:20.817Z","avatar_url":"https://github.com/LaPetiteSouris.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Data Quality Control\n\n[![Kafka](https://img.shields.io/badge/streaming_platform-kafka-black.svg?style=flat-square)](https://kafka.apache.org)\n[![Docker Images](https://img.shields.io/badge/docker_images-confluent-orange.svg?style=flat-square)](https://github.com/confluentinc/cp-docker-images)\n[![Python](https://img.shields.io/badge/python-3.5+-blue.svg?style=flat-square)](https://www.python.org)\n\nThis is the implementation of my small blogpost: [To stream or to not stream. That is a question.](https://medium.com/@tunghoang/to-stream-or-to-not-stream-that-is-a-question-92cf0adbfe65).\n\nInspired by the article written here\n[Building A Streaming Fraud Detection System With Kafka And Python](https://florimond.dev/blog/articles/2018/09/building-a-streaming-fraud-detection-system-with-kafka-and-python/)\n\nThe codebase for Kafka/Zookeeper stack ithes taken from the same blog post, as the author has done a brilliant work of puting the whole Kafka stack togheter in `docker-compse`. Many thanks to this tremendous effort, which allows my work to be implemented much easier.\n\n## Install\n\nThis fraud detection system is fully containerised. You will need [Docker](https://docs.docker.com/install/) and [Docker Compose](https://docs.docker.com/compose/) to run it.\n\nYou simply need to create a Docker network called `kafka-network` to enable communication between the Kafka cluster and the apps:\n\n```bash\n$ docker network create kafka-network\n```\n\nAll set!\n\n## Quickstart\n\n- Spin up the local single-node Kafka cluster (will run in the background):\n\n```bash\n$ docker-compose -f docker-compose.kafka.yml up -d\n```\n\n- Check the cluster is up and running (wait for \"started\" to show up):\n\n```bash\n$ docker-compose -f docker-compose.kafka.yml logs -f broker | grep \"started\"\n```\n\n- Start the stack, including `gin`, `vodka` and `bourbon`. Summazie\n\n```\ngin: Kafka producer, take the sample JSON file and simulate a coming of stream\n\nvodka: cleaning service. Clean data and re-publish to a new topic. It also calculate sever data\nstats on the flight.\n\nbourbon: data quality watchdog. It calculates data stats from raw data\n```\nNOTE:  Only `bourbon` and `vodka` print out\ntheir log. `gin` produces message to Kafka silently.\n\n```bash\n$ docker-compose up\n```\n\n## Usage\n\nShow a stream of transactions in the topic `T` (optionally add `--from-beginning`):\n\n```bash\n$ docker-compose -f docker-compose.kafka.yml exec broker kafka-console-consumer --bootstrap-server localhost:9092 --topic T\n```\n\nTopics:\n\n- `streaming.user_activity`: raw generated user event\n- `derivedstream.cleaned_user_events`: user event cleaned by `vodka`\n- `derivedstream.quality_user_activity`: quality control , publish by `bourbon`\n\nExample message:\n\n```json\n{\"id\":1,\"first_name\":\"Barthel\",\"last_name\":\"Kittel\",\"email\":\"bkittel0@printfriendly.com\",\"gender\":\"Male\",\"ip_address\":\"130.187.82.195\",\"date\":\"06/05/2018\",\"country\":\"france\"}\n```\n\n## Teardown\n\nTo stop the transaction generator and fraud detector:\n\n```bash\n$ docker-compose down\n```\n\nTo stop the Kafka cluster (use `down`  instead to also remove contents of the topics):\n\n```bash\n$ docker-compose -f docker-compose.kafka.yml stop\n```\n\nTo remove the Docker network:\n\n```bash\n$ docker network rm kafka-network\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flapetitesouris%2Fkuronososhiki","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flapetitesouris%2Fkuronososhiki","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flapetitesouris%2Fkuronososhiki/lists"}