{"id":18037457,"url":"https://github.com/fpgmaas/stream-iot","last_synced_at":"2025-08-18T08:33:10.083Z","repository":{"id":191349956,"uuid":"684468039","full_name":"fpgmaas/stream-iot","owner":"fpgmaas","description":"An end-to-end workflow for processing streaming data on Azure.","archived":false,"fork":false,"pushed_at":"2024-09-20T11:49:36.000Z","size":402,"stargazers_count":15,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-06T01:41:18.186Z","etag":null,"topics":["airflow","azure","data-engineering","kafka","kubernetes","streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fpgmaas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-29T07:28:27.000Z","updated_at":"2025-04-04T04:21:17.000Z","dependencies_parsed_at":"2024-10-30T13:10:35.177Z","dependency_job_id":"af2af83a-063b-4aef-a3b0-28fcfa4cdcad","html_url":"https://github.com/fpgmaas/stream-iot","commit_stats":null,"previous_names":["fpgmaas/stream-iot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fpgmaas/stream-iot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpgmaas%2Fstream-iot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpgmaas%2Fstream-iot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpgmaas%2Fstream-iot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpgmaas%2Fstream-iot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fpgmaas","download_url":"https://codeload.github.com/fpgmaas/stream-iot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fpgmaas%2Fstream-iot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270965136,"owners_count":24676595,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-18T02:00:08.743Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","azure","data-engineering","kafka","kubernetes","streaming"],"created_at":"2024-10-30T13:10:30.708Z","updated_at":"2025-08-18T08:33:09.803Z","avatar_url":"https://github.com/fpgmaas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# stream-iot\n\nWelcome to **stream-iot**! This project contains an example of an end-to-end workflow for processing streaming data.\nThe workflow consists of mocking sensor data, channeling it through Kafka, and then storing the parsed data in a MongoDB database.\n\nI started this project as a way to learn and get some initial hands-on experience with Kafka. I hope that this project can help others get started on similar projects. At the same time, I am sure I can learn from others viewing this project. If you have feedback or improvement suggestions, please [create an issue](https://github.com/fpgmaas/stream-iot/issues/new) in this repository.\n\n\n## Tools \u0026 Technologies\n\n* Cloud - [**Azure**](https://azure.microsoft.com/en-us/)\n* Infrastructure as Code - [**Terraform**](https://www.terraform.io/)\n* Containerization - [**Docker**](https://www.docker.com/)\n* Container Orchestration - [**Kubernetes**](https://kubernetes.io/)\n* Monitoring - [**Prometheus**](https://prometheus.io), [**Grafana**](https://grafana.com/)\n* Workflow Orchestration - [**Airflow**](https://airflow.apache.org/)\n* Messaging \u0026 Stream Processing - [**Kafka**](https://kafka.apache.org/)\n* Data Storage - [**CosmosDB**](https://azure.microsoft.com/en-us/products/cosmos-db), [**MongoDB**](https://www.mongodb.com)\n* Languages - [**Python**](https://www.python.org/)\n\n\n## Architecture\n\n![Alt text](./architecture.png)\n\n\n## Prerequisites\n\n- You have [az cli](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) installed.\n- You have kubectl installed with [`az aks install-cli`](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli#connect-to-the-cluster).\n\n## Installation\n\nFollow the installation instructions in the README's of these directories in order:\n\n- [terraform](./terraform/README.md#installation)\n- [kafka](./kafka/README.md#installation)\n- [airflow](./airflow/README.md#installation)\n- [python](./python/README.md)\n- [monitoring](./monitoring/README.md) (optional)\n\nWhen this is done, open the Airflow UI with\n\n```sh\nkubectl port-forward svc/airflow-webserver 8080:8080 -n airflow\n```\n\nand trigger the DAG's manually.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffpgmaas%2Fstream-iot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffpgmaas%2Fstream-iot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffpgmaas%2Fstream-iot/lists"}