{"id":25972053,"url":"https://github.com/tomaston1996/raspi-streamer","last_synced_at":"2026-02-22T23:05:41.250Z","repository":{"id":279107433,"uuid":"937735730","full_name":"TomAston1996/raspi-streamer","owner":"TomAston1996","description":"🤖 Raspberry Pi IoT data streaming with AWS (IoT Core, SNS, SQS, Lambda). Utilise an event-driven pipeline with a FastAPI service running in ECS to query IoT data from DynamoDB.","archived":false,"fork":false,"pushed_at":"2025-04-08T18:11:31.000Z","size":203,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-02T06:33:13.612Z","etag":null,"topics":["aws","dynamodb","ecs","iot","lambda","raspberry-pi","sns","sqs"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TomAston1996.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-23T19:18:43.000Z","updated_at":"2025-08-27T16:24:57.000Z","dependencies_parsed_at":"2025-02-23T20:29:44.222Z","dependency_job_id":"010bf5d4-af21-4207-a820-659d531942c7","html_url":"https://github.com/TomAston1996/raspi-streamer","commit_stats":null,"previous_names":["tomaston1996/raspi-streamer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TomAston1996/raspi-streamer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomAston1996%2Fraspi-streamer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomAston1996%2Fraspi-streamer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomAston1996%2Fraspi-streamer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomAston1996%2Fraspi-streamer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TomAston1996","download_url":"https://codeload.github.com/TomAston1996/raspi-streamer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TomAston1996%2Fraspi-streamer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273244306,"owners_count":25070958,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-02T02:00:09.530Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","dynamodb","ecs","iot","lambda","raspberry-pi","sns","sqs"],"created_at":"2025-03-05T00:18:57.126Z","updated_at":"2026-02-22T23:05:41.167Z","avatar_url":"https://github.com/TomAston1996.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Contributors][contributors-shield]][contributors-url]\n[![Forks][forks-shield]][forks-url]\n[![Stargazers][stars-shield]][stars-url]\n[![Issues][issues-shield]][issues-url]\n[![MIT License][license-shield]][license-url]\n[![LinkedIn][linkedin-shield]][linkedin-url]\n\n# ᯤ Raspberry Pi IoT Streamer\n\nThe goal of this project is to extract performance metrics (such as CPU utilisation etc.) from a Raspberry Pi and feed it into AWS IoT Core.\nIoT Core rules will publish events to AWS middleware which will in turn transform and store the data in DynamoDB. A simple FastAPI app will be hosted on ECS\nfor user access to the data.\n\n## 🧑‍💻 Tech Stack\n\n![Python]\n![AWS]\n![Docker]\n![FastAPI]\n![Raspberry Pi]\n\n## 🏢 Architecture\n\nBelow is the data flow description:\n1. __Raspberry Pi__ publishes CPU usage data as MQTT messages to AWS IoT Core.\n2. __IoT Core__ applies an IoT Rule to extract relevant fields and forward the message to SNS.\n3. __SNS (Simple Notification Service)__ pushes the message to an SQS Queue.\n4. __Lambda Function__ polls the __SQS Queue__, processes the message, and stores it in DynamoDB.\n5. __DynamoDB__ keeps a historical record of CPU metrics, allowing for future queries and analysis.\n6. A __FastAPI__ app hosted on __ECS__ provides a REST API interface for the client to retrieve CPU usage data from DynamoDB.\n\n![alt text](https://github.com/TomAston1996/raspi-streamer/blob/main/images/rpi-iot-project.png?raw=true)\n\n\n## ⚙️ Setup\n\n### Prerequisites\n| Dependencies | Install Guide | Applicable Devices |\n|--------------|---------------|--------------------|\n| AWS CLI |  [aws-cli-download](https://aws.amazon.com/cli/) | Local PC/Raspberry Pi |\n| AWS SAM CLI| [sam-cli-download](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) | Local PC/Raspberry Pi |\n| Python 3.11 (or Later)| [python-download](https://www.python.org/downloads/) | Local PC/Raspberry Pi |\n| Docker | [docker-download](https://www.docker.com/products/docker-desktop/) | Local PC |\n| UV| [uv-github](https://github.com/astral-sh/uv) | Local PC/Raspberry Pi |\n\n### AWS ETL Pipeline Setup\n\n1. Clone the repository onto your Local PC.\n```\ngit clone https://github.com/TomAston1996/raspi-streamer.git\ncd rasppi-streamer\n```\n2. Configure your AWS credentials in the aws CLI using ```aws configure```\n```\nAWS Access Key ID [None]: YOUR_ACCESS_KEY_ID\nAWS Secret Access Key [None]: YOUR_SECRET_ACCESS_KEY\nDefault region name [None]: YOUR_REGION (e.g., us-east-1)\nDefault output format [None]: json (or text, table)\n```\n3. Move into the sam directory with ```cd .\\aws\\sam```\n4. Compile and package code with ```sam build```\n5. Deploy your code to AWS with ```sam deploy```\n6. Navigate to AWS to set up IoT Core or this can also be done in the terminal. The following steps are required:\n   - Create an IoT Thing\n   - Generate certificates for AWS certificate, private key, public key and root CA and attach to the IoT Thing.\n   - Create an IoT Policy and attach it to the certificate.\n   - Save the IoT Core endpoint for Raspberry Pi configuration later.\n   - You can test the set up later using MQTT Test Client once the Raspberry Pi has been set up.\n\n### AWS ECS API Setup\n\n#### Create Cognito User Pool for Authentication\n1. Create a cognito user pool in AWS.\n2. Enable the ```ALLOW_USER_PASSWORD_AUTH``` OAuth flow in your App Client.\n3. Ensure the following parameters are updated in your ```.env``` file\n```\nCOGNITO_USER_POOL_ID=\u003cyour_user_pool_id\u003e\nCOGNITO_USER_POOL_CLIENT_ID=\u003cyour_app_client_id\u003e\nCOGNITO_USER_POOL_REGION=\u003cyour_aws_region\u003e\nCOGNITO_CLIENT_SECRET=\u003cyour_client_secret_for_secret_hash\u003e\n```\n\n#### Deploy Docker File to ECS\n1. Build docker image locally with ```docker build -t my-app .```\n2. Tag the image ```docker tag my-app:latest \u003caws_account_id\u003e.dkr.ecr.\u003cyour-region\u003e.amazonaws.com/my-app:latest```\n3. Push the image to ECR ```docker push \u003caws_account_id\u003e.dkr.ecr.\u003cyour-region\u003e.amazonaws.com/my-app:latest```\n4. Create an ECS cluster.\n5. Create a new task definition with the appropriate port mappings.\n6. Create and attach an IAM role to allow your ECS task to perform CRUD operations on DynamoDB.\n7. Ensure the following environment variables are set in your ```.env``` file for dev and prod environments.\n```\nDB_TABLE_NAME=\u003cyour_table_name\u003e\nDYNAMODB_ENDPOINT=\u003cyour_local_docker_endpoint\u003e\nDYNAMODB_REGION=\u003cyour_aws_region\u003e\n```\n\n### Raspberry Pi Setup\n1. Clone the repository onto your Raspberry Pi\n```\ngit clone https://github.com/TomAston1996/raspi-streamer.git\ncd raspi-streamer\n```\n2. Install dependencies with UV\n```\nuv pip install -r pyproject.toml\n#or\nuv sync\n```\n3. Create a certificates directory in the app folder to hold IoT Core certificates and ensure to add the folder to your ```.gitignore```\n4. Place the following certificates and keys in the folder for use when authenticating with IoT Core:\n\n| Certificate Name | Description  |\n| ---------------- | -------------|\n| ```certificate.pem.crt``` | Device’s unique identity certificate issued by AWS IoT Core or a Certificate Authority (CA) |\n| ```private.pem.key``` | Private key associated with the device certificate obtained from IoT Core. |\n| ```public.pem.key``` | Public key corresponding to the private key obtained from IoT Core. |\n| ```rootCA.pem``` |  Root CA certificate that is used to verify the authenticity of the device certificate. ```wget -O rootCA.pem https://www.amazontrust.com/repository/AmazonRootCA1.pem``` |\n7. Create a ```.env``` file in the root directory with the certificate file paths like below:\n```\nRPI_AWS_IOT_ENDPOINT=\u003cstring\u003e-ats.iot.eu-east-1.amazonaws.com\nRPI_AWS_IOT_CERTIFICATE=\u003cpath\u003e-certificate.pem.crt\nRPI_AWS_IOT_PRIVATE_KEY=\u003cpath\u003e--private.pem.key\nRPI_AWS_IOT_ROOT_CA=\u003cpath\u003eAmazonRootCA1.pem\n```\n6. Run ```uv run .\\raspberry_pi\\src\\main.py``` to start sending CPU metric data to the IoT Core topic from your Raspberry Pi\n7. Now you can check you're data is being entered into DynamoDB.\n\n## 🧑‍🤝‍🧑 Developers \n\n| Name           | Email                      |\n| -------------- | -------------------------- |\n| Tom Aston      | mailto:mail@tomaston.dev     |\n\n\u003c!-- MARKDOWN LINKS \u0026 IMAGES --\u003e\n\u003c!-- https://www.markdownguide.org/basic-syntax/#reference-style-links --\u003e\n[contributors-shield]: https://img.shields.io/github/contributors/TomAston1996/raspi-streamer.svg?style=for-the-badge\n[contributors-url]: https://github.com/TomAston1996/raspi-streamer/graphs/contributors\n[forks-shield]: https://img.shields.io/github/forks/TomAston1996/raspi-streamer.svg?style=for-the-badge\n[forks-url]: https://github.com/TomAston1996/raspi-streamer/network/members\n[stars-shield]: https://img.shields.io/github/stars/TomAston1996/raspi-streamer.svg?style=for-the-badge\n[stars-url]: https://github.com/TomAston1996/raspi-streamer/stargazers\n[issues-shield]: https://img.shields.io/github/issues/TomAston1996/raspi-streamer.svg?style=for-the-badge\n[issues-url]: https://github.com/TomAston1996/raspi-streamer/issues\n[license-shield]: https://img.shields.io/github/license/TomAston1996/raspi-streamer.svg?style=for-the-badge\n[license-url]: https://github.com/TomAston1996/raspi-streamer/blob/master/LICENSE.txt\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge\u0026logo=linkedin\u0026colorB=555\n[linkedin-url]: https://linkedin.com/in/tomaston96\n[Python]: https://img.shields.io/badge/python-3670A0?style=for-the-badge\u0026logo=python\u0026logoColor=ffdd54\n[Pandas]: https://img.shields.io/badge/pandas-%23150458.svg?style=for-the-badge\u0026logo=pandas\u0026logoColor=white\n[AWS]: https://img.shields.io/badge/AWS-%23FF9900.svg?style=for-the-badge\u0026logo=amazon-aws\u0026logoColor=white\n[Docker]: https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white\n[FastAPI]: https://img.shields.io/badge/FastAPI-005571?style=for-the-badge\u0026logo=fastapi\n[Raspberry Pi]: https://img.shields.io/badge/-Raspberry_Pi-C51A4A?style=for-the-badge\u0026logo=Raspberry-Pi\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomaston1996%2Fraspi-streamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftomaston1996%2Fraspi-streamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftomaston1996%2Fraspi-streamer/lists"}