{"id":13623288,"url":"https://github.com/vikramaditya91/iot-ingest-large-size","last_synced_at":"2026-05-01T23:31:37.583Z","repository":{"id":119154537,"uuid":"429145974","full_name":"vikramaditya91/iot-ingest-large-size","owner":"vikramaditya91","description":"Leverage MQTT and pre-signed URLs to upload large files with AWS IoT Core","archived":false,"fork":false,"pushed_at":"2021-12-21T23:40:42.000Z","size":40,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-03T07:41:38.039Z","etag":null,"topics":["aws","iot","python","s3","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vikramaditya91.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-11-17T17:49:54.000Z","updated_at":"2023-05-04T16:37:10.000Z","dependencies_parsed_at":"2023-07-10T01:00:59.560Z","dependency_job_id":null,"html_url":"https://github.com/vikramaditya91/iot-ingest-large-size","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/vikramaditya91/iot-ingest-large-size","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikramaditya91%2Fiot-ingest-large-size","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikramaditya91%2Fiot-ingest-large-size/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikramaditya91%2Fiot-ingest-large-size/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikramaditya91%2Fiot-ingest-large-size/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vikramaditya91","download_url":"https://codeload.github.com/vikramaditya91/iot-ingest-large-size/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikramaditya91%2Fiot-ingest-large-size/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32517129,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","iot","python","s3","terraform"],"created_at":"2024-08-01T21:01:29.917Z","updated_at":"2026-05-01T23:31:37.565Z","avatar_url":"https://github.com/vikramaditya91.png","language":"HCL","funding_links":[],"categories":["HCL"],"sub_categories":[],"readme":"# IoT Ingest Large-Sized Payloads\n\nSecurely ingest large sized payloads from IoT devices using AWS.\n\n## Description\n\nThis project is mainly based on this [AWS Blog](https://aws.amazon.com/blogs/iot/securely-ingesting-large-sized-payloads-from-iot-devices-to-the-aws-cloud/) which leverages the power of S3 pre-signed URLs, MQTT and IoT Core to upload large files.\nOnce the object is uploaded, the S3 Notification triggers a Lambda which sends an email to view the file.\n\n![alt text](https://d2908q01vomqb2.cloudfront.net/f6e1126cedebf23e1463aee73f9df08783640400/2021/08/20/AWS-IoT-Blog_V2.jpg)\n\n## Getting Started\n\n### Dependencies\n\n* An AWS Account\n  * Generate the [certificate](https://docs.aws.amazon.com/iot/latest/developerguide/device-certs-create.html) needed to authenticate the IoT device(s). The following are needed:\n    * Amazon Root CA certificate file (Amazon-root-CA-1.pem)\n    * Private key file (private.pem.key)\n    * A certificate for this thing (device.pem.crt)\n  * Obtain the device-data-endpoint. Looks similar to `account-specific-prefix.jobs.iot.aws-region.amazonaws.com`.\n  * Register your [email address](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-addresses-and-domains.html) for the SES to be able to send emails.\n* An IoT device (eg. a Raspberry Pi).\n* [Terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)\n\n### Terraform setup\n\n* Clone this repository on the laptop/desktop/server\n```\ngit clone https://github.com/vikramaditya91/iot-ingest-large-size.git\n```\n* Get the ARN of the certificate and set it in the `terraform/variables.tf`\n* Adjust other variables in `terraform/variables.tf` as required\n* Apply the terraform configuration:\n```\ncd terraform\nterraform apply\n```\n\n### Set-up on IoT device\n\n* Copy the certificates onto the IoT device\n* Clone this repository on the IoT device\n```\ngit clone https://github.com/vikramaditya91/iot-ingest-large-size.git\n```\n* Create a virtual environment of Python3+\n* Install the dependencies listed in pip_requirements.txt\n* Set the correct values in `send_large_file.py`\n  * paths to the certificates\n  * bucket-name as set in `variables.tf`\n  * end-point url as obtained above\n  * request url topic as set in `variables.tf`\n* Run the script `python send_large_file.py \u003cfull_path_to_file_to_be_uploaded\u003e`\n\n\n## Additional information\n\n* This project was originally meant for a motion detection system for when my cat Doudou wanted to be let in\n![Dudu cat](https://i.ibb.co/HtcgdWN/Whats-App-Image-2021-12-20-at-11-18-42-PM.jpg)\n* Motion was detected using the [Motion Project](https://motion-project.github.io/)\n\n\n\n## Authors\n\n[Vikramaditya Gaonkar](https://github.com/vikramaditya91)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvikramaditya91%2Fiot-ingest-large-size","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvikramaditya91%2Fiot-ingest-large-size","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvikramaditya91%2Fiot-ingest-large-size/lists"}