{"id":15396198,"url":"https://github.com/jamesseanwright/iot-events-api","last_synced_at":"2025-04-16T00:18:08.193Z","repository":{"id":151706501,"uuid":"361182710","full_name":"jamesseanwright/iot-events-api","owner":"jamesseanwright","description":"A REST API for storing and retrieving IoT device events, built with Terraform, AWS Lambda, and MongoDB Atlas","archived":false,"fork":false,"pushed_at":"2021-07-18T15:57:51.000Z","size":245,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-16T00:17:58.707Z","etag":null,"topics":["aws","docker","lambda","mongodb","mongodb-atlas","terraform"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jamesseanwright.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-24T14:26:30.000Z","updated_at":"2024-10-03T21:49:33.000Z","dependencies_parsed_at":"2023-07-15T23:31:02.668Z","dependency_job_id":null,"html_url":"https://github.com/jamesseanwright/iot-events-api","commit_stats":{"total_commits":98,"total_committers":2,"mean_commits":49.0,"dds":"0.061224489795918324","last_synced_commit":"04562f85d6843ff28ee6da0ea17fe37ea4bbd3cf"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesseanwright%2Fiot-events-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesseanwright%2Fiot-events-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesseanwright%2Fiot-events-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jamesseanwright%2Fiot-events-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jamesseanwright","download_url":"https://codeload.github.com/jamesseanwright/iot-events-api/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249173099,"owners_count":21224485,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","docker","lambda","mongodb","mongodb-atlas","terraform"],"created_at":"2024-10-01T15:32:05.159Z","updated_at":"2025-04-16T00:18:08.173Z","avatar_url":"https://github.com/jamesseanwright.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# IoT Events API\n\nA REST API for storing and retrieving IoT device events, built with Terraform, AWS Lambda, and MongoDB Atlas.\n\n## Running Locally\n\nEnsure [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) are present on your system. You can then run `docker-compose up`. **Note that** you'll need to rebuild with `docker-compose build` when making changes to the function code.\n\n### Example Requests\n\n#### Adding a New Event\n\n```sh\n$ curl -XPOST \"http://localhost:8080/events\" -H \"Content-Type: application/json\" -d '{\"date\":\"'$(date -u +%FT%TZ)'\",\"deviceID\":\"8f188304-e7b3-4a16-a243-b9470468478a\",\"eventType\":\"temp_celcius\",\"value\":3}'\n```\n\n#### Retrieving events for a given device ID by event type and date\n\n```sh\n$ curl -XGET \"http://localhost:8080/events?deviceID=8f188304-e7b3-4a16-a243-b9470468478a\u0026eventType=temp_celcius\u0026date=$(date -u +%F)\" -H \"Content-Type: application/json\"\n```\n\n## Deploying to AWS and MongoDB Atlas\n\nThe infrastructure of this project is managed with [Terraform](https://www.terraform.io/); make sure you have this installed before proceeding.\n\nFirstly, you'll need to ensure that you have [valid AWS credentials configured on your machine](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication); I recommend using [`aws configure`](https://docs.aws.amazon.com/cli/latest/reference/configure/) to write your key ID and secret key value to the AWS CLI's shared credentials file.\n\nNext, [create an Atlas API key](https://docs.atlas.mongodb.com/configure-api-access/#programmatic-api-keys) with the _Organization Project Creator_ scope, which must be present on your system as environment variables:\n\n```sh\n$ export MONGODB_ATLAS_PUBLIC_KEY=\u003cAtlas public key\u003e MONGODB_ATLAS_PRIVATE_KEY=\u003cAtlas private key\u003e\n```\n\nThere are a handful of root module variables to which values must be assigned; this can be achieved by renaming `terraform.tfvars.example` to `terraform.tfvars` (i.e. `mv terraform.tfvars.example terraform.tfvars`) and populating it accordingly:\n\n| Variable name  | Description                                                                                                                                                     |\n| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `region`       | The AWS region to which the lambdas and AWS-backed Atlas cluster are deployed e.g. `eu-west-1`                                                                  |\n| `atlas_org_id` | The ID of the organisation under which the Atlas project will be created                                                                                        |\n| `rest_api_key` | The key that consumers will need to provide when calling our REST API i.e. this value must be present in the `x-api-key` HTTP header when requesting a resource |\n\nThe final prerequisite is to authenticate Docker against AWS ECR, to which our Lambda container images will be pushed:\n\n```sh\n$ aws ecr get-login-password --region \u003cAWS region\u003e | docker login --username AWS --password-stdin \u003cAWS account ID\u003e.dkr.ecr.\u003cAWS region\u003e.amazonaws.com\n```\n\nYou can now deploy the service with `terraform apply`. Upon successful deployment, the full API URL will be displayed as an output. **Note that** it can take ~7-10 minutes to create the Atlas cluster.\n\n### Example Requests\n\n#### Adding a New Event\n\n```sh\n$ curl -XPOST \"\u003cAPI URL in Terraform outputs\u003e\" -H \"Content-Type: application/json\" -H \"x-api-key: \u003cvalue provided to rest_api_key Terraform variable\u003e\" -d '{\"date\":\"'$(date -u +%FT%TZ)'\",\"deviceID\":\"8f188304-e7b3-4a16-a243-b9470468478a\",\"eventType\":\"temp_celcius\",\"value\":3}'\n```\n\n#### Retrieving events for a given device ID by event type and date\n\n```sh\n$ curl -XGET \"\u003cAPI URL in Terraform outputs\u003e?deviceID=8f188304-e7b3-4a16-a243-b9470468478a\u0026eventType=temp_celcius\u0026date=$(date -u +%F)\" -H \"Content-Type: application/json\" -H \"x-api-key: \u003cvalue provided to rest_api_key Terraform variable\u003e\"\n```\n\n## Architecture\n\n![Architecture diagram](images/architecture.png)\n\n## Data Schema\n\nThe schema of our MongoDB events collection follows the [bucket pattern](https://www.mongodb.com/blog/post/building-with-patterns-the-bucket-pattern); by placing individual events within an array field held by a document representing a particular device ID, date range, and event type combination, we can drastically reduce the number of documents the database engine has to scan on retrieval and shrink any indexes against the collection.\n\n```ts\n{\n  _id: ObjectId;\n  date: Date; // the date for the given day e.g. 2021-05-05T00:00:00.000+0000\n  deviceID: string;\n  eventType: string;\n  events: {\n    date: Date; // specific event date e.g. 2021-05-05T17:19:35.000+0000\n    value: any;\n  }\n  [];\n}\n```\n\n## Next Steps\n\n- The Docker images for the Lambda functions are only pushed to ECR when the infrastructure is first provisioned, given the use of the `local-exec` provisioner within the `aws_ecr_repository.lambda_repo` resource. Perhaps it's possible to push the containers when they're updated with the [`kreuzwerker/docker` provider](https://registry.terraform.io/providers/kreuzwerker/docker/latest) and version-tagged images, although we'd still have to manage the latter concern outside of Terraform\n\n- Given our data schema is built upon bucketing, we should create a unique index for the events collection against the date, device ID, and event type fields, to ensure that duplicate buckets can't be created. Unfortunately, [the `mongodbatlas` provider doesn't include a resource for creating indexes](https://github.com/mongodb/terraform-provider-mongodbatlas/issues/308), so we'd have to call the Atlas API directly\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesseanwright%2Fiot-events-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjamesseanwright%2Fiot-events-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjamesseanwright%2Fiot-events-api/lists"}