{"id":45164839,"url":"https://github.com/alexmilowski/redis-aqi","last_synced_at":"2026-02-20T06:31:46.838Z","repository":{"id":73570918,"uuid":"292129665","full_name":"alexmilowski/redis-aqi","owner":"alexmilowski","description":"A demonstration of the geospatial features of Redis to interpolate an AQI (Air Quality Index) from sensor data.","archived":false,"fork":false,"pushed_at":"2020-10-22T19:50:38.000Z","size":5286,"stargazers_count":4,"open_issues_count":1,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-14T07:19:15.686Z","etag":null,"topics":["air-quality","geospatial","python3","redis","sensor-data"],"latest_commit_sha":null,"homepage":"https://alexmilowski.github.io/redis-aqi/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexmilowski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-09-01T23:28:08.000Z","updated_at":"2022-06-23T08:26:02.000Z","dependencies_parsed_at":null,"dependency_job_id":"2d8460d8-7bf2-48b2-a93d-75fc6ff348a3","html_url":"https://github.com/alexmilowski/redis-aqi","commit_stats":{"total_commits":71,"total_committers":1,"mean_commits":71.0,"dds":0.0,"last_synced_commit":"2b5128ea159708129d4093796a022b8642c38ff1"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alexmilowski/redis-aqi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmilowski%2Fredis-aqi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmilowski%2Fredis-aqi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmilowski%2Fredis-aqi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmilowski%2Fredis-aqi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexmilowski","download_url":"https://codeload.github.com/alexmilowski/redis-aqi/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexmilowski%2Fredis-aqi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29643280,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-20T05:21:04.652Z","status":"ssl_error","status_checked_at":"2026-02-20T05:21:04.238Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["air-quality","geospatial","python3","redis","sensor-data"],"created_at":"2026-02-20T06:31:46.689Z","updated_at":"2026-02-20T06:31:46.833Z","avatar_url":"https://github.com/alexmilowski.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# redis-aqi\n\nThis project provides a demonstration of the use of the geospatial features of Redis\nfor the purpose store air quality sensor readings and provides a sample\nWeb application that can AQI (Air Quality Index) measurements over a map.\n\nThe raw data is collected from air quality sensors (see [Data sources](#data-sources))\nand is transformed into [AQI (Air Quality Index)](https://www.airnow.gov/aqi/aqi-basics/)\nmeasurements and stored as in geospatial partitions.\n\nThe demonstration application is a simple Flask-based application\nthat creates a map-based interactive experience with the date/time and geospatial\npartitions of the data which displays an interpolated (estimated) surface\nof AQI measurements (see [Interpolation](#interpolation)).\n\nThe following is a quick overview and instructions for running the data\ncollection, ingest, and application. A complete overview is available on the [website](https://alexmilowski.github.io/redis-aqi/).\n\n# Data sources\n\n[PurpleAir sells](https://www.purpleair.com) air quality sensors that measure\nparticulate matter in the air and upload that data to a data repository. They\nprovide access to this [aggregated data via an API](https://docs.google.com/document/d/15ijz94dXJ-YAZLi9iZ_RaBwrZ4KtYeCy08goGBwnbCU/edit#heading=h.2tzq9j55gsj6).\n\nAt its simplest, the API returns a list of sensor readings for a given\nbounding box defined by a pair of coordinate parameters defining the north west\nand south east corners of the bounds. The resulting data contains the current\nsensor data with rolling averages for the particulate matter readings.\n\nPurpleAir provides documentation for how to turn the PM readings into an [AQI (Air Quality Index)](https://www.airnow.gov/aqi/aqi-basics/)\nmeasure.\n\n# Setup\n\nYou can create a python environment with all the required packages by:\n\n```\npip install -r requirements.txt\n```\n\n# Collecting Data\n\nThe python program [collect.py](collect.py) provides a simple command line\ninterface to data collection that can poll at regular intervals and\ncollect the data from the API. This data is aggregated by the program and can\nbe stored in a variety of ways (e.g., in an S3-compatible object storage).\n\n## Running data collection\n\nThis program can be run as:\n\n```\n# collect for the bay area every 5 minutes and partition by 30 minutes\npython collect.py --bounding-box 38.41646632263371,-124.02669995117195,36.98663820370443,-120.12930004882817  --interval 300 --partition 30 --s3-bucket yourbuckethere\n```\n\nsome common options are:\n\n * --bounding-box nwlat,nwlon,selat,selon\n\n   The bounding box for collection; floating poing numbers representing the north west and south east corners of the quadrangle\n * --internal seconds\n\n   The collection interval in seconds\n * --partition minutes\n\n   The elapsed time in seconds at which to partition the data for storage\n * --prefix value\n\n     The data file prefix\n * --s3-endpoint url\n\n   A endpoint for the S3 protocol (e.g., https://storage.googleapis.com or [AWS endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html))\n * --s3-bucket name\n\n   The S3 bucket name for storage\n * --s3-key aws_access_key_id\n\n   The AWS access key\n\n * --s3-secret aws_secret_access_key\n\n   The AWS secret access key\n\nAny boto3 authentication method can be used (e.g., the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables) instead of the command-line parameters.\n\n## Where data is stored\n\nThe collect.py program retrieves data from the API at the interval you\nspecify. It will aggregate the collected data until the partition time\nlimit has been reached and then store the tabular data as a JSON artifact. By\ndefault, the data is sent to stdout in [JSON Text Sequences](https://tools.ietf.org/html/rfc7464).\n\nYou can store data in files by specifying the --dir parameter with a directory name.\n\nYou can store data in an S3 bucket by specifying the --s3-bucket parameter with\na bucket name.\n\nIn each case, the file name generated is the prefix appended with the ISO 8601\ndate and time format and suffixed with .json extension. For example, `data-2020-09-02T14:30:00.json`\nis the data for the time partition starting at 14:30:00 on 2020-09-02 and\nextending through the end of duration (i.e., 30 minutes till 15:00:00).\n\n\n# Running the application\n\nThe application is an interactive map that allows you to interact with the\nAQI interpolation. You can select various time periods or run animations of\nthe AQI interpolations over a map.\n\nThe overall architecture is:\n\n * the data collection runs continuously storing the collected data into\n   object storage (S3) in date/time partitions.\n\n * an ingestion process reads these partitions from object storage\n   and store them into geospatial partitions in [Redis](https://redis.io) (i.e., via [GEOADD](https://redis.io/commands/geoadd)). These partitions are stored as separate keys in Redis to facilitate scale-out.\n\n * a Flask application provides an API for interacting with the data\n   partitions, querying based on geospatial parameters, and running\n   interpolations. In addition, it provides an interactive map-based\n   interface to the data and interpolations.\n\n## Quick start\n\nAssuming you have your data collection running and likely storing data into\nobject storage (although, files will work), you can run everything locally\nas a quick test.\n\n 1. If you haven't already done so, setup a python environment and install the requirements:\n\n    ```\n    pip install -r requirements.txt\n    ```\n\n 1. Start redis:\n\n    ```\n    docker run -it --rm -p 6379:6379 redis\n    ```\n\n 1. Ingest some data.\n\n    * if you have local files:\n\n      ```\n      python ingest.py --confirm --precision 0 --index 1 --type data file1.json file2.json ...\n      ```\n    * If you have data in object storage:\n\n      ```\n      python ingest.py --confirm --precision 0 --index 1 --type at --bucket-url https://storage.googleapis.com/yourbuckethere/data- 2020-09-10T00:00:00,2020-09-10T23:30:00\n      ```\n\n      For the above, the assumption is you've made your bucket of collected data\n      publicly accessible. The date range specified will enumerate through the partitions which defaults to 30 minutes. You can change the partition size\n      via the --partition parameter but this parameter must match the partitions\n      you used for collection.\n\n    The ingest program has a number of options for controlling what is ingested and\n    how it finds the data\n\n 1. Run the web application:\n\n    ```\n    python app.py\n    ```\n\n    You can connect to a Redis database with the following parameters:\n\n     * --host ip\n\n       The Redis host address\n     * --port nnn\n\n       The Redis port\n\n     * --password password\n\n       The Redis password\n\n    The data is stored in Redis by ISO 8601 dateTime labeled partitions. You can provide alternate key prefix and partition period information by:\n\n     * --key-prefix prefix\n     * --partition nnn\n\n    Note: The partition time is in minutes.\n\n    Alternatively, all of the above settings can be set via environment variables\n    or in the Flask configuration file (via the --config option).\n\n    The keys are:\n\n     * REDIS_HOST\n     * REDIS_PORT\n     * REDIS_PASSWORD\n     * KEY_PREFIX\n     * PARTITION\n\n 1. Visit http://localhost:5000/\n\n# Interpolation\n\nInterpolation of AQI values relies on having an atmospheric model for\nthe distribution of particulate matter that takes into account weather,\nwind conditions, elevation, etc. Absent such a model, standard\ninterpolation methods such as [linear interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) can be used as a gross estimation of the AQI\nover a geospatial area.\n\nThe following example (from the Bay Area 2028-08-28) uses the krige exponential method of interpolation:\n\n![Example interpolation of AQI from 2020-08-28](example-2020-08-28T13%3a30%3a00PT30M.png)\n\nYou can use a variety of method to interpolate a grid of AQI values from\nthe observed values. The python program [interpolate.py](interpolate.py)\nprovides an implementation basic linear, cubic, nearest, and krige-based\nmethods of interpolation as a library as well as a program.\n\nYou can try the interpolation on collected data by:\n\n```\npython interpolate.py url [url ...]\n```\n\nThe options are:\n\n   * --verbose\n\n     enable verbose output\n   * --size nn\n\n     The grid mesh size (integer)\n   * --resolution nn.nnn\n\n     The grid resolution (float)\n   * --index n\n\n     The pm measurement to use - a value from 0 to 6\n   * --method linear|cubic|nearest|krige-linear|krige-power|krige-gaussian|krige-spherical|krige-exponential|krige-hole-effect\n   * --bounding-box' nwlat,nwlon,selat,selon\n\n     The bounding box (quadrangle) for the interpolation\n\nNote: You should only specify --size or --resolution but not both.\n\nThe library provides a function called `aqiFromPM` for calculating the AQI\nfrom the PM value.\n\nThere is also a `AQIInterpolator` class that can be used directly and\nprovides the same functionality as the command-line program.\n\n**Note:** The image was generated via the Web application. See the configuration\nof Redis and the Web application for how to produce your own map-based interpolations.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmilowski%2Fredis-aqi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexmilowski%2Fredis-aqi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexmilowski%2Fredis-aqi/lists"}