{"id":37649008,"url":"https://github.com/manaakiwhenua/pygeoapi-redis-manager-plugin","last_synced_at":"2026-01-16T11:29:40.735Z","repository":{"id":81753590,"uuid":"276759321","full_name":"manaakiwhenua/pygeoapi-redis-manager-plugin","owner":"manaakiwhenua","description":"Pygeoapi plugin for a processing manager using Redis and RQ","archived":false,"fork":false,"pushed_at":"2020-10-30T00:39:21.000Z","size":98,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-03-14T15:56:50.804Z","etag":null,"topics":["ogc-api","plugin","pygeoapi","redis","rq"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manaakiwhenua.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-07-02T22:48:37.000Z","updated_at":"2021-09-30T08:15:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"c912b928-e45e-4ff8-94b3-4bc55bacd3de","html_url":"https://github.com/manaakiwhenua/pygeoapi-redis-manager-plugin","commit_stats":{"total_commits":10,"total_committers":1,"mean_commits":10.0,"dds":0.0,"last_synced_commit":"85b42cf4855b55643a112c89bc8aabf9fb00c685"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/manaakiwhenua/pygeoapi-redis-manager-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manaakiwhenua%2Fpygeoapi-redis-manager-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manaakiwhenua%2Fpygeoapi-redis-manager-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manaakiwhenua%2Fpygeoapi-redis-manager-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manaakiwhenua%2Fpygeoapi-redis-manager-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manaakiwhenua","download_url":"https://codeload.github.com/manaakiwhenua/pygeoapi-redis-manager-plugin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manaakiwhenua%2Fpygeoapi-redis-manager-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ogc-api","plugin","pygeoapi","redis","rq"],"created_at":"2026-01-16T11:29:40.677Z","updated_at":"2026-01-16T11:29:40.727Z","avatar_url":"https://github.com/manaakiwhenua.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![manaakiwhenua-standards](https://github.com/manaakiwhenua/pygeoapi-redis-manager-plugin/workflows/manaakiwhenua-standards/badge.svg)](https://github.com/manaakiwhenua/manaakiwhenua-standards)\n\n**In-development; do not use for production applications.**\n\nContributions welcome.\n\n# Pygeoapi Redis manager plugin\n\nA [pygeoapi](https://github.com/geopython/pygeoapi) processing manager plugin. This plugin is a drop-in replacement plugin for the default pygeoapi processing manager. The default manager (as of pygeoapi v0.8)\nuses the file system as an output cache, and is only capable of doing jobs asynchronously by spawning new threads. This plugin introduces the use of Redis as a separate microservice to store job state, and also the use of RQ (Redis Queue) to manager worker processes that do the actual computation.\n\nIt is capable of supporting any processing task with input that can be de/serialised to/from JSON. This means that file uploads and other complex inputs are not supported.\n\n## Sample usage\n\nHere is a sample Docker Compose YAML demonstrating the use of pygeoapi, this plugin, with separate Redis (broker) and RQ (worker) microservices.\n\n```yaml\nversion: \"3\"\nservices:\n\n  pygeoapi:\n    build: .\n    image: pygeoapi\n    container_name: pygeoapi\n    ports:\n      - \"5000:80\"\n    environment:\n      - TZ=Pacific/Auckland\n      - LANG=en_NZ.UTF-8\n    volumes:\n      - ./local.config.yml:/pygeoapi/local.config.yml\n      - ./pygeoapi:/pygeoapi/pygeoapi\n      - ./tests:/pygeoapi/tests\n      - /etc/localtime:/etc/localtime:ro\n    depends_on:\n      - broker\n      - worker\n    networks:\n      frontend:\n        ipv4_address: 172.21.0.24\n\n  broker:\n    image: redis:6.0.3\n    container_name: broker\n    command:\n      - redis-server\n      - /etc/redis.conf\n    environment:\n      - REDISCLI_AUTH=some-long-random-password\n    networks:\n      frontend:\n        ipv4_address: 172.21.0.3\n    ports:\n      - '6379'\n    volumes:\n      - \"./redis-data:/data\"\n      - \"./redis.conf:/etc/redis.conf\"\n\n  worker:\n    image: pygeoapi\n    entrypoint: python3\n    restart: always\n    command: /usr/local/lib/python3.8/dist-packages/pygeoapi_redis_manager_plugin/redis_worker_.py -q hello-world -q wkt-reprojector\n    volumes:\n      - ./pygeoapi:/pygeoapi/pygeoapi\n    networks:\n      - frontend\n    environment:\n      - REDIS=172.21.0.3\n      - PORT=6379\n    depends_on:\n      - broker\n\nvolumes:\n  redis-data:\n\nnetworks:\n  frontend:\n    ipam:\n      config:\n        - subnet: 172.21.0.0/24\n```\n\nThis depends on your pygeoapi instance being built with an additional dependency:\n\n`requirements-processes.txt`\n```txt\ngit+https://github.com/manaakiwhenua/pygeoapi-redis-manager-plugin.git@master\n```\n\nAdditionally, add this to `pygeoapi/plugin.py` (pending a better way to integrate plugins into pygeoapi):\n\n```python\n'process_manager': {\n    'TinyDB': 'pygeoapi.process.manager.tinydb_.TinyDBManager',\n    'Redis': 'pygeoapi_redis_manager_plugin.RedisManager' # \u003c---\n}\n```\n\nFinally, set your pygeoapi YAML configuration to include this under the `server` block:\n\n```yaml\nmanager:\n  name: Redis\n  connection: 172.21.0.3\n```\n\nIf you run `docker-compose up --scale worker=4`, then all the defined services will be brought up in orchestra, with four worker processes running. This means that as many as four processing jobs can be executed at once, with any other jobs after that being put onto the worker queue and executed as workers complete existing jobs.\n\nThe `worker` `command` `/pygeoapi/pygeoapi/process/manager/redis_worker_.py -q hello-world -q wkt-reprojector` includes two job queues, `hello-world` and `wkt-reprojector`. The first is the built-in sample process for pygeoapi; the second is defined as a [plugin](https://github.com/manaakiwhenua/wkt-reprojector-plugin) (incude in your `requirements-processes.txt` as `git+https://github.com/manaakiwhenua/wkt-reprojector-plugin.git@master` or otherwise follow instructions in that repository). Workers need to be told which queues to work on, and do not need to work on all of them.\n\nWorkers depend on two environment variables, `REDIS` and `PORT` which specify the hostname or IP address of the redis instance they can use, and the appropriate port.\n\n### Redis configuration\n\nIn addition, you will need to set up the `redis.conf` (Redis configuration file) that matches these binds (if appropriate for your networking pattern). You can get a default Redis configuration for your version of Redis [here](https://redis.io/topics/config)\n\n\u003e By default, if no \"bind\" configuration directive is specified, Redis listens for connections from all the network interfaces available on the server. It is possible to listen to just one or multiple selected interfaces using the \"bind\" configuration directive, followed by one or more IP addresses.\n\nAdd this line (or replace the line beginning with `bind`) to your `redis.conf`:\n\n```\nbind 172.21.0.3\n```\n\nThis matches the block in the `docker-compose.yml`:\n\n```yaml\nservices:\n  broker:\n    networks:\n      frontend:\n        ipv4_address: 172.21.0.3 # \u003c----\n```\n\n### Dashboard\n\nIf you define an additional service:\n\n```yaml\ndashboard:\n  build: ./dashboard\n  image: dashboard\n  container_name: dashboard\n  ports:\n    - 9181:9181\n  command: rq-dashboard -H 172.21.0.3\n  networks:\n    frontend:\n      ipv4_address: 172.21.0.16\n  depends_on:\n    - broker\n    - worker\n```\n\nWhere there is a file `./dashboard/Dockerfile`:\n\n```Dockerfile\nFROM python:3.8.0-alpine\n\nRUN pip install rq-dashboard\n\nEXPOSE 9181\n\nCMD [\"rq-dashboard\"]\n```\n\nYou can use the RQ Dashboard to inspect jobs and workers. This is not a production application, but it is useful and fun in development.\n\nPygeoapi's process state is independent of the RQ state, although both are managed in the same Redis database (the `broker` service above).\n\n### Volumes\n\nUsing the redis-data volume means that the database created and used by the `broker` service will be available on the host OS, and can therefore be persisted, backed-up, audited and removed.\n\n## Building for release\n\nRequires wheel.\n\n`python setup.py sdist bdist_wheel`\n\nThis can be included in a requirements.txt as: `git+https://github.com/manaakiwhenua/pygeoapi-redis-manager-plugin.git@master`\n\n`master` branch is for release, changes should be proposed in a separate branch and a PR submitted for merging into master, including rebuilding the source distributions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanaakiwhenua%2Fpygeoapi-redis-manager-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanaakiwhenua%2Fpygeoapi-redis-manager-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanaakiwhenua%2Fpygeoapi-redis-manager-plugin/lists"}