{"id":19215853,"url":"https://github.com/vanvalenlab/kiosk-redis-consumer","last_synced_at":"2025-09-11T02:13:51.304Z","repository":{"id":33872467,"uuid":"146685580","full_name":"vanvalenlab/kiosk-redis-consumer","owner":"vanvalenlab","description":"Consume the Kiosk's Redis Events","archived":false,"fork":false,"pushed_at":"2023-07-12T23:32:47.000Z","size":531,"stargazers_count":0,"open_issues_count":7,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-22T15:47:15.978Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vanvalenlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-30T02:35:22.000Z","updated_at":"2021-12-15T19:00:43.000Z","dependencies_parsed_at":"2025-01-04T18:28:57.761Z","dependency_job_id":"0b2d5d14-225b-434e-8d45-d1d247d7e025","html_url":"https://github.com/vanvalenlab/kiosk-redis-consumer","commit_stats":null,"previous_names":[],"tags_count":37,"template":false,"template_full_name":null,"purl":"pkg:github/vanvalenlab/kiosk-redis-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fkiosk-redis-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fkiosk-redis-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fkiosk-redis-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fkiosk-redis-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vanvalenlab","download_url":"https://codeload.github.com/vanvalenlab/kiosk-redis-consumer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vanvalenlab%2Fkiosk-redis-consumer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274564873,"owners_count":25308662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-11T02:00:13.660Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T14:14:59.293Z","updated_at":"2025-09-11T02:13:51.285Z","avatar_url":"https://github.com/vanvalenlab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ![DeepCell Kiosk Banner](https://raw.githubusercontent.com/vanvalenlab/kiosk-console/master/docs/images/DeepCell_Kiosk_Banner.png)\n\n[![Build Status](https://github.com/vanvalenlab/kiosk-redis-consumer/workflows/build/badge.svg)](https://github.com/vanvalenlab/kiosk-redis-consumer/actions)\n[![Coverage Status](https://coveralls.io/repos/github/vanvalenlab/kiosk-redis-consumer/badge.svg?branch=master)](https://coveralls.io/github/vanvalenlab/kiosk-redis-consumer?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/kiosk-redis-consumer/badge/?version=master)](https://deepcell-kiosk.readthedocs.io/projects/kiosk-redis-consumer/)\n[![Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](/LICENSE)\n\nThe `kiosk-redis-consumer` reads events in Redis, downloads image data from the cloud, and sends the data to TensorFlow Serving via gRPC. The prediction is post-processed, zipped, and uploaded to the cloud.\n\nThis repository is part of the [DeepCell Kiosk](https://github.com/vanvalenlab/kiosk-console). More information about the Kiosk project is available through [Read the Docs](https://deepcell-kiosk.readthedocs.io/en/master) and our [FAQ](http://www.deepcell.org/faq) page.\n\n## Custom Consumers\n\nCustom consumers can be used to implement custom model pipelines. This documentation is a continuation of a [tutorial](https://deepcell-kiosk.readthedocs.io/en/master/CUSTOM-JOB.html) on building a custom job pipeline.\n\nConsumers consume Redis events. Each type of Redis event is put into a queue (e.g. `predict`, `track`), and each queue has a specific consumer type will pop items off the queue.\nConsumers call the `_consume` method to consume each item it finds in the queue.\nThis method must be implemented for every consumer.\n\nThe quickest way to get a custom consumer up and running is to:\n\n1. Add a new file for the consumer: `redis_consumer/consumers/my_new_consumer.py`\n2. Create a new class, inheriting from `TensorFlowServingConsumer` ([docs](https://deepcell-kiosk.readthedocs.io/projects/kiosk-redis-consumer/en/master/redis_consumer.consumers.html)), which uses the `preprocess`, `predict`, and `postprocess` methods to easily process data with the model.\n3. Implement the `_consume` method, which should download the data, run inference on the data, save and upload the results, and finish the job by updating the Redis fields.\n4. Import the new consumer in \u003ctt\u003e\u003ca href=\"https://github.com/vanvalenlab/kiosk-redis-consumer/blob/master/redis_consumer/consumers/__init__.py\"\u003eredis_consumer/consumers/\\_\\_init\\_\\_.py\u003c/a\u003e\u003c/tt\u003e and add it to the `CONSUMERS` dictionary with a correponding queue type (`queue_name`). The script \u003ctt\u003e\u003ca href=\"https://github.com/vanvalenlab/kiosk-redis-consumer/blob/master/consume-redis-events.py\"\u003econsume-redis-events.py\u003c/a\u003e\u003c/tt\u003e will load the consumer class based on the `CONSUMER_TYPE`.\n\nSee below for a basic implementation of `_consume()` making use of the methods inherited from `ImageFileConsumer`:\n\n```python\ndef _consume(self, redis_hash):\n    # get all redis data for the given hash\n    hvals = self.redis.hgetall(redis_hash)\n\n    # only work on unfinished jobs\n    if hvals.get('status') in self.finished_statuses:\n        self.logger.warning('Found completed hash `%s` with status %s.',\n                            redis_hash, hvals.get('status'))\n        return hvals.get('status')\n\n    # Load input image\n    fname = hvals.get('input_file_name')\n    image = self.download_image(fname)\n\n    # the model can be passed in as an environment variable,\n    # and parsed in settings.py.\n    model = 'NuclearSegmentation:1'\n\n    # Use a custom Application from deepcell.applications\n    app = self.get_grpc_app(model, deepcell.applications.NuclearSegmentation)\n\n    # Run the predictions on the image\n    results = app.predict(image)\n\n    # save the results as an image file and upload it to the bucket\n    save_name = hvals.get('original_name', fname)\n    dest, output_url = self.save_output(image, redis_hash, save_name)\n\n    # save the results to the redis hash\n    self.update_key(redis_hash, {\n        'status': self.final_status,\n        'output_url': output_url,\n        'upload_time': timeit.default_timer() - _,\n        'output_file_name': dest,\n        'total_jobs': 1,\n        'total_time': timeit.default_timer() - start,\n        'finished_at': self.get_current_timestamp()\n    })\n\n    # return the final status\n    return self.final_status\n```\n\nFor guidance on how to complete the deployment of a custom consumer, please return to [Tutorial: Custom Job](https://deepcell-kiosk.readthedocs.io/en/master/CUSTOM-JOB.html).\n\n## Configuration\n\nThe consumer is configured using environment variables. Please find a table of all environment variables and their descriptions below.\n\n| Name | Description | Default Value |\n| :--- | :--- | :--- |\n| `QUEUE` | **REQUIRED**: The Redis job queue to check for items to consume. | `\"predict\"` |\n| `CONSUMER_TYPE` | **REQUIRED**: The type of consumer to run, used in `consume-redis-events.py`. | `\"image\"` |\n| `STORAGE_BUCKET` | **REQUIRED**: The name of the storage bucket used to download and upload files. | `\"s3://default-bucket\"` |\n| `INTERVAL` | How frequently the consumer checks the Redis queue for items, in seconds. | `5` |\n| `REDIS_HOST` | The IP address or hostname of Redis. | `\"redis-master\"` |\n| `REDIS_PORT` | The port used to connect to Redis. | `6379` |\n| `REDIS_TIMEOUT` | Timeout for each Redis request, in seconds. | `3` |\n| `EMPTY_QUEUE_TIMEOUT` | Time to wait after finding an empty queue, in seconds. | `5` |\n| `DO_NOTHING_TIMEOUT` | Time to wait after finding an item that requires no work, in seconds. | `0.5` |\n| `STORAGE_MAX_BACKOFF` | Maximum time to wait before retrying a Storage request | `60` |\n| `EXPIRE_TIME` | Expire Redis items this many seconds after completion. | `3600` |\n| `METADATA_EXPIRE_TIME` | Expire cached model metadata after this many seconds. | `30` |\n| `TF_HOST` | The IP address or hostname of TensorFlow Serving. | `\"tf-serving\"` |\n| `TF_PORT` | The port used to connect to TensorFlow Serving. | `8500` |\n| `GRPC_TIMEOUT` | Timeout for gRPC API requests, in seconds. | `30` |\n| `GRPC_BACKOFF` | Time to wait before retrying a gRPC API request. | `3` |\n| `MAX_RETRY` | Maximum number of retries for a failed TensorFlow Serving request. | `5` |\n\n## Contribute\n\nWe welcome contributions to the [kiosk-console](https://github.com/vanvalenlab/kiosk-console) and its associated projects. If you are interested, please refer to our [Developer Documentation](https://deepcell-kiosk.readthedocs.io/en/master/DEVELOPER.html), [Code of Conduct](https://github.com/vanvalenlab/kiosk-console/blob/master/CODE_OF_CONDUCT.md) and [Contributing Guidelines](https://github.com/vanvalenlab/kiosk-console/blob/master/CONTRIBUTING.md).\n\n## License\n\nThis software is license under a modified Apache-2.0 license. See [LICENSE](/LICENSE) for full  details.\n\n## Copyright\n\nCopyright © 2018-2022 [The Van Valen Lab](http://www.vanvalen.caltech.edu/) at the California Institute of Technology (Caltech), with support from the Paul Allen Family Foundation, Google, \u0026 National Institutes of Health (NIH) under Grant U24CA224309-01.\nAll rights reserved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvanvalenlab%2Fkiosk-redis-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvanvalenlab%2Fkiosk-redis-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvanvalenlab%2Fkiosk-redis-consumer/lists"}