{"id":16842226,"url":"https://github.com/gunthercox/datahub","last_synced_at":"2025-07-25T08:04:12.139Z","repository":{"id":147610764,"uuid":"115721373","full_name":"gunthercox/DataHub","owner":"gunthercox","description":"A fast data feed designed for machine consumption","archived":false,"fork":false,"pushed_at":"2025-04-02T12:15:09.000Z","size":52,"stargazers_count":8,"open_issues_count":2,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-04T04:04:05.440Z","etag":null,"topics":["data-feed","iot","robotics"],"latest_commit_sha":null,"homepage":"https://hub.docker.com/r/gunthercox/datahub/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gunthercox.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-12-29T12:38:33.000Z","updated_at":"2025-04-02T12:15:12.000Z","dependencies_parsed_at":"2023-05-31T19:01:29.975Z","dependency_job_id":"47b405d1-c292-4a17-a4f0-59d1080c7410","html_url":"https://github.com/gunthercox/DataHub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gunthercox/DataHub","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gunthercox%2FDataHub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gunthercox%2FDataHub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gunthercox%2FDataHub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gunthercox%2FDataHub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gunthercox","download_url":"https://codeload.github.com/gunthercox/DataHub/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gunthercox%2FDataHub/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266977737,"owners_count":24015478,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-feed","iot","robotics"],"created_at":"2024-10-13T12:44:54.132Z","updated_at":"2025-07-25T08:04:12.079Z","avatar_url":"https://github.com/gunthercox.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DataHub\n\nDataHub is an experimental high-throughput data feed designed for machine\nconsumption.\n\n## Use cases\n\nI need a centralized data feed with a defined format so that distributed\napplications can consume and respond to events.\n\n1. Robotics: Multitudes of sensors and systems will gather and post data\n   to this data feed. Subscribing applications will watch the feed for\n   relevant events that require actions to be performed.\n2. Prediction: Event prediction can be performed by analyzing the data\n   stream for patterns.\n\n## Architecture / Tech Stack\n\n- A scalable application layer written in [Python](https://www.python.org).\n- Load balanced behind [Nginx](https://nginx.org).\n- Data stored using [Redis](https://redis.io).\n\nRedis instances use a single thread which is optimal for robotics\napplications running on Raspberry Pis. Although a single instance\nof redis has exceptional performance, redis can be scaled as needed\nas an application usage grows.\n\nThe API design is optimized for scaling behind a load balancer in architectures where many separate systems transmit information to a central endpoint. This design means that only a single directional connection (from the sensor systems to the API) is required, which is optimal when data is being transmitted from devices that aren't publicly-accessible on a network.\n\n## Data Format\n\nData should be sent to the API in the form of a POST request with the\nfollowing JSON content.\n\n- name: The name of the 'thing' that is reporting the data.\n        For example, a GPS sensor.\n- value: The sensor value being recorded.\n- expires: Optional date at which the record is no longer valid.\n           The application that sends the data determines when it expires.\n\n## Performance benchmarks\n\nFor 1000 requests sent using\n`tests/benchmark.py`\n\n```\nPost time: 2.61 seconds\nGet time:  1.60 seconds\nTotal execution time: 4.21 seconds\n```\n\n## Data Projections\n\nThe following projects the number requests per second that the system\nwill need to be able to handle based on a hypothetical robotics project.\n\n- Camera\n  - Up to 5 events per second\n- Audio\n  - Up to 20 events per second\n- Temperature sensor\n  - 1 sample per minute\n- Encoder readings\n  - Up to 100 events per second\n\nSo about 125 requests per second for incoming event data.\n\nLet's also estimate that there will be 50 subscribers reading\nthe incoming data at a rate of 1 request per second.\n\nThat gives us a grand total of 175 request per second that this\nsystem needs to be able to handle. This is definitely doable.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgunthercox%2Fdatahub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgunthercox%2Fdatahub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgunthercox%2Fdatahub/lists"}