{"id":19813818,"url":"https://github.com/azer0s/tinygator","last_synced_at":"2025-05-01T10:31:01.537Z","repository":{"id":40901006,"uuid":"228016407","full_name":"Azer0s/TinyGator","owner":"Azer0s","description":"A tiny, Kafka based, data aggregator for storing JSON metrics in timescaledb","archived":false,"fork":false,"pushed_at":"2022-12-08T09:26:27.000Z","size":71,"stargazers_count":6,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-06T12:06:49.342Z","etag":null,"topics":["data-aggregation","kafka","postgres","timescaledb"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Azer0s.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-14T12:05:28.000Z","updated_at":"2023-01-31T16:45:10.000Z","dependencies_parsed_at":"2023-01-25T05:45:09.601Z","dependency_job_id":null,"html_url":"https://github.com/Azer0s/TinyGator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azer0s%2FTinyGator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azer0s%2FTinyGator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azer0s%2FTinyGator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azer0s%2FTinyGator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Azer0s","download_url":"https://codeload.github.com/Azer0s/TinyGator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251859777,"owners_count":21655617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-aggregation","kafka","postgres","timescaledb"],"created_at":"2024-11-12T09:37:24.572Z","updated_at":"2025-05-01T10:31:01.234Z","avatar_url":"https://github.com/Azer0s.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"assets/TinyGator.png\" alt=\"logo\" width=\"200\" align=\"left\"/\u003e\n\n## TinyGator\n![License](https://img.shields.io/github/license/Azer0s/TinyGator)\n[![Dockerhub Aggregator](https://img.shields.io/badge/dockerhub-Aggregator-blue)](https://hub.docker.com/r/azer0s/tinygator.aggregator)\n[![Dockerhub Collector](https://img.shields.io/badge/dockerhub-Collector-blue)](https://hub.docker.com/r/azer0s/tinygator.collector)\n\nA tiny, Kafka based, data aggregator for storing JSON metrics in timescaledb\n\n### Configure the aggregator\n\n* `KAFKA_BOOTSTRAP_SERVER` - The Kafka bootstrap server to connect to\n* `KAFKA_TOPIC` - The Kafka topic to push the data to\n* `METRIC_ENDPOINT` - The endpoint from which to pull the data\n* `METRIC_METHOD` - The HTTP method used when accessing the endpoint\n* `METRIC_INTERVAL` - The interval the endpoint is accessed in (in ms)\n* `METRIC_HEADERS` - The headers to use when accessing the endpoint\n\n#### Example:\n\n```bash\nKAFKA_BOOTSTRAP_SERVER=localhost:9092\nKAFKA_TOPIC=test\nMETRIC_ENDPOINT=\"http://localhost:15672/api/queues/%2F/Test?sort=message_stats.publish_details.rate\u0026sort_reverse=true\u0026columns=name,message_stats.publish_details.rate,message_stats.deliver_get_details.rate\"\nMETRIC_METHOD=GET\nMETRIC_INTERVAL=5000\nMETRIC_HEADERS={\"authorization\":\"Basic Z3Vlc3Q6Z3Vlc3Q=\"}\n```\n\n### Configure the collector\n\n* `KAFKA_BOOTSTRAP_SERVER` - The Kafka bootstrap server to connect to\n* `KAFKA_TOPIC` - The Kafka topic to pull the data from\n* `KAFKA_GROUP_ID` - The Kafka group id for the consumer\n* `JSON_POINTER_EXTRACTORS` - The JSON pointer configuration to extract data from the metrics\n* `POSTGRESQL_CONNECTION_STRING` - The PostgreSQL connection string (in ADO.NET form)\n* `POSTGRESQL_INSERT_STATEMENT` - The PostgreSQL insert statement\n* `ADD_TIMESTAMP` - The field to add the timestamp to (leave empty if your data already contains a timestamp)\n\n#### Example\n\n```bash\nKAFKA_BOOTSTRAP_SERVER=localhost:9092\nKAFKA_TOPIC=test\nKAFKA_GROUP_ID=test-group\nJSON_POINTER_EXTRACTORS={\"foo\": {\"pointer\": \"/name\", \"type\": \"string\"}}\nPOSTGRESQL_CONNECTION_STRING=\"Host=localhost\\;Username=postgres\\;Password=timescale\"\nPOSTGRESQL_INSERT_STATEMENT='INSERT INTO \"test\".\"default\" (\"timestamp\", \"value\") VALUES (@timestamp, @foo)'\nADD_TIMESTAMP=timestamp\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazer0s%2Ftinygator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazer0s%2Ftinygator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazer0s%2Ftinygator/lists"}