{"id":25633971,"url":"https://github.com/vostok/hercules","last_synced_at":"2025-10-03T19:50:19.724Z","repository":{"id":41207461,"uuid":"160483178","full_name":"vostok/hercules","owner":"vostok","description":"Vostok Hercules is an open-source distributed system based on Apache Kafka and used for reliable delivery of telemetry data from microservices to storage and analytics engines.","archived":false,"fork":false,"pushed_at":"2023-06-20T16:36:19.000Z","size":5676,"stargazers_count":43,"open_issues_count":4,"forks_count":9,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-28T08:01:37.446Z","etag":null,"topics":["hercules","java","kafka","telemetry","transport","vostok"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vostok.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-05T08:10:06.000Z","updated_at":"2024-12-21T14:50:38.000Z","dependencies_parsed_at":"2023-01-25T21:30:20.800Z","dependency_job_id":null,"html_url":"https://github.com/vostok/hercules","commit_stats":null,"previous_names":[],"tags_count":58,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fhercules","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fhercules/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fhercules/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fhercules/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vostok","download_url":"https://codeload.github.com/vostok/hercules/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248946744,"owners_count":21187564,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hercules","java","kafka","telemetry","transport","vostok"],"created_at":"2025-02-22T22:33:37.051Z","updated_at":"2025-10-03T19:50:14.669Z","avatar_url":"https://github.com/vostok.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.com/vostok/hercules.svg?branch=master)](https://travis-ci.com/vostok/hercules)\n\n# Vostok Hercules\nVostok Hercules is an open-source distributed system\nbased on Apache Kafka and used for reliable delivery of telemetry data from microservices\nto storage and analytics engines.\n\nVostok Hercules is developed by [Kontur](https://github.com/skbkontur) as a part of Vostok project —\nan ecosystem for easy and fast development of microservices primarily using .NET Core / C#.\n\nVostok Hercules provides abstractions for application code\nto encapsulate specific data format in telemetry engines depending on a telemetry type.\nVostok Hercules provides out of the box 4 telemetry types:\n* logs,\n* metrics,\n* distributed traces,\n* annotations.\n\n## Telemetry types\nTelemetry is an event which has a timestamp and tags.\nSet of tags depends on a [telemetry type](doc/event-schema/README.md).\n\n### Logs\nVostok Hercules delivers logs to ElasticSearch.\nAlso, logs with level `ERROR` or `FATAL` are delivered to [Sentry](https://github.com/getsentry/sentry).\nSee supported [log event schema](doc/event-schema/log-event-schema.md) for details.\n\n### Metrics\nVostok Hercules delivers metrics to Graphite, [Moira](https://github.com/moira-alert)\nor any other tool which supports Graphite format.\nMetrics can be in [the plaintext format](https://graphite.readthedocs.io/en/stable/feeding-carbon.html#the-plaintext-protocol),\nwhen a metric name consists of several components joined by a dot\nor in the modern [tagged format](https://graphite.readthedocs.io/en/stable/tags.html).\nSee supported [metric event schema](doc/event-schema/metric-event-schema.md) for details.\n\n### Distributed traces\nVostok Hercules stores distributed traces into ClickHouse.\nTo analyze traces use [Contrails](https://github.com/vostok/contrails.web) web interface.\nContrails is a tool for tree span analysis.\nSee supported [trace span schema](doc/event-schema/trace-span-schema.md) for details.\n\n### Annotations\nAnnotation marks point on graphs in Grafana and Kibana.\nVostok Hercules delivers annotations to ElasticSearch.\nSee supported [annotation event schema](doc/event-schema/annotation-event-schema.md) for details.\n\n### Other telemetry\nVostok Hercules supports an extendable processing of telemetry events and custom telemetry types as well.\nHercules can be extended by one of the following ways:\n* Use one of provided Sinks (e.g. see [Elastic Sink](hercules-elastic-sink/README.md)),\n* Write an integration via universal HTTP [Stream API](hercules-stream-api/README.md) or [Timeline API](hercules-timeline-api/README.md) in any language you prefer,\n* Write a new Sink in Java.\n\n## System Design\n![System Design](system-design.png)\n\nMinimal data unit of Vostok Hercules is an `Event`.\nEvents do not have predefined schema except well-known tags in [some cases](doc/event-schema/README.md).\nTo support schemaless processing Vostok Hercules stores and transfers events\nin the binary format with a self-described schema (see [Hercules Protocol](hercules-protocol/README.md)).\n\nA `Stream` is a collection of events of the same type.\nEach stream has a backing topic in Apache Kafka.\n\nAn application uses [hercules client](https://github.com/vostok/hercules.client)\nto write events to [Gate](hercules-gate/README.md) into streams.\nOther applications can read these events from [Stream API](hercules-stream-api/README.md)\nor [Timeline API](hercules-timeline-api/README.md) using the same client.\n\nVarious sink daemons can deliver events to various backends at the same time.\nList of supported sink daemons:\n* [Elastic Sink](hercules-elastic-sink/README.md) delivers events to ElasticSearch,\n* [Graphite Sink](hercules-graphite-sink/README.md) delivers metrics to Graphite and Moira,\n* [Sentry Sink](hercules-sentry-sink/README.md) delivers errors to Sentry.\n\nAlso, there are several auxiliary sink daemons:\n* [Stream Sink](hercules-stream-sink/README.md) combines different streams into single one,\n* [Timeline Sink](hercules-timeline-sink/README.md) delivers events to Cassandra\nto build time series or `Timelines`, see [Timeline API](hercules-timeline-api/README.md) for details,\n* [Tracing Sink](hercules-tracing-sink-clickhouse/README.md) delivers tracing spans to ClickHouse,\nsee [Tracing API](hercules-tracing-api/README.md) for details.\n\nTo analyze distributed traces use [Contrails](https://github.com/vostok/contrails.web).\nContrails read tracing spans from [Tracing API](hercules-tracing-api/README.md).\n\n### Management\nThere are several applications to manage streams and timelines:\n* [Management API](hercules-management-api/README.md) has HTTP JSON API,\n* [Stream Manager](hercules-stream-manager/README.md),\n* [Timeline Manager](hercules-timeline-manager/README.md).\n\n### Other components\nVostok Hercules uses internally this list of tools:\n* Apache Kafka for storing streams,\n* Apache ZooKeeper for meta-data (configuration for streams, timelines and others),\n* Apache Cassandra for storing timelines,\n* ClickHouse for storing tracing spans.\n\n## How to Use\n[Spaceport](https://github.com/vostok/spaceport) is an all-in-one development infrastructure.\n\nAlso, each component of Vostok Hercules can be run separately using Docker\nor as simple java application.\n\nThe official Docker images are on [Docker Hub](https://hub.docker.com/u/vstk).\n\n## How to Build\n### Build jars\nYou need to have Java 8 or higher and [Maven](https://maven.apache.org) 3.3.9 or higher.\n\nBuild project with tests:\n```\nmvn package\n```\n\nBuild project without tests:\n```\nmvn -DskipTests=true package\n```\n\n### Build Docker images\nYou need make and Docker 19.03 or higher.\n\nBuild Docker images:\n```\nmake build_images\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvostok%2Fhercules","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvostok%2Fhercules","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvostok%2Fhercules/lists"}