{"id":25633968,"url":"https://github.com/vostok/spaceport","last_synced_at":"2025-04-14T19:35:30.914Z","repository":{"id":25747552,"uuid":"105993056","full_name":"vostok/spaceport","owner":"vostok","description":"All-in-one development infrastructure","archived":false,"fork":false,"pushed_at":"2022-06-17T13:41:27.000Z","size":10383,"stargazers_count":12,"open_issues_count":1,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-28T08:01:37.271Z","etag":null,"topics":["docker","docker-compose","infrastructure","integration","tests"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vostok.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-10-06T10:34:03.000Z","updated_at":"2023-07-10T11:57:29.000Z","dependencies_parsed_at":"2022-07-27T05:32:14.015Z","dependency_job_id":null,"html_url":"https://github.com/vostok/spaceport","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fspaceport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fspaceport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fspaceport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vostok%2Fspaceport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vostok","download_url":"https://codeload.github.com/vostok/spaceport/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248946734,"owners_count":21187560,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","infrastructure","integration","tests"],"created_at":"2025-02-22T22:33:36.752Z","updated_at":"2025-04-14T19:35:30.863Z","avatar_url":"https://github.com/vostok.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Spaceport\n\nSpaceport is used to run Hercules and its infrastructure on a single host for development purposes.\n\n[Hercules ](https://github.com/vostok/hercules)\nis a system for reliable delivery of telemetry events from microservices to storage and analytics engines.\nSpaceport contains Hercules components and infrastructural services\nwhich are configured to work with each other out-of-the-box.\n\nHercules contributors can also use Spaceport to fix bugs or develop new Hercules components.\n\n## Interface description\n\n[Gate](https://github.com/vostok/hercules/tree/master/hercules-gate) is an entrance of Hercules for telemetry events.\nGate has http API.\n\nThere are several event types which are used for handling in Spaceport: logs, metrics and traces.\n\n- Hercules delivers log events to [Elasticsearch](https://www.elastic.co/elasticsearch/) -\nsearch engine for collecting and storing logs.\nYou can see these log events in web application [Kibana](https://www.elastic.co/kibana).\n\n- Hercules delivers metric events to [Graphite](https://graphiteapp.org) - metrics collection and analysis system.\nThese metric events are visualized in web interface [Grafana](https://grafana.com/).\n\n- Hercules collects trace events to DB. [Cassandra](https://cassandra.apache.org/) is used as trace DB in Spaceport.\nHercules contains [Tracing API](https://github.com/vostok/hercules/tree/master/hercules-tracing-api) for reading traces.\nIt is http API which returns traces as json.\nSpaceport contains web application [Contrails](https://github.com/vostok/contrails.web) for traces visualizing.\n\nYou can tune Hercules using [Management API](https://github.com/vostok/hercules/tree/master/hercules-management-api) (http API).\n\n## Installation\n\n[Docker](https://docs.docker.com/engine/installation/) (and `docker-compose`) are prerequisites. Check that you have them in your `PATH`:\n\n```\n$ docker --version\nDocker version 20.10.0, build 7287ab3\n\n$ docker-compose --version\ndocker-compose version 1.27.4, build 40524192\n```\nGo to the directory where the Spaceport project will be located.\n\nClone this repository:\n\n```\n$ git clone https://github.com/vostok/spaceport.git\n```\n\n## Usage\n\n### Launch\n\nGo to the root directory of Spaceport project.\n\nYou can use utility `make` to run Spaceport. Run the command `make` to download and run all necessary containers. Wait for the command to complete.\nThe full launch of the services may take another minute.\n\n\u003e **Beware.** You won't be able to run Spaceport on a very old or weak machine. Mid-2014 MacBook Pro with 8GB RAM shows acceptable performance.\n\nGo to [localhost:6300](http://localhost:6300) to check if all components are up and running.\nThe indicators in front of the component names should be green.\n\n![](images/health-monitor.png)\n\nOther commands are also available:\n\n- `make down` will stop and remove all containers\n- `make pull` will pull latest versions of containers and overwrite your changes to containers\n\nIf you don't have utility `make`, look inside the `Makefile` for commands.\n\n### Web Applications\n\nSpaceport provides several end-user applications.\n\n- [Grafana](https://grafana.com) at [localhost:6304](http://localhost:6304) to explore and plot metrics\n- [Kibana](https://www.elastic.co/products/kibana) at [localhost:6305](http://localhost:6305) to explore logs\n- [Management API](https://github.com/vostok/hercules/tree/master/hercules-management-api) at [localhost:6307](http://localhost:6307) to manage Hercules\n- [Tracing API](https://github.com/vostok/hercules/tree/master/hercules-tracing-api) at [localhost:6308](http://localhost:6308) to see traces as json\n- [Contrails](https://github.com/vostok/contrails.web) at [localhost:6301](http://localhost:6301) to visualize traces\n\n## Test\n\nYou can test Spaceport work. Go to root directory of Spaceport project and run test script:\n```\n$ sh test.sh\n```\n\nTest script will send one test log event, one test metric event and one test trace event to Hercules.\nSee below how to check if the Spaceport is working properly.\n\n### Test logs\n\nGo to **Kibana** to see test log event: [localhost:6305](http://localhost:6305).\n\nCreate index-pattern if you are for first time in Kibana: go to **Management** menu, then click **Index Patterns** and **Create index pattern**.\nEnter `test-index*` and click **Next step**.\n\n![](images/kibana-pattern.png)\n\nSelect **@timestamp** as timer filter field on the next step and click **Create index pattern**.\n\nGo to **Discover** menu, then select index pattern: `test-index*`.\n\nIf successful, you will see the timing diagram and log event information below.\nYou can change displayed time period in the upper right corner of the window.\n\n![](images/kibana-log.png)\n\n### Test metrics\n\nGo to **Grafana** to see test metric event: [localhost:6304](http://localhost:6304).\n\nGrafana will ask for login and password if you are for first time in Grafana. Entry login: `admin`, password: `admin`.\nThen Grafana will ask you to set new password. Entry new or old password.\n\nClick **New dashboard** on new window. Click **Add query** on next window.\n\nThen click **Select metric** in **Series** row on new window.\nSelect `test-metric` on drop-down list. Then select `some-value` on the right similarly.\n\nIf successful, you will see point on the graph.\nkeep in mind that points may be displayed with a delay of several minutes.\n\n![](images/grafana-metric.png)\n\n### Hercules metrics\n\nYou can see metrics of Hercules components, it may be Gate for example.\n\nClick to the pencil icon on the last Grafana window of test metric.\nEntry query: `vostok.hercules.gate.production.default.1.SendRequestProcessor.sentEvents.count`\n\n![](images/grafana-hercules-metric.png)\n\n### Test traces\n\nYou can see test trace.\n\nFind line with `traceId` among the logs of test script (look it in the terminal where you have launched test script).\nThe line looks like this:\n```\n2020-12-24T08:50:30.292Z [main] DEBUG r.k.v.h.cli.common.EventSamples - Trace event is generated with id: d85f68a3-ba77-404d-8759-4a3c5bfa241a, traceId: f8bf1d3b-74ca-4be5-97d1-7fbd84324a4c\n```\nCopy value of `traceId` from this line.\n\nSubmit request to **Tracing API** with copied value instead of `\u003ctraceId\u003e` in sample:\n```\ncurl \"http://vm-hercules-sp:6308/trace?traceId=\u003ctraceId\u003e\"\n```\nIf successful, you will see json with trace information like this:\n```\n{\"result\":[{\"beginTimestamp\":\"2020-12-10T11:08:44.000000000+03:00\",\"endTimestamp\":\"2020-12-10T11:10:43.000000000+03:00\",\"traceId\":\"f8bf1d3b-74ca-4be5-97d1-7fbd84324a4c\",\"spanId\":\"f419a60e-160d-4fc2-aa92-aa20234f9f28\",\"annotations\":{\"application\":\"TestApplication\",\"some_key\":\"test\"},\"parentSpanId\":\"4ebf52fe-ee9b-4ab9-a3a9-8ab751380819\"}]}\n```\nYou can see the trace in **Contrails**.\nGo to [localhost:6301](http://localhost:6301). Entry copied `traceId` value.\nIf successful, you will see diagram. Click to `TestApplication` line to see trace details.\n\n![](images/contrails.png)\n\n# Additional notes\n\n###To add new stream\nBy default, stream name should contains prefix 'logs', 'metrics' or 'traces'. \nSubmit request to **Management API** with stream name instead of `\u003cstream name\u003e`:\n\n```\ncurl -v -X POST 'management-api:6307/streams/create' --header 'Content-Type: text/plain' -H 'Authorization: Hercules masterApiKey 123' --data '{\n    \"type\": \"base\",\n    \"name\": \"\u003cstream name\u003e\",\n    \"partitions\": 1,\n    \"shardingKey\": [],\n    \"ttl\": 86400000\n}'\n```\nMore informations in [Management API documentation](https://github.com/vostok/hercules/blob/master/hercules-management-api/README.md).\n\n###To add new apiKey\nSubmit request to **Management API** with new apiKey instead of `\u003capiKey\u003e` and stream prefix instead of `\u003cprefix\u003e`:\n```\ncurl -v -X POST 'management-api:6307/rules/set?key=\u003capiKey\u003e\u0026pattern=\u003cprefix\u003e*\u0026rights=rwm' -H 'Authorization: Hercules masterApiKey 123'\n```\nMore informations in [Management API documentation](https://github.com/vostok/hercules/blob/master/hercules-management-api/README.md).\n\n###To use custom stream name:\nIf your stream name don't start with default prefix (logs*, metrics* or traces*),\nchange value in line `sink.pattern=\u003cprefix name\u003e`.\n\nTo change prefix for logs, edit file\n`/etc/properties/elastic-sink/application.properties`.\n\nTo change prefix for metrics, edit file\n`/etc/properties/graphite-sink/application.properties`.\n\nTo change prefix for traces, edit file\n`/etc/properties/tracing-sink/application.properties`.\n\n###To configure events\nEdit properties file:\n\n`/etc/properties/gateway-client/log/application.properties` - for logs\n\n`/etc/properties/gateway-client/metric/application.properties` - for metrics\n\n`/etc/properties/gateway-client/trace/application.properties` - for traces\n\nYou can configure:\n\n`gate-client.sender.stream` - stream in Kafka used for events writing\n\n`gate-client.sender.type` - event type. Default value: `log`. Values:\n- `log` - log-event for ElasticSearch and Sentry\n- `trace` - trace-event\n- `metric`- metric-event\n\n`gate-client.sender.apiKey` - API-key of specified stream\n\n`gate=client.sender.url` - URL of Hercules Gate server\n\n`gate-client.sender.eventCount` - event count per sending request. Default value: 1\n\n`gate-client.sender.requestCount` - sending request count. Default value: 1\n\nYou can manage your events by changing file `test.sh` or write your own script.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvostok%2Fspaceport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvostok%2Fspaceport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvostok%2Fspaceport/lists"}