Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/expediadotcom/haystack-collector

haystack component that collects spans from various sources and publish to kafka
https://github.com/expediadotcom/haystack-collector

distributed-tracing http kinesis

Last synced: 1 day ago
JSON representation

haystack component that collects spans from various sources and publish to kafka

Host: GitHub
URL: https://github.com/expediadotcom/haystack-collector
Owner: ExpediaDotCom
License: apache-2.0
Created: 2017-09-13T06:28:52.000Z (about 7 years ago)
Default Branch: master
Last Pushed: 2022-07-01T21:23:54.000Z (over 2 years ago)
Last Synced: 2023-07-26T21:48:09.323Z (over 1 year ago)
Topics: distributed-tracing, http, kinesis
Language: Scala
Size: 382 KB
Stars: 4
Watchers: 16
Forks: 0
Open Issues: 5
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

[![Build Status](https://travis-ci.org/ExpediaDotCom/haystack-collector.svg?branch=master)](https://travis-ci.org/ExpediaDotCom/haystack-collector)
[![License](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg)](https://github.com/ExpediaDotCom/haystack/blob/master/LICENSE)

# haystack-collector
This haystack component collects spans from various sources and publish to kafka. As of today, we support two sources:

1. Kinesis: Kinesis span collector reads proto serialized spans from a kinesis stream, validates it and write the data to configured kafka topic.
2. Http: Http span collector listens on port 8080 for proto or json serialized spans, validate them and write to configured kafka topic. For more detail read [this](./http/README.md)

Spans are validated to ensure they dont't contain an empty service and operation name. The startTime and duration should be non-zero.

## Building

####
Since this repo contains haystack-idl as the submodule, so use the following to clone the repo
* git clone --recursive [email protected]:ExpediaDotCom/haystack-collector.git .

####Prerequisite:

* Make sure you have Java 1.8
* Make sure you have maven 3.3.9 or higher
* Make sure you have docker 1.13 or higher

Note : For mac users you can download docker for mac to set you up for the last two steps.

####Build

For a full build, including unit tests and integration tests, docker image build, you can run -
```
make all
```

####Integration Test

####Prerequisite:
1. Install docker using Docker Tools or native docker if on mac
2. Verify if docker-compose is installed by running following command else install it.
```
docker-compose

```

Run the build and integration tests for individual components with
```
make kinesis

make http

```