{"id":20337319,"url":"https://github.com/expediadotcom/haystack-collector","last_synced_at":"2025-04-11T22:41:53.317Z","repository":{"id":50117209,"uuid":"103362150","full_name":"ExpediaDotCom/haystack-collector","owner":"ExpediaDotCom","description":"haystack component that collects spans from various sources and publish to kafka","archived":false,"fork":false,"pushed_at":"2022-07-01T21:23:54.000Z","size":391,"stargazers_count":4,"open_issues_count":5,"forks_count":0,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-25T18:45:10.347Z","etag":null,"topics":["distributed-tracing","http","kinesis"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ExpediaDotCom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-13T06:28:52.000Z","updated_at":"2022-04-15T19:31:56.000Z","dependencies_parsed_at":"2022-08-26T09:41:35.600Z","dependency_job_id":null,"html_url":"https://github.com/ExpediaDotCom/haystack-collector","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ExpediaDotCom","download_url":"https://codeload.github.com/ExpediaDotCom/haystack-collector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248493022,"owners_count":21113159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-tracing","http","kinesis"],"created_at":"2024-11-14T21:08:38.689Z","updated_at":"2025-04-11T22:41:53.285Z","avatar_url":"https://github.com/ExpediaDotCom.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/ExpediaDotCom/haystack-collector.svg?branch=master)](https://travis-ci.org/ExpediaDotCom/haystack-collector)\n[![License](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg)](https://github.com/ExpediaDotCom/haystack/blob/master/LICENSE)\n\n# haystack-collector\nThis haystack component collects spans from various sources and publish to kafka. As of today, we support two sources:\n\n1. Kinesis: Kinesis span collector reads proto serialized spans from a kinesis stream, validates it and write the data to configured kafka topic.\n2. Http: Http span collector listens on port 8080 for proto or json serialized spans, validate them and write to configured kafka topic. For more detail read [this](./http/README.md)\n\nSpans are validated to ensure they dont't contain an empty service and operation name. The startTime and duration should be non-zero.\n\n## Building\n\n####\nSince this repo contains haystack-idl as the submodule, so use the following to clone the repo\n* git clone --recursive git@github.com:ExpediaDotCom/haystack-collector.git .\n\n####Prerequisite: \n\n* Make sure you have Java 1.8\n* Make sure you have maven 3.3.9 or higher\n* Make sure you have docker 1.13 or higher\n\n\nNote : For mac users you can download docker for mac to set you up for the last two steps.\n\n####Build\n\nFor a full build, including unit tests and integration tests, docker image build, you can run -\n```\nmake all\n```\n\n####Integration Test\n\n####Prerequisite:\n1. Install docker using Docker Tools or native docker if on mac\n2. Verify if docker-compose is installed by running following command else install it.\n```\ndocker-compose\n\n```\n\nRun the build and integration tests for individual components with\n```\nmake kinesis\n\nor\n\nmake http\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexpediadotcom%2Fhaystack-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexpediadotcom%2Fhaystack-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexpediadotcom%2Fhaystack-collector/lists"}