{"id":33178948,"url":"https://github.com/ExpediaDotCom/haystack","last_synced_at":"2025-11-20T21:03:00.750Z","repository":{"id":23728902,"uuid":"98865832","full_name":"ExpediaDotCom/haystack","owner":"ExpediaDotCom","description":"Top level repository for Haystack, containing documentation and deployment scripts","archived":false,"fork":false,"pushed_at":"2025-11-11T23:11:29.000Z","size":48833,"stargazers_count":312,"open_issues_count":55,"forks_count":47,"subscribers_count":31,"default_branch":"master","last_synced_at":"2025-11-12T01:08:19.221Z","etag":null,"topics":["distributed-tracing","hacktoberfest","metrics","opentracing","telemetry","tracing","trending"],"latest_commit_sha":null,"homepage":"http://expediadotcom.github.io/haystack","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ExpediaDotCom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2017-07-31T08:23:56.000Z","updated_at":"2025-09-26T17:38:00.000Z","dependencies_parsed_at":"2023-11-22T18:39:16.384Z","dependency_job_id":"f4a1df2a-9af8-40b8-bb39-216987cdfcdb","html_url":"https://github.com/ExpediaDotCom/haystack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ExpediaDotCom/haystack","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ExpediaDotCom","download_url":"https://codeload.github.com/ExpediaDotCom/haystack/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack/sbom","scorecard":{"id":47778,"data":{"date":"2025-08-11","repo":{"name":"github.com/ExpediaDotCom/haystack","commit":"863fc405c5f1cedd68452f6f27ab74f08fb0865a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":9,"reason":"Found 12/13 approved changesets -- score normalized to 9","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: deployment/grafana/build/docker/Dockerfile:1: pin your Docker image by updating alpine:3.11.3 to alpine:3.11.3@sha256:ab00606a42621fb68f2ed6ad3c88be54397f981a7b70a79db3d1172b11c4367d","Warn: npmCommand not pinned by hash: docsite/generate_docsite.sh:8","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 npmCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"69 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-968p-4wvh-cqc8","Warn: Project is vulnerable to: GHSA-67hx-6x53-jw92","Warn: Project is vulnerable to: GHSA-93q8-gq69-wqmw","Warn: Project is vulnerable to: GHSA-fwr7-v2mv-hh25","Warn: Project is vulnerable to: GHSA-pp7h-53gx-mx7r","Warn: Project is vulnerable to: GHSA-qwcr-r2fm-qrc7","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-grv7-fg5c-xmjg","Warn: Project is vulnerable to: GHSA-w8qv-6jwh-64r5","Warn: Project is vulnerable to: GHSA-257v-vj4p-3w2h","Warn: Project is vulnerable to: GHSA-pxg6-pf52-xh8x","Warn: Project is vulnerable to: GHSA-3xgq-45jj-v275","Warn: Project is vulnerable to: GHSA-gxpj-cx7g-858c","Warn: Project is vulnerable to: GHSA-w573-4hg7-7wgq","Warn: Project is vulnerable to: GHSA-6h5x-7c5m-7cr7","Warn: Project is vulnerable to: GHSA-rv95-896h-c2vc","Warn: Project is vulnerable to: GHSA-qw6h-vgh9-j6wx","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-pfrx-2q88-qq97","Warn: Project is vulnerable to: GHSA-7wwv-vh3v-89cq","Warn: Project is vulnerable to: GHSA-43f8-2h32-f4cj","Warn: Project is vulnerable to: GHSA-rc47-6667-2j5j","Warn: Project is vulnerable to: GHSA-qqgx-2p2h-9c37","Warn: Project is vulnerable to: GHSA-7r28-3m3f-r2pr","Warn: Project is vulnerable to: GHSA-r8j5-h5cx-65gg","Warn: Project is vulnerable to: GHSA-896r-f27r-55mw","Warn: Project is vulnerable to: GHSA-9c47-m6qq-7p4h","Warn: Project is vulnerable to: GHSA-76p3-8jx3-jpfq","Warn: Project is vulnerable to: GHSA-3rfm-jhwj-7488","Warn: Project is vulnerable to: GHSA-hhq3-ff78-jv3g","Warn: Project is vulnerable to: GHSA-29mw-wpgm-hmr9","Warn: Project is vulnerable to: GHSA-35jh-r3h4-6jhm","Warn: Project is vulnerable to: GHSA-p6mc-m468-83gw","Warn: Project is vulnerable to: GHSA-952p-6rrq-rcjv","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-xvch-5gv4-984h","Warn: Project is vulnerable to: GHSA-rp65-9cf3-cjxr","Warn: Project is vulnerable to: GHSA-hj48-42vr-x3v9","Warn: Project is vulnerable to: GHSA-9wv6-86v2-598j","Warn: Project is vulnerable to: GHSA-rhx6-c78j-4q9w","Warn: Project is vulnerable to: GHSA-566m-qj78-rww5","Warn: Project is vulnerable to: GHSA-hwj9-h5mp-3pm3","Warn: Project is vulnerable to: GHSA-7fh5-64p2-3v2j","Warn: Project is vulnerable to: GHSA-h4hr-7fg3-h35w","Warn: Project is vulnerable to: GHSA-gj77-59wh-66hg","Warn: Project is vulnerable to: GHSA-hqhp-5p83-hx96","Warn: Project is vulnerable to: GHSA-3949-f494-cm99","Warn: Project is vulnerable to: GHSA-x7hr-w5r2-h6wg","Warn: Project is vulnerable to: GHSA-hrpp-h998-j3pp","Warn: Project is vulnerable to: GHSA-5q6m-3h65-w53x","Warn: Project is vulnerable to: GHSA-p8p7-x288-28g6","Warn: Project is vulnerable to: GHSA-c2qf-rxjj-qqgw","Warn: Project is vulnerable to: GHSA-44c6-4v22-4mhx","Warn: Project is vulnerable to: GHSA-4x5v-gmq8-25ch","Warn: Project is vulnerable to: GHSA-m6fv-jmcg-4jfg","Warn: Project is vulnerable to: GHSA-cm22-4g7w-348p","Warn: Project is vulnerable to: GHSA-jv35-xqg7-f92r","Warn: Project is vulnerable to: GHSA-g4rg-993r-mgx7","Warn: Project is vulnerable to: GHSA-4rq4-32rv-6wp6","Warn: Project is vulnerable to: GHSA-64g7-mvw6-v9qj","Warn: Project is vulnerable to: GHSA-52f5-9888-hmc6","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3","Warn: Project is vulnerable to: GHSA-7p7h-4mm5-852v","Warn: Project is vulnerable to: GHSA-9m6j-fcg5-2442","Warn: Project is vulnerable to: GHSA-hh27-ffr2-f2jc","Warn: Project is vulnerable to: GHSA-rqff-837h-mm52","Warn: Project is vulnerable to: GHSA-8v38-pw62-9cw2","Warn: Project is vulnerable to: GHSA-hgjh-723h-mx2j","Warn: Project is vulnerable to: GHSA-jf5r-8hm2-f872"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-14T23:07:44.960Z","repository_id":23728902,"created_at":"2025-08-14T23:07:44.960Z","updated_at":"2025-08-14T23:07:44.960Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285511775,"owners_count":27184237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-20T02:00:05.334Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-tracing","hacktoberfest","metrics","opentracing","telemetry","tracing","trending"],"created_at":"2025-11-16T03:00:36.816Z","updated_at":"2025-11-20T21:03:00.745Z","avatar_url":"https://github.com/ExpediaDotCom.png","language":"HCL","funding_links":[],"categories":["分布式跟踪"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/ExpediaDotCom/haystack.svg?branch=master)](https://travis-ci.org/ExpediaDotCom/haystack)\n[![Docs](https://img.shields.io/badge/docs-current-brightgreen.svg)](https://expediadotcom.github.io/haystack/)\n[![License](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg)](https://github.com/ExpediaDotCom/haystack/blob/master/LICENSE)\n[![Join the chat at https://gitter.im/expedia-haystack/Lobby](https://badges.gitter.im/expedia-haystack/Lobby.svg)](https://gitter.im/expedia-haystack/Lobby?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n# haystack\n\nThis is an uber repository for all Haystack components. Visit our documentation site @ https://expediadotcom.github.io/haystack/\n\n## Overview\n[Haystack](https://expediadotcom.github.io/haystack/) is an Expedia-backed open source project to facilitate detection \nand remediation of problems with enterprise-level web services and websites.\n\n### The Problem\n\nModern websites rely on dozens or even hundreds of services to function. These services are often spread across many platforms, data centers, and teams. Each service logs information of various kinds:\n\n* Telemetry data with tags or key/value pairs associated with the event\n* Request/response logs (the XML, JSON, etc. sent to and received from the service)\n* Application logs (typically errors logged via log4j or a similar system)\n* Metrics\n\nWith so much information spread across many different places, it can be challenging and time consuming to figure out where to look for the cause of a failure or slowdown.\n\n### The Solution\n\nHaystack uses tracing data from a central store to help you locate the source of the problem -- to drill down to the precise part of a service transaction where failures or latency are occurring -- and find the proverbial \"needle in a haystack\". Once you know specifically where the problem is happening, it's much easier to identify the appropriate diagnostic data, understand the data, find the problem, and fix it.\n\n![High Level Block Diagram](docsite/static/img/Haystack_Components.png)\n\n#### Infrastructure\nThe Haystack system includes an easy-to-use \"one click\" deployment mechanism, based on \n[Kubernetes](https://en.wikipedia.org/wiki/Kubernetes), that deploys a working development environment with working\nimplementations of all of the services in the block diagram above. This same mechanism, with different configurations, \ndeploys to test and production environments as well. See the collection of scripts, CloudFormation templates, and YAML \nfiles in the [deployment](deployment) module for details.\n\n#### Kafka\nInternal [Kafka](https://en.wikipedia.org/wiki/Apache_Kafka) is the Haystack message bus. The messages that enter the\nHaystack system are [Span](https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/span.proto) objects in\n[protobuf](https://en.wikipedia.org/wiki/Protocol_Buffers) format, and the modules below usually communicate with\neach other via this message bus.\n\n#### haystack-trends\nThe [haystack-trends](https://github.com/ExpediaDotCom/haystack-trends) module detects anomalies in metrics, based on \nuser-defined \"trend templates\" that define the levels at which metrics should be considered \"out of trend.\" A single \nanomalous metric does not necessarily merit alarming as out of trend, so the haystack-trends unit aggregates as directed\nby the trend templates. For example, a particular service call might have a typical TP99 value of 100 milliseconds. \n(\"TP\" means \"top percentile\" and TP99 refers to the minimum time under which 99% of the calls to the service have \nfinished.) The trend template for such a service might declare that the TP99 metric is out of trend when it exceeds 150 \nmilliseconds, a value that was chosen to be low enough to notify interested parties of a potential problem before it \nbecomes serious but high enough to minimize false positive alarms. The haystack-trends module stores its data in a Time \nSeries Data Base (TSDB).\n\n#### haystack-pipes\nThe [haystack-pipes](https://github.com/ExpediaDotCom/haystack-pipes) module delivers a human-friendly JSON version of \nHaystack spans to zero or more \"durable\" locations for more permanent storage. Current \"plug in\" candidates for such \nstorage include:\n* [Amazon Kinesis Firehose](https://aws.amazon.com/kinesis/firehose/) is an AWS service that facilitates loading \nstreaming data into AWS. Note that its \n[PutRecordBatch API](http://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html) accepts up to\n500 records, with a maximum size of 4 MB for each put request. The plug in will batch the records appropriately, to\nminimize AWS costs. Kinesis Firehose can be configured to deliver the data to\n    * [Amazon S3](https://aws.amazon.com/s3/)\n    * [Amazon Redshift](https://aws.amazon.com/redshift/)\n    * [Amazon Elasticsearch Service](https://aws.amazon.com/elasticsearch-service/)\n\n#### Indexer\nThe Indexer module collects Span objects sharing a trace ID into\n[SpanBuffer](https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/spanBuffer.proto) protobuf objects \nwhich are written to the Kafka message bus. The Span objects are identified as related by the parent span IDs; the Span\nwith a null parent Span ID is the root Span, child Spans have a parent Span ID of the root Span, grandchild Spans have a\nparent span ID of the appropriate child Span, etc.\n\n#### Spans\nThe Spans module writes Span objects into a persistent store. That persistent store consists of two pieces: the Span\ndata is stored in Cassandra, and the Span metadata is stored in ElasticSearch. Sampling, with the ability to force\nstoring a particular Span, will be available (under configuration) to keep the size of the Cassandra and ElasticSearch\nstores reasonable, given the large volume of Span objects in a production system. \n\n#### Dependencies\nThe Dependencies module uses the parent/child relationships of Span objects to create dependency graphs for each\nservice, stored in a graph database.\n\n#### haystack-ui\nThe [haystack-ui](https://github.com/ExpediaDotCom/haystack-ui) (User Interface) module exposes (through a website) the\nSpans, Stitched Spans, haystack-trends, and Dependencies created by the other modules. To simplify the API from the \nhaystack-ui module to the Span databases, a small Query module bundles the data from the Cassandra store with the\nmetadata from the ElasticSearch store.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FExpediaDotCom%2Fhaystack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FExpediaDotCom%2Fhaystack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FExpediaDotCom%2Fhaystack/lists"}