{"id":20337313,"url":"https://github.com/expediadotcom/haystack-trends","last_synced_at":"2025-09-29T07:50:01.433Z","repository":{"id":23952135,"uuid":"100239899","full_name":"ExpediaDotCom/haystack-trends","owner":"ExpediaDotCom","description":null,"archived":false,"fork":false,"pushed_at":"2022-10-04T23:34:12.000Z","size":905,"stargazers_count":11,"open_issues_count":3,"forks_count":3,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-08-11T22:46:12.810Z","etag":null,"topics":["distributed-tracing","trends"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ExpediaDotCom.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-08-14T07:22:20.000Z","updated_at":"2024-03-21T17:32:47.000Z","dependencies_parsed_at":"2022-07-25T13:52:08.521Z","dependency_job_id":null,"html_url":"https://github.com/ExpediaDotCom/haystack-trends","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/ExpediaDotCom/haystack-trends","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-trends","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-trends/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-trends/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-trends/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ExpediaDotCom","download_url":"https://codeload.github.com/ExpediaDotCom/haystack-trends/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ExpediaDotCom%2Fhaystack-trends/sbom","scorecard":{"id":47792,"data":{"date":"2025-08-11","repo":{"name":"github.com/ExpediaDotCom/haystack-trends","commit":"ab434e31d6fde601ac71b12a3b79876fc1aa6b48"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":9,"reason":"binaries present in source code","details":["Warn: binary detected: .mvn/wrapper/maven-wrapper.jar:1"],"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: containerImage not pinned by hash: span-timeseries-transformer/build/docker/Dockerfile:1: pin your Docker image by updating openjdk:8-jre to openjdk:8-jre@sha256:667a15e7bc533a90fb39ddb7e5bed63162ac3c13a97e6c698bf4f139f51b7d33","Warn: containerImage not pinned by hash: timeseries-aggregator/build/docker/Dockerfile:1: pin your Docker image by updating openjdk:8-jre to openjdk:8-jre@sha256:667a15e7bc533a90fb39ddb7e5bed63162ac3c13a97e6c698bf4f139f51b7d33","Info:   0 out of   2 containerImage dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"86 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-vmq6-5m68-f53m","Warn: Project is vulnerable to: GHSA-668q-qrv7-99fm","Warn: Project is vulnerable to: GHSA-6v67-2wr5-gvf4","Warn: Project is vulnerable to: GHSA-pr98-23f8-jwxv","Warn: Project is vulnerable to: GHSA-h46c-h94j-95f3","Warn: Project is vulnerable to: GHSA-wf8f-6423-gfxg","Warn: Project is vulnerable to: GHSA-27xj-rqx5-2255","Warn: Project is vulnerable to: GHSA-288c-cq4h-88gq","Warn: Project is vulnerable to: GHSA-4gq5-ch57-c2mg","Warn: Project is vulnerable to: GHSA-4w82-r329-3q67","Warn: Project is vulnerable to: GHSA-57j2-w4cx-62h2","Warn: Project is vulnerable to: GHSA-58pp-9c76-5625","Warn: Project is vulnerable to: GHSA-5949-rw7g-wx7w","Warn: Project is vulnerable to: GHSA-5p34-5m6p-p58g","Warn: Project is vulnerable to: GHSA-5r5r-6hpj-8gg9","Warn: Project is vulnerable to: GHSA-5ww9-j83m-q7qx","Warn: Project is vulnerable to: GHSA-645p-88qh-w398","Warn: Project is vulnerable to: GHSA-6fpp-rgj9-8rwc","Warn: Project is vulnerable to: GHSA-6wqp-v4v6-c87c","Warn: Project is vulnerable to: GHSA-758m-v56v-grj4","Warn: Project is vulnerable to: GHSA-85cw-hj65-qqv9","Warn: Project is vulnerable to: GHSA-89qr-369f-5m5x","Warn: Project is vulnerable to: GHSA-8c4j-34r4-xr8g","Warn: Project is vulnerable to: GHSA-8w26-6f25-cm9x","Warn: Project is vulnerable to: GHSA-95cm-88f5-f2c7","Warn: Project is vulnerable to: GHSA-9gph-22xh-8x98","Warn: Project is vulnerable to: GHSA-9m6f-7xcq-8vf8","Warn: Project is vulnerable to: GHSA-9mxf-g3x6-wv74","Warn: Project is vulnerable to: GHSA-9vvp-fxw6-jcxr","Warn: Project is vulnerable to: GHSA-c265-37vj-cwcc","Warn: Project is vulnerable to: GHSA-c2q3-4qrh-fm48","Warn: Project is vulnerable to: GHSA-c8hm-7hpq-7jhg","Warn: Project is vulnerable to: GHSA-cf6r-3wgc-h863","Warn: Project is vulnerable to: GHSA-cggj-fvv3-cqwv","Warn: Project is vulnerable to: GHSA-cjjf-94ff-43w7","Warn: Project is vulnerable to: GHSA-cmfg-87vq-g5g4","Warn: Project is vulnerable to: GHSA-cvm9-fjm9-3572","Warn: Project is vulnerable to: GHSA-f3j5-rmmp-3fc5","Warn: Project is vulnerable to: GHSA-f9hv-mg5h-xcw9","Warn: Project is vulnerable to: GHSA-f9xh-2qgp-cq57","Warn: Project is vulnerable to: GHSA-fmmc-742q-jg75","Warn: Project is vulnerable to: GHSA-fqwf-pjwf-7vqv","Warn: Project is vulnerable to: GHSA-gjmw-vf9h-g25v","Warn: Project is vulnerable to: GHSA-gwp4-hfv6-p7hw","Warn: Project is vulnerable to: GHSA-gww7-p5w4-wrfv","Warn: Project is vulnerable to: GHSA-h3cw-g4mq-c5x2","Warn: Project is vulnerable to: GHSA-h4rc-386g-6m85","Warn: Project is vulnerable to: GHSA-h592-38cm-4ggp","Warn: Project is vulnerable to: GHSA-h822-r4r5-v8jg","Warn: Project is vulnerable to: GHSA-j823-4qch-3rgm","Warn: Project is vulnerable to: GHSA-jjjh-jjxp-wpff","Warn: Project is vulnerable to: GHSA-m6x4-97wx-4q27","Warn: Project is vulnerable to: GHSA-mc6h-4qgp-37qh","Warn: Project is vulnerable to: GHSA-mph4-vhrx-mv67","Warn: Project is vulnerable to: GHSA-mx7p-6679-8g3q","Warn: Project is vulnerable to: GHSA-mx9v-gmh4-mgqw","Warn: Project is vulnerable to: GHSA-p43x-xfjf-5jhr","Warn: Project is vulnerable to: GHSA-q93h-jc49-78gg","Warn: Project is vulnerable to: GHSA-qjw2-hr98-qgfh","Warn: Project is vulnerable to: GHSA-qmqc-x3r4-6v39","Warn: Project is vulnerable to: GHSA-qr7j-h6gg-jmgc","Warn: Project is vulnerable to: GHSA-r3gr-cxrf-hg25","Warn: Project is vulnerable to: GHSA-r695-7vr9-jgc2","Warn: Project is vulnerable to: GHSA-rf6r-2c4q-2vwg","Warn: Project is vulnerable to: GHSA-rfx6-vp9g-rh7v","Warn: Project is vulnerable to: GHSA-rgv9-q543-rqg4","Warn: Project is vulnerable to: GHSA-rpr3-cw39-3pxh","Warn: Project is vulnerable to: GHSA-v3xw-c963-f5hc","Warn: Project is vulnerable to: GHSA-v585-23hc-c647","Warn: Project is vulnerable to: GHSA-vfqx-33qm-g869","Warn: Project is vulnerable to: GHSA-w3f4-3q6j-rh82","Warn: Project is vulnerable to: GHSA-wh8g-3j2c-rqj5","Warn: Project is vulnerable to: GHSA-x2w5-5m2g-7h5m","Warn: Project is vulnerable to: GHSA-5mg8-w23w-74h3","Warn: Project is vulnerable to: GHSA-7g45-4rm6-3mm3","Warn: Project is vulnerable to: GHSA-mvr2-9pj6-7w5j","Warn: Project is vulnerable to: GHSA-4gg5-vx3j-xwc7","Warn: Project is vulnerable to: GHSA-735f-pc8j-v9w8","Warn: Project is vulnerable to: GHSA-77rm-9x9h-xj3g","Warn: Project is vulnerable to: GHSA-g5ww-5jh7-63cx","Warn: Project is vulnerable to: GHSA-h4h5-3hr4-j3g2","Warn: Project is vulnerable to: GHSA-wrvw-hg22-4m67","Warn: Project is vulnerable to: GHSA-55g7-9cwv-5qfv","Warn: Project is vulnerable to: GHSA-fjpj-2g6w-x25r","Warn: Project is vulnerable to: GHSA-pqr6-cmr2-h8hf","Warn: Project is vulnerable to: GHSA-qcwq-55hx-v3vh"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-14T23:07:51.747Z","repository_id":23952135,"created_at":"2025-08-14T23:07:51.747Z","updated_at":"2025-08-14T23:07:51.747Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":277483294,"owners_count":25825561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-29T02:00:09.175Z","response_time":84,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-tracing","trends"],"created_at":"2024-11-14T21:08:36.596Z","updated_at":"2025-09-29T07:50:01.413Z","avatar_url":"https://github.com/ExpediaDotCom.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/ExpediaDotCom/haystack-trends.svg?branch=master)](https://travis-ci.org/ExpediaDotCom/haystack-trends)\n[![codecov](https://codecov.io/gh/ExpediaDotCom/haystack-trends/branch/master/graph/badge.svg)](https://codecov.io/gh/ExpediaDotCom/haystack-trends)\n\n# Haystack Trends\n\nhaystack-trends contains the required modules for trending the spans pushed to haystack. We currently plan to compute three trends for each \ncombination `service_name` and `operation_name` contained in the spans (refer to the [span schema](https://github.com/ExpediaDotCom/haystack-idl/blob/master/proto/span.proto) for details of the fields in the span  )\n\n1. success_count `[1min, 5min, 15min, 1hour]`\n2. failure_count `[1min, 5min, 15min, 1hour]`\n3. duration `[mean, median, std-dev, 99 percentile, 95 percentile]`\n\n\u003e *Note:* If an error tag is present and has a value of true, then the span will be treated as a failure. In all other scenarios, it will be treated as a success. \n\nMore trends can be computed by adding a [transformer](https://github.com/ExpediaDotCom/haystack-trends/tree/master/span-timeseries-transformer/src/main/scala/com/expedia/www/haystack/trends/transformer)\nto create the metric point and adding an [aggregation-rule](https://github.com/ExpediaDotCom/haystack-trends/tree/master/timeseries-aggregator/src/main/scala/com/expedia/www/haystack/trends/aggregation/rules) for it\n\n## Required Reading\n \nIn order to understand the haystack-trends one must be familiar with the [haystack](https://github.com/ExpediaDotCom/haystack) project. Its written in kafka-streams(http://docs.confluent.io/current/streams/index.html) \nand hence some prior knowledge of kafka-streams would be useful. \n \n## Technical Details\n![High Level Block Diagram](documents/diagrams/haystack_trends.png)\n\n\nHaystack trends is a collection of modules which reads spans and pushes aggregated metric points to kafka, each module runs as individual apps and talk to each other via kafka.\n    \n* [span-timeseries-transformer](https://github.com/ExpediaDotCom/haystack-trends/tree/master/span-timeseries-transformer) - this app is responsible \nfor reading spans, converting them to metric points and pushing raw metric points to kafka partitioned by metric-key\n\n* [timeseries-aggregator](https://github.com/ExpediaDotCom/haystack-trends/tree/master/timeseries-aggregator) - this app is responsible \nfor reading metric points, aggregating them based on rules and pushing the aggregated metric points to kafka\n\nThe timeseries metric points are metrictank complient and can be directly consumed by [metrictank](https://github.com/grafana/metrictank), one can write their own serde if they want to push the metrics out in some other timeseries format.\n\nSample [MetricPoint](https://github.com/ExpediaDotCom/haystack-trends/blob/master/commons/src/main/scala/com/expedia/www/haystack/trends/commons/entities/MetricPoint.scala) : \n```json\n{\n\t\"type\": \"count\",\n\t\"metric\": \"duration\",\n\t\"tags\": {\n\t\t\"client\": \"expweb\",\n\t\t\"operationName\": \"getOffers\"\n\t},\n\t\"epochTimeInSeconds\": 1492641000,\n\t\"value\": 420.02\n}\n```\n \nThe raw and aggregated metric points are of the same json schema but are pushed to different kafka topics\n\n## Building\n\n#### Prerequisite: \n\n* Make sure you have Java 1.8\n* Make sure you have docker 1.13 or higher\n\n#### Build\n\nYou can choose to build the individual subdirectories if you're working on any specific sub-app but in case you are making changes to the contract \nsuch as span or metric point which would effect multiple modules you should run\n\n```\nmake all\n```\nThis would build all the individual apps and including unit tests, jar + docker image build and run integration tests for haystack-trends.\n\n\n#### Integration Test\n\nIf you are developing and just want to run integration tests \n```\nmake integration_test\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexpediadotcom%2Fhaystack-trends","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexpediadotcom%2Fhaystack-trends","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexpediadotcom%2Fhaystack-trends/lists"}