{"id":15607931,"url":"https://github.com/qxip/queryda","last_synced_at":"2025-10-12T22:13:14.957Z","repository":{"id":57221402,"uuid":"45381465","full_name":"QXIP/queryda","owner":"QXIP","description":"Lightweight, Extensible Query Watcher for Elasticsearch / Elassandra / Cassandra","archived":false,"fork":false,"pushed_at":"2018-05-11T16:00:41.000Z","size":68,"stargazers_count":12,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-07T07:48:52.841Z","etag":null,"topics":["alarm","alert","alerts","anomalies","cassandra","commandline","cql","elassandra","elasticsearch","jexl","notification","query","range","series","tolerance","watcher"],"latest_commit_sha":null,"homepage":"http://qxip.net","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/QXIP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-02T08:27:16.000Z","updated_at":"2020-03-25T13:29:50.000Z","dependencies_parsed_at":"2022-08-29T01:51:04.293Z","dependency_job_id":null,"html_url":"https://github.com/QXIP/queryda","commit_stats":null,"previous_names":["lmangani/elasticwatch-js"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QXIP%2Fqueryda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QXIP%2Fqueryda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QXIP%2Fqueryda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/QXIP%2Fqueryda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/QXIP","download_url":"https://codeload.github.com/QXIP/queryda/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242458420,"owners_count":20131536,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alarm","alert","alerts","anomalies","cassandra","commandline","cql","elassandra","elasticsearch","jexl","notification","query","range","series","tolerance","watcher"],"created_at":"2024-10-03T05:05:15.842Z","updated_at":"2025-10-12T22:13:09.918Z","avatar_url":"https://github.com/QXIP.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg width=200 alt=\"queryda\" src=\"https://user-images.githubusercontent.com/1423657/34468954-46e1d2fe-ef14-11e7-9b1f-279caaaa5b02.png\"/\u003e\n\n\u003cimg width=150 src=\"https://user-images.githubusercontent.com/1423657/34464911-076e9432-ee96-11e7-8c0c-0d893eef2429.png\"/\u003e\n\n\n**Queryda** loves your data as much as you do! Designed for lightweight, intelligent alarming, it will execute and _\"watch\"_ **Elasticsearch** and **Cassandra** queries via ```workers```, comparing their results to one or more given expectations via a pipeline of ```validators```. When query results does not match the expectations, a ```reporter``` is notified and can perform any kind of action _(e.g. heat up the coffeemaker via IFTTT before sending an email to your dev team)_.\n\nFor a natively ELK/Elassandra UI integrated and advanced alerting plarform, check out our Kibana App [SENTINL](https://github.com/sirensolutions/sentinl)\n\n## Getting started\n\n#### npm\nInstall globally using npm\n```\nnpm install -g queryda\n```\n#### Manual\nor clone the git repository and install the dependencies locally\n```\ngit clone https://github.com/lmangani/queryda.git\ncd queryda\nnpm install\n```\n\n\n## Usage\n\n### Quick Examples\n##### Elasticsearch\nTo see an Elasticsearch example, see [examples/elasticsearch](https://github.com/QXIP/queryda/blob/master/examples/elasticsearch.md)\n\n##### Cassandra\nTo see a Cassandra example, see [examples/cassandra](https://github.com/QXIP/queryda/blob/master/examples/cassandra.md)\n\n---------\n\n### Set Alert from Command-Line\nLet's run queryda with the following commandline (or using the *example.json* from the `jobs` dir). \n```\nbin/queryda \\\n--elasticsearch='{\"host\":\"localhost\",\"port\":9200,\"index\":\"monitoring\",\"type\":\"rum\"}' \\\n--query='{\"range\":{\"timestamp\":{\"gt\":\"2018-01-01T00:00:01\",\"lt\":\"2018-01-01T23:59:59\"}}}' \\\n--aggs='{}' \\\n--validators='{\"range\":{\"fieldName\":\"renderTime\",\"min\":0,\"max\":500,\"tolerance\":4}}' \\\n--reporters='{\"console\":{}}' --debug --name test\n```\n\n### Set Alert from Config\nqueryda can also be executed using a self-contained configuration file (see [example.json](jobs/example.json))\n```\nbin/queryda --configfile /path/to/watcherjob.json\n```\n\n## Configuration\nqueryda can be configured either via commandline or using a JSON file _(suggested method via `--configfile` parameter)_. Both ways require to specify option groups with individual settings (e.g. for elasticsearch, for the reporters, for the validator, ..). A set of example JSON files for Cassandra and Elasticsearch can be found in the `jobs`dir.\n\nThe following options are currently available:\n\n### *name (required)*\nA name of your choice to identify this job. This will be used by the reporters to identitfy this individual call.\n\n## Elasticsearch \n### *elasticsearch (elasticsearch only, required)*\nSettings for elasticsearch, expects the following madatory fields:\n- *host*: where to find the elasticsearch host\n- *port*: which port elasticsearch is running on\n- *index*: the index name to send youe query to\n- *type*: the document type to query\n\n### *query* (required)\nAn elasticsearch query statement. Refer to the [elasticsearch documentation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current) for details about syntax and features. Should return a result set that contains the supplied *fieldName* to match against.\n\n### *aggs* (elasticsearch only, required)\nAn elasticsearch aggregation statement. Refer to the [elasticsearch documentation](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current) for details about syntax and features. Should return a result set that contains the supplied *aggName* to match against.\n\n## Cassandra\n### *cassandra (cassandra only, required)*\nSettings for elasticsearch, expects the following madatory fields:\n- *host*: where to find the cassandra host\n- *keyspace*: which to use for queries\n\n### *cqlquery* (required)\nA Cassandra query statement. Refer to the [cassandra documentation](http://cassandra.apache.org/doc/latest/cql/) for details about syntax and features. Should return a result set that contains the supplied *fieldName* to match against.\n\n### *params* (cassandra only, required)\nA set of Parameters/Variable to be used by the CQL Query.\n\n### *validators (required)*\nValidator(s) to compare the query results against. Expects an object with key/value pairs where *key* ist the name of the validator and *value* is the validator-specific configuration. See [Validators](#validators) for more details.\n\n### *reporters (required)*\nReporter(s) to notify about alarms. Expects an object with key/value pairs where *key* ist the name of the reporter and *value* is the reporter-specific configuration. See [Reporters](#reporters) for more details.\n\n### *configfile*\nName of JSON file to read config from. Expects main options as top-level properties (see [example.json](jobs/example.json) for a live example).\n\n## Validators\nA Validator takes a query result received from elasticsearch and compares it against a given expectation. This can be as easy as checking if a value equals a given constant or as complex as checking the average of a series of values against an allowed range with an explicit threshold.\n\n### Available Validators\n#### Range\nThe Range Validator checks a given Field for mix/max boundaries with tolerance factor. \n\nExpects the following mandatory fields:\n- *fieldName*: The name of the field in the result set, that is compared against the defined expectation.\n- *min*: The minimum allowed value for all values within the query. If a series of values (as defined through the *tolerance* property) in the result is lower than this minimum an alarm is raised and reported.\n- *max*: The maxmimum allowed value for all values within the query. If a series of values (as defined through the *tolerance* property) in the result exceed this maximum an alarm is raised and reported.\n- *tolerance*: If a queried series of values exceeds either *min* or *max* for *tolerance*+1 times an alarm is raised.\n\n##### Range Example\n```javascript\n \"validators\": {\n    \"range\": {\n\t    \"fieldName\": \"value\",\n\t    \"min\": 0,\n\t    \"max\": 500,\n\t    \"tolerance\": 4\n    }\n  },\n\n```\n#### Anomalies\nThe Anomalies Validator can determine clusters of data and then also identify values which\ndo not identify with any derived cluster and delcare them outliers.\n\nExpects the following mandatory field:\n- *fieldName*: The name of the field in the result set, that is tested for series anomalies.\n\n##### Anomalies Example\n```javascript\n \"validators\": {\n    \"anomalies\": {\n\t    \"fieldName\": \"value\",\n\t    \"tolerance\": 0\n\n    }\n  },\n\n```\n### Custom validators\nYou can create custom validators by creating a new class that extends the `Validator` class (see [RangeValidator](src/validators/range.js) for an example).\n\n## Reporters\nBy default queryda does nothing more than executing its configured jobs, raising alarms if expectations aren't met. If you want to perform any action in such an alarm case, you have to define a reporter.\n\nTo put it simple - reporters are notified about alarms, which means a configured expectation isn't met for a given number of times. They can then do helpful things depending on their type like sending an email, creating a ticket in your ticket system, etc.\n\nReporters are defined inside a job's config, you can set either one or multiple of them. Most reporters need a specific configuration that is based on the reporter type.\n\n### Available reporters\n\n#### ConsoleReporter\nThe ConsoleReporter is just meant for demonstration purpose and simply logs a message to the console and has no configuration options.\n\n#### MailReporter\nThe MailReporter sends an email to one (or multiple) given e-mail address(es). It offers the following configuration:\n```javascript\n\"reporters\": {\n  \"mail\": {\n    // comma-separated list of target addresses for notification\n    \"targetAddress\": \"me@example.com,peng@example.com\"\n    // number of retry attempts if sending mail fails (defaults to 3)\n    \"maxRetries\": 3\n  }\n}\n```\n\n### Custom reporters\nYou can create custom reporters by creating a new class that extends the `Reporter` class (see [ConsoleReporter](src/reporters/console.js) for an example).\n\n## TODO\nThere's plenty of work to be done in order to make this tool powerful - any help and contribution is appreciated!\n\n* [ ] Cleanup, Revamp output for usefulness.\n* [ ] Port transform, validate, actions from SENTINL\n* [ ] Properly modularize input Workers\n* [x] Implement Cassandra Support\n* [ ] Implement Clickhouse Support\n* [ ] Implement InfluxDB Support\n* [ ] Implement GunDB Support\n  * [ ] Integrate SENTINL Clustering\n* [ ] Implement pseudo-cascading programming as in Kapacitor\n\n## Credits and Acknowledgements\nAll rights reserved by their respective owners.\n\nOriginal Fork based on Elasticwatch-JS by QXIP, and its [Coffeescript](https://github.com/ryx/elasticwatch) parent.\n\nApache Cassandra, Apache Lucene, Apache, Lucene, Solr, TinkerPop, and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.\n\nElasticsearch and Kibana are trademarks of Elasticsearch BV, registered in the U.S. and in other countries.\n\nElassandra is a trademark of Strapdata SAS.\n\nSentinl is a trademark of QXIP BV and Siren Solutions.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqxip%2Fqueryda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqxip%2Fqueryda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqxip%2Fqueryda/lists"}