{"id":21862891,"url":"https://github.com/myntra/cortex","last_synced_at":"2025-04-14T19:42:56.973Z","repository":{"id":57514773,"uuid":"140840282","full_name":"myntra/cortex","owner":"myntra","description":"A fault-tolerant events/alerts correlation engine","archived":false,"fork":false,"pushed_at":"2019-04-09T07:51:16.000Z","size":4973,"stargazers_count":25,"open_issues_count":29,"forks_count":14,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-28T08:04:16.372Z","etag":null,"topics":["alerting","alerts","alerts-correlation-engine","boltdb","cloudevents","correlation","event-gateway","events","events-correlation-engine","faas","fault-tolerance","incident-classification","incidents","metrics","raft","rule-engine","rules","rules-engine"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/myntra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-13T11:43:12.000Z","updated_at":"2024-08-17T13:50:12.000Z","dependencies_parsed_at":"2022-09-26T18:00:45.301Z","dependency_job_id":null,"html_url":"https://github.com/myntra/cortex","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fcortex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fcortex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fcortex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myntra%2Fcortex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/myntra","download_url":"https://codeload.github.com/myntra/cortex/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248949176,"owners_count":21188022,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alerting","alerts","alerts-correlation-engine","boltdb","cloudevents","correlation","event-gateway","events","events-correlation-engine","faas","fault-tolerance","incident-classification","incidents","metrics","raft","rule-engine","rules","rules-engine"],"created_at":"2024-11-28T03:17:56.602Z","updated_at":"2025-04-14T19:42:56.942Z","avatar_url":"https://github.com/myntra.png","language":"Go","readme":"\n**Cortex** is a fault-tolerant events correlation engine. It groups and correlates incoming events for further actions:\ncreating/resolving incidents/alerts or for doing root cause analysis.\n\n- Built-in regex matcher for capturing events into groups(*here called as a bucket*). \n- Built-in ES6 javascript interpreter(https://docs.k6.io/docs/modules) for executing correlation logic on buckets.\n- React UI for creating new rules, correlation scripts, list of rule execution history and a playground to simulate correlation executions.\n- REST API crud for rules, scripts and execution history.\n- Cloudevents input and output(https://cloudevents.io/).\n- Fault Tolerance built on top of https://github.com/hashicorp/raft and https://github.com/boltdb/bolt .\n- Single fat self-supervising binary using https://github.com/crawshaw/littleboss .\n- MessagePack encoding/decoding for raft entries using https://github.com/tinylib/msgp .\n\nThe project is **alpha** quality and not yet ready for production.\n\n## Summary: \n\n*Find relationship between N events received at M different points in time using regex matchers and javascript*\n\nTo know more about event correlation in general, please read: https://en.wikipedia.org/wiki/Event_correlation\n\n### Similar Commercial Products\n\n1. https://console.bluemix.net/catalog/services/event-management\n2. https://www.bigpanda.io/blog/algorithmic-alert-correlation/\n3. https://docs.servicenow.com/bundle/kingston-it-operations-management/page/product/event-management/concept/c_EMEventCorrelationRules.html\n\n\n## Use Cases\n- Alerts/Events Correlation\n- Event Gateway\n- FAAS\n- Incidents Management\n\n## How it works:\n\nCortex runs the following steps to achieve event corrrelation:\n\n1. **Match** : incoming alert --\u003e (convert from site 24x7/icinga ) --\u003e (match rule) --\u003e **Collect**\n2. **Collect** --\u003e (add to the rule bucket which *dwells* around until the configured time) --\u003e  **Execute**\n3. **Execute** --\u003e (flush after Dwell period) --\u003e (execute configured script) --\u003e *Post*\n4. **Post** --\u003e (if result is set from script, post the result to the HookEndPoint or post the bucket itself if result is nil)\n\n\n## Screenshots\n\n![Rules](https://cdn.rawgit.com/myntra/cortex/master/assets/cortex_rules.png)\n\n![Scripts](https://cdn.rawgit.com/myntra/cortex/master/assets/cortex_scripts.png)\n\n![Playground](https://cdn.rawgit.com/myntra/cortex/master/assets/cortex_playground.png)\n\n## Rules\n\nA rule contains an array of patterns used to capture events in a *bucket*\n\n```json\n{\n\t\"title\": \"a test rule\",\n\t\"id\": \"test-rule-id-1\",\n\t\"eventTypePatterns\": [\"acme.prod.icinga.check_disk\", \"acme.prod.site247.*\"],\n\t\"scriptID\": \"myscript.js\",\n\t\"dwell\": 4000,\n\t\"dwellDeadline\": 3800,\n\t\"maxDwell\": 8000,\n\t\"hookEndpoint\": \"http://localhost:3000/testrule\",\n\t\"hookRetry\": 2\n}\n```\n\nwhere:\n\n*EventTypePatterns* is the pattern of events to be collected in a bucket.\n\n*Dwell* is the wait duration since the first matched event.\n\n\nPossible patterns:\n\n```\n\t{rule pattern, incoming event type, expected match}\n\t{\"acme*\", \"acme\", false},\n\t{\"acme*\", \"acme.prod\", true},\n\t{\"acme.prod*\", \"acme.prod.search\", true},\n\t{\"acme.prod*.checkout\", \"acme.prod.search\", false},\n\t{\"acme.prod*.*\", \"acme.prod.search\", false},\n\t{\"acme.prod*.*\", \"acme.prod-1.search\", true},\n\t{\"acme.prod.*.*.*\", \"acme.prod.search.node1.check_disk\", true},\n\t{\"acme.prod.*.*.check_disk\", \"acme.prod.search.node1.check_disk\", true},\n\t{\"acme.prod.*.*.check_loadavg\", \"acme.prod.search.node1.check_disk\", false},\n\t{\"*.prod.*.*.check_loadavg\", \"acme.prod.search.node1.check_loadavg\", true},\n\t{\"acme.prod.*\", \"acme.prod.search.node1.check_disk\", true},\n\t{\"acme.prod.search.node*.check_disk\", \"acme.prod.search.node1.check_disk\", true},\n\t{\"acme.prod.search.node*.*\", \"acme.prod.search.node1.check_disk\", true},\n\t{\"acme.prod.search.dc1-node*.*\", \"acme.prod.search.node1.check_disk\", false},\n```\n\n## Events \n\nAlerts are accepted as a cloudevents.io event(https://github.com/cloudevents/spec/blob/master/json-format.md). Site 24x7 and Icinga integration sinks are also provided.\n\nThe engine collects similar events in a bucket over a time window using a regex matcher and then executes a JS(ES6) script. The script contains the correlation logic which can further create incidents or alerts. The JS environment is limited and is achieved by embedding k6.io javascript interpreter(https://docs.k6.io/docs/modules). This is an excellent library built on top of https://github.com/dop251/goja\n\n\nFor the above example rule, incoming events with `eventType` matching one of `eventTypePatterns` will be put in the same bucket:\n\n```json\n{\n\t\"rule\": {},\n\t\"events\": [{\n\t\t\"cloudEventsVersion\": \"0.1\",\n\t\t\"eventType\": \"acme.prod.site247.search_down\",\n\t\t\"source\": \"site247\",\n\t\t\"eventID\": \"C234-1234-1234\",\n\t\t\"eventTime\": \"2018-04-05T17:31:00Z\",\n\t\t\"extensions\": {\n\t\t\t\"comExampleExtension\": \"value\"\n\t\t},\n\t\t\"contentType\": \"application/json\",\n\t\t\"data\": {\n\t\t\t\"appinfoA\": \"abc\",\n\t\t\t\"appinfoB\": 123,\n\t\t\t\"appinfoC\": true\n\t\t}\n\t}]\n}\n```\n\n## Scripts\n\nAfter the `dwell` period, the configured `myscript.js` will be invoked and the bucket will be passed along:\n\n```js\nimport http from \"k6/http\";\n// result is a special variable\nlet result = null\n// the entry function called by default\nexport default function(bucket) {\n    bucket.events.foreach((event) =\u003e {\n        // create incident or alert or do nothing\n        http.Post(\"http://acme.com/incident\")\n        // if result is set. it will picked up the engine    and posted to hookEndPoint\n    })\n}`\n```\n\nIf `result` is set, it will be posted to the hookEndPoint. The `bucket` itself will be reset and evicted from the `collect` loop. The execution `record` will then be stored and can be fetched later.\n\nA new `bucket` will be created when an event matches the rule again.\n\n## Hooks\n\nRule results can be posted to a configured http endpoint. The remote endpoint should be able to accept a `POST : application/json` request.\n\n```\n\"hookEndpoint\": \"http://localhost:3000/testrule\",\n\"hookRetry\": 2\n```\n\n\n## Local Deployment\n\n1. git clone https://github.com/myntra/cortex\n2. ./release.sh\n\nStarts a single node server.\n\n## Production Deployment\n\nTODO\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyntra%2Fcortex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmyntra%2Fcortex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyntra%2Fcortex/lists"}