{"id":13989884,"url":"https://github.com/codeactual/mainevent","last_synced_at":"2025-07-22T11:31:58.536Z","repository":{"id":2580994,"uuid":"3561885","full_name":"codeactual/mainevent","owner":"codeactual","description":"Log file aggregation, search, dashboards using node.js, backbone.js, socket.io, MongoDB, Redis","archived":true,"fork":false,"pushed_at":"2013-07-07T23:59:58.000Z","size":3400,"stargazers_count":9,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-29T09:39:12.500Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://codeactual.github.io/mainevent/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codeactual.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-02-27T16:03:34.000Z","updated_at":"2023-01-28T18:41:42.000Z","dependencies_parsed_at":"2022-08-28T12:13:09.754Z","dependency_job_id":null,"html_url":"https://github.com/codeactual/mainevent","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/codeactual/mainevent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeactual%2Fmainevent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeactual%2Fmainevent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeactual%2Fmainevent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeactual%2Fmainevent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codeactual","download_url":"https://codeload.github.com/codeactual/mainevent/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeactual%2Fmainevent/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266483815,"owners_count":23936423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T13:02:08.557Z","updated_at":"2025-07-22T11:31:57.134Z","avatar_url":"https://github.com/codeactual.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# [mainevent](http://codeactual.github.com/mainevent)\n\nmainevent provides a suite of tools to gain insight into log files.\n\n* Collect, parse and store updates from local and SSH-accessible log files.\n* Easily write new parser modules for any file format.\n* Searchable timeline with optional real-time updates.\n* Dashboard analytics generated by [MongoDB MapReduce](http://www.mongodb.org/display/DOCS/MapReduce), [Redis](http://redis.io/) and [jqPlot](http://www.jqplot.com/).\n* Write custom [Pub/Sub](http://redis.io/topics/pubsub) listeners to real-time updates.\n* [more ...](http://codeactual.github.com/mainevent)\n\n\u003ca href=\"http://codeactual.github.com/mainevent\"\u003e\u003cimg alt=\"3-screenshot set\" src=\"http://codeactual.github.com/mainevent/img/screenshot-set.png\" /\u003e\u003c/a\u003e\n\n## Use Cases\n\n* Search development/production environment logs which span a variety of formats.\n* Produce dashboard data and graphs specific to your insight needs.\n* Filter and replicate real-time updates into additional systems for specialized processing or alerting.\n\n## How To\n\n### Create a new parser for an unsupported log format\n\nEach parser lives in a separate directory under \u003ccode\u003e[app/parsers/](https://github.com/codeactual/mainevent/tree/master/app/parsers/)\u003c/code\u003e which holds its JS, CSS, templates and tests.\n\nAll parser classes extend a [base](https://github.com/codeactual/mainevent/blob/master/app/parsers/prototype.js) and only need to implement a small number of interfaces.\n\n* **Required**\n  * `parse(log)`: Accepts a log line string, returns an object of parsed fields.\n* Optional\n  * `buildTemplateContext(template, log)`: Modify the context object sent to dust.js based on the type of template. Currently there are two types: `preview` and `event`. See the [extension example](http://codeactual.github.com/mainevent/#extension-example) to see them in action.\n  * `buildPreviewText(log)`: Build the preview string manually (rather than using on a preview template).\n  * `extractTime(log)`: The default implementation will detect millisecond/second timestamps and `Date.parse()`-able strings in `log.time` values. For incompatible formats, define this function to extract the millisecond timestamp manually.\n* Utilities\n  * `namedCapture(subject, regex)`: Wrapper around [XRegExp](https://github.com/slevithan/XRegExp) named capture expression handling.\n  * `candidateCapture(subject, candidates)`: Wrapper around `namedCapture` that lets you define multiple potential patterns and the first match wins.\n\nSee \u003ccode\u003e[app/parsers/prototype.js](https://github.com/codeactual/mainevent/blob/master/app/parsers/prototype.js)\u003c/code\u003e for more interface details.\n\nExtending the base class is a simple one-call process via a [backbone.js](http://documentcloud.github.com/backbone/)-like `extend()` function. See the [extension example](http://codeactual.github.com/mainevent/#extension-example) for a working implementation and screenshots of the output content. Or browse any of the modules under \u003ccode\u003e[app/parsers/](https://github.com/codeactual/mainevent/tree/master/app/parsers/)\u003c/code\u003e.\n\n### Create a [Pub/Sub](http://redis.io/topics/pubsub) listener for log updates\n\n1. Create a module that exports an `on(logs)` function that receives an array of one or more log objects.\n\n2. Perform any non-native tasks you need.\n\n3. Find this \u003ccode\u003e[config/app.js](https://github.com/codeactual/mainevent/blob/master/config/app.js.dist)\u003c/code\u003e section and add the location of your listener module to the `subscribers` list:\n\n```javascript\n{\n  // ...\n  mongodb: {\n    // ...\n    listeners: [\n      {\n        // Customize this event but do not remove.\n        event: 'InsertLog',\n        enabled: true,\n        subscribers: [\n          'app/modules/redis/InsertLogPubSub.js'\n        ],\n      }\n    ],\n    // ...\n  },\n  // ...\n}\n```\n\nSee \u003ccode\u003e[app/modules/redis/InsertLogPubSub.js](https://github.com/codeactual/mainevent/blob/master/app/modules/redis/InsertLogPubSub.js)\u003c/code\u003e for a working example.\n\n### Display a real-time update with Web Notifications\n\n* Enable Desktop Notifications via Timeline drop-down menu.\n* Required event attribute(s):\n  * `WebNotifyApiBody`\n    * OR `WebNotifyApiBodyAttr` to specify the attribute to use instead.\n* Optional event attribute(s):\n  * `WebNotifyApiTitle`\n    * OR `WebNotifyApiTitleAttr` to specify the attribute to use instead.\n  * `WebNotifyApiUrl` (icon location)\n    * OR `WebNotifyApiUrlAttr` to specify the attribute to use instead.\n\n## Configuration\n\n`$ cp config/app.js.dist config/app.js`\n\nNotes about the main properties:\n\n* `sources`\n  * `path`: Absolute path to the log file.\n  * `parser`: Parser class/class-file name, e.g. `Json` or `NginxAccess`.\n  * `tags`: (Optional) One or more tags to automatically attach to every event.\n  * `timeAttr`: (Optional) By default, mainevent expects parsers to return a `time` property to represent the event's timestamp. Select a different property name here.\n  * To specify a remote host (all are required):\n     * `sshKey`: Ex. /path/to/my/ec2.pem\n     * `sshPort`: Ex. 22\n     * `sshUser`: Ex. ubuntu\n     * `sshHost`: Ex. ec2-135-28-52-91.compute-1.amazonaws.com\n  * `previewAttr`: (Optional) Allows parsers like `Json`, which do not have preview templates, to know which properties should be included in preview text.\n* `express`\n  * `mainevent_server`: Verify the default `port`/`ip`.\n  * `test_server`: Verify the default `port`/`ip`.\n* `mongodb`\n  * Customize the collection name in `collections.event`.\n  * Select a different pagination maximum in `maxResultSize` if needed.\n  * Add additional indexes if needed. Future versions may automate that process based on metrics.\n* `redis`\n  * `host`/`port`/`options`: Passed to `createClient()` in [node_redis](https://github.com/mranney/node_redis).\n\n## Main Components\n\n### Frontend\n\n#### [mainevent_server.js](https://github.com/codeactual/mainevent/blob/master/bin/mainevent_server.js)\n\n`$ bin/mainevent_server.js`\n\nHas three responsibilities:\n\n1. `/` serves the [backbone.js](http://documentcloud.github.com/backbone/) MVC app via [express.js](http://expressjs.com/).\n1. `/api` serves JSON data including graph points, event objects and timeline pages via [express.js](http://expressjs.com/).\n1. `/socket.io` serves real-time timelime updates.\n\nTriggers \u003ccode\u003e[public/build.js](https://github.com/codeactual/mainevent/blob/master/public/build.js)\u003c/code\u003e on startup to build the `static/` directory.\n\n#### [public/build.js](https://github.com/codeactual/mainevent/blob/master/public/build.js)\n\n`$ public/build.js [--prod]`\n\n* Combines and compresses (`--prod`) JS/CSS files located in \u003ccode\u003e[public/](https://github.com/codeactual/mainevent/blob/master/public/)\u003c/code\u003e.\n* Relies on \u003ccode\u003e[public/js/app.build.js](https://github.com/codeactual/mainevent/blob/master/public/js/app.build.js)\u003c/code\u003e for [RequireJS](http://requirejs.org/) configuration.\n* Triggers \u003ccode\u003e[public/js/templates.build.js](https://github.com/codeactual/mainevent/blob/master/public/js/templates.build.js)\u003c/code\u003e to compile [dust.js](http://akdubya.github.com/dustjs/) templates.\n\nOutputs all files into `static/`.\n\n#### [public/js/templates.build.js](https://github.com/codeactual/mainevent/blob/master/public/js/templates.build.js)\n\nCompiles [dust.js](http://akdubya.github.com/dustjs/) templates in \u003ccode\u003e[app/views/](https://github.com/codeactual/mainevent/blob/master/app/views/)\u003c/code\u003e and \u003ccode\u003e[app/parsers/*/templates/](https://github.com/codeactual/mainevent/blob/master/app/parsers/)\u003c/code\u003e.\n\n#### [public/js/app.build.js](https://github.com/codeactual/mainevent/blob/master/public/js/app.build.js)\n\n[RequireJS](http://requirejs.org/) configuration for client-side dependencies.\n\n### Background\n\n#### [bin/tail.js](https://github.com/codeactual/mainevent/blob/master/bin/tail.js)\n\n`$ bin/tail.js`\n`$ bin/tail.js --help`\n\nSpawns `tail` instances for each source described in \u003ccode\u003e[config/app.js](https://github.com/codeactual/mainevent/blob/master/config/app.js.dist)\u003c/code\u003e.\n\n#### [app/graphs/CountAllPartitioned.js](https://github.com/codeactual/mainevent/blob/master/app/graphs/CountAllPartitioned.js)\n\n**Required by the dashboard.**\n\n`$ app/graphs/CountAllPartitioned.js --verbose --jobWait 2 --chunkWait 2 --limit 1000`\n`$ app/graphs/CountAllPartitioned.js --help`\n\nGenerates the cached data used by the graph on `/dashboard` visualizing total events. Runs continually and sleeps for 1 minute if no new events are read.\n\n### Utilities\n\n#### [bin/import.js](https://github.com/codeactual/mainevent/blob/master/bin/import.js)\n\n`$ bin/import.js --parser json --path /var/log/myApp/prod.json --tags myApp,import`\n`$ bin/import.js --help`\n\nLike \u003ccode\u003e[tail.js](https://github.com/codeactual/mainevent/blob/master/bin/tail.js)\u003c/code\u003e except it processes the entire file. (The file does not need to be described in \u003ccode\u003e[config/app.js](https://github.com/codeactual/mainevent/blob/master/config/app.js.dist)\u003c/code\u003e.)\n\n## File Layout Notes\n\n* \u003ccode\u003e[app/](https://github.com/codeactual/mainevent/blob/master/app/)\u003c/code\u003e : Holds most server-side modules and classes.\n  * \u003ccode\u003e[controllers/](https://github.com/codeactual/mainevent/blob/master/app/controllers/)\u003c/code\u003e: Handlers for [express.js](http://expressjs.com/) routes defined in \u003ccode\u003e[bin/mainevent_server.js](https://github.com/codeactual/mainevent/blob/master/bin/mainevent_server.js)\u003c/code\u003e.\n  * \u003ccode\u003e[graphs/](https://github.com/codeactual/mainevent/blob/master/app/graphs/)\u003c/code\u003e: Background scripts which run at intervals to cache point data in Redis.\n  * \u003ccode\u003e[jobs/](https://github.com/codeactual/mainevent/blob/master/app/jobs/)\u003c/code\u003e: Classes used by `graphs` scripts which define the [MapReduce](http://www.mongodb.org/display/DOCS/MapReduce) logic.\n  * \u003ccode\u003e[modules/](https://github.com/codeactual/mainevent/blob/master/app/modules/)\u003c/code\u003e: Covers [MongoDB](http://www.mongodb.org/), [Redis](http://redis.io/), static builds and global objects like `mainevent`.\n  * \u003ccode\u003e[parsers/](https://github.com/codeactual/mainevent/blob/master/app/parsers/)\u003c/code\u003e: Self-contained parser modules, their prototype, and a test utility module.\n  * \u003ccode\u003e[sockets/](https://github.com/codeactual/mainevent/blob/master/app/sockets/)\u003c/code\u003e: Like `controllers` except for socket messages rather than routes.\n  * \u003ccode\u003e[views/](https://github.com/codeactual/mainevent/blob/master/app/views/)\u003c/code\u003e: All non-parser [dust.js](http://akdubya.github.com/dustjs/) templates.\n* \u003ccode\u003e[bin/](https://github.com/codeactual/mainevent/blob/master/bin/)\u003c/code\u003e: All HTTP servers and background processes like \u003ccode\u003e[tail.js](https://github.com/codeactual/mainevent/blob/master/bin/tail.js)\u003c/code\u003e.\n* \u003ccode\u003e[public/](https://github.com/codeactual/mainevent/blob/master/public/)\u003c/code\u003e\n  * \u003ccode\u003e[js/](https://github.com/codeactual/mainevent/blob/master/public/js/)\u003c/code\u003e\n    * \u003ccode\u003e[backbone/](https://github.com/codeactual/mainevent/blob/master/public/js/backbone/)\u003c/code\u003e: Additions to [backbone.js](http://documentcloud.github.com/backbone/) prototypes like `Backbone.View.prototype`.\n    * \u003ccode\u003e[collections/](https://github.com/codeactual/mainevent/blob/master/public/js/collections/)\u003c/code\u003e: [backbone.js](http://documentcloud.github.com/backbone/#Collection) collections.\n    * \u003ccode\u003e[controllers/](https://github.com/codeactual/mainevent/blob/master/public/js/controllers/)\u003c/code\u003e: Handlers for [backbone.js](http://documentcloud.github.com/backbone/#Router-route) routes.\n    * \u003ccode\u003e[helpers/](https://github.com/codeactual/mainevent/blob/master/public/js/helpers/)\u003c/code\u003e: Ex. `mainevent.helpers.Socket` for creating new [socket.io](http://socket.io/) connections.\n    * \u003ccode\u003e[models/](https://github.com/codeactual/mainevent/blob/master/public/js/models/)\u003c/code\u003e: [backbone.js](http://documentcloud.github.com/backbone/#Model) models.\n    * \u003ccode\u003e[observers/](https://github.com/codeactual/mainevent/blob/master/public/js/observers/)\u003c/code\u003e: Global listeners of custom events like `ContentPreRender`.\n    * \u003ccode\u003e[shared/](https://github.com/codeactual/mainevent/blob/master/public/js/shared/)\u003c/code\u003e: Modules/classes available server-side and client-side, ex. `mainevent.shared.Date`.\n    * \u003ccode\u003e[views/](https://github.com/codeactual/mainevent/blob/master/public/js/views/)\u003c/code\u003e: [backbone.js](http://documentcloud.github.com/backbone/#View) views.\n* \u003ccode\u003estatic/\u003c/code\u003e: JS/CSS/images from \u003ccode\u003e[public/](https://github.com/codeactual/mainevent/blob/master/public/)\u003c/code\u003e processed by \u003ccode\u003e[public/build.js](https://github.com/codeactual/mainevent/blob/master/public/build.js)\u003c/code\u003e.\n* \u003ccode\u003e[test/](https://github.com/codeactual/mainevent/blob/master/test/)\u003c/code\u003e\n  * \u003ccode\u003e[browser/](https://github.com/codeactual/mainevent/blob/master/test/browser/)\u003c/code\u003e: Client-side tests processed by \u003ccode\u003e[app/views/test.html](https://github.com/codeactual/mainevent/blob/master/app/views/test.html)\u003c/code\u003e.\n  * \u003ccode\u003e[modules/](https://github.com/codeactual/mainevent/blob/master/test/modules/)\u003c/code\u003e: Test helpers.\n\n## Testing\n\nServer-side tests rely on [nodeunit](https://github.com/caolan/nodeunit). Example:\n\n`$ nodeunit test/redis.js`\n\nRun all tests found under \u003ccode\u003e[app/parsers/](https://github.com/codeactual/mainevent/blob/master/app/parsers/)\u003c/code\u003e and \u003ccode\u003e[test/](https://github.com/codeactual/mainevent/blob/master/test/)\u003c/code\u003e.\n\n`$ npm test`\n\n### Remote Logs\n\n`$ cp test/fixtures/tail-config.js test/fixtures/tail-config-remote.js`\n\nUpdate `ssh*` configuration values in `test/fixtures/tail-config-remote.js`.\n\n## Events\n\n### Server-side\n\n* `InsertLog`\n  * Triggered in \u003ccode\u003e[app/modules/mongodb.js](https://github.com/codeactual/mainevent/blob/master/app/modules/mongodb.js)\u003c/code\u003e in `insertLog()`.\n  * Callbacks receive the array of document objects.\n  * [Example listener](https://github.com/codeactual/mainevent/blob/master/app/modules/redis/InsertLogPubSub.js).\n\n### Client-side\n\n* `LinkClick`\n  * Triggered in \u003ccode\u003e[public/js/helpers/Event.js](https://github.com/codeactual/mainevent/blob/master/public/js/helpers/Event.js)\u003c/code\u003e on any link with a relative `href`.\n* `CritFetchError`\n  * Triggered when a view cannot fetch data critical to its presentation, ex. the event view cannot retrieve the object describing the event.\n  * Callbacks receive the `response` object from \u003ccode\u003e[$.ajax](http://api.jquery.com/jQuery.ajax/)\u003c/code\u003e.\n* `ContentPreRender`\n  * Triggered in \u003ccode\u003e[public/js/index.js](https://github.com/codeactual/mainevent/blob/master/public/js/index.js)\u003c/code\u003e before the \u003ccode\u003e[content.html](https://github.com/codeactual/mainevent/blob/master/app/views/content.html)\u003c/code\u003e template is rendered.\n\n### Bundled dependencies and their licenses\n\n* [backbone.js](http://documentcloud.github.com/backbone/) (MIT)\n* [clientsiiide](https://github.com/codeactual/clientsiiide) (MIT)\n* [dust.js](http://akdubya.github.com/dustjs/) (MIT)\n* [Glyphicons Free](http://glyphicons.com/) (CC BY 3.0)\n* [jqPlot](http://www.jqplot.com/) (MIT/GPLv2)\n* [jQuery](http://jquery.com/) (MIT/GPLv2)\n* [jQuery UI](http://jqueryui.com/) (MIT/GPLv2)\n  * Core, Widget, Mouse, Datepicker, Slider\n* [jQuery-Timepicker-Addon](https://github.com/trentrichardson/jQuery-Timepicker-Addon) (MIT/GPLv2)\n* [moment.js](http://momentjs.com/) (MIT)\n* [RequireJS](http://requirejs.org/) (MIT/New BSD)\n* [socket.io](http://socket.io/) (MIT)\n* [Twitter Bootstrap](http://twitter.github.com/bootstrap/index.html) (Apachev2)\n* [underscore.js](http://documentcloud.github.com/underscore/) (MIT)\n* [XRegExp](https://github.com/slevithan/XRegExp) (MIT)\n\n### Copyright and license (MIT)\n\nCopyright (c) 2012 David Smith, codeactual@gmail.com\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is furnished\nto do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeactual%2Fmainevent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodeactual%2Fmainevent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeactual%2Fmainevent/lists"}