{"id":20348498,"url":"https://github.com/onebeyond/prepper","last_synced_at":"2025-04-12T01:15:12.902Z","repository":{"id":8438037,"uuid":"58413593","full_name":"onebeyond/prepper","owner":"onebeyond","description":"Prepper is an event based api for pre-processing log events before routing them to your logging framework of choice","archived":false,"fork":false,"pushed_at":"2023-07-20T11:46:02.000Z","size":186,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":17,"default_branch":"master","last_synced_at":"2025-04-12T01:15:06.193Z","etag":null,"topics":["hacktoberfest"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onebeyond.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-09T23:01:32.000Z","updated_at":"2023-01-27T08:13:22.000Z","dependencies_parsed_at":"2024-06-19T05:26:07.453Z","dependency_job_id":"31326481-9a1c-4b13-b0ad-834904ef4682","html_url":"https://github.com/onebeyond/prepper","commit_stats":null,"previous_names":["guidesmiths/prepper"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onebeyond%2Fprepper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onebeyond%2Fprepper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onebeyond%2Fprepper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onebeyond%2Fprepper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onebeyond","download_url":"https://codeload.github.com/onebeyond/prepper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248501861,"owners_count":21114684,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest"],"created_at":"2024-11-14T22:20:52.319Z","updated_at":"2025-04-12T01:15:12.867Z","avatar_url":"https://github.com/onebeyond.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Prepper\nPrepper is an event based api for filtering, decorating and validating log events before routing them to your logging framework of choice. It's especially useful in a micro-service environment where it is otherwise hard to ensure consistent and responsible logging practice.\n\n[![NPM version](https://img.shields.io/npm/v/prepper.svg?style=flat-square)](https://www.npmjs.com/package/prepper)\n[![NPM downloads](https://img.shields.io/npm/dm/prepper.svg?style=flat-square)](https://www.npmjs.com/package/prepper)\n[![Build Status](https://img.shields.io/travis/guidesmiths/prepper/master.svg)](https://travis-ci.org/guidesmiths/prepper)\n[![Code Climate](https://codeclimate.com/github/guidesmiths/prepper/badges/gpa.svg)](https://codeclimate.com/github/guidesmiths/prepper)\n[![Test Coverage](https://codeclimate.com/github/guidesmiths/prepper/badges/coverage.svg)](https://codeclimate.com/github/guidesmiths/prepper/coverage)\n[![Code Style](https://img.shields.io/badge/code%20style-imperative-brightgreen.svg)](https://github.com/guidesmiths/eslint-config-imperative)\n[![Dependency Status](https://david-dm.org/guidesmiths/prepper.svg)](https://david-dm.org/guidesmiths/prepper)\n[![devDependencies Status](https://david-dm.org/guidesmiths/prepper/dev-status.svg)](https://david-dm.org/guidesmiths/prepper?type=dev)\n[![Maintainability](https://api.codeclimate.com/v1/badges/03d3439aa152219a9f36/maintainability)](https://codeclimate.com/github/onebeyond/prepper/maintainability)\n[![Test Coverage](https://api.codeclimate.com/v1/badges/03d3439aa152219a9f36/test_coverage)](https://codeclimate.com/github/onebeyond/prepper/test_coverage)\n\n## tl;dr\n```js\nconst prepper = require('prepper')\nconst pkg = require('./package.json')\nconst os = require('os')\nconst Sequence = prepper.handlers.Sequence\nconst Merge = prepper.handlers.Merge\n\nconst logger = new prepper.Logger([\n    new Merge({ system: { hostname: os.hostname() } }),\n    new Merge({ package: { name: pkg.name } }),\n    new Merge({ process: { pid: process.pid } })\n]).on('message', function(event) {\n    // Replace with your logger of choice\n    console.log(event.level.toUpperCase(), event.system.hostname, event.process.pid, event.package.name, event.message)\n})\n\nlogger.debug('Server started')\n```\nResults in\n```\nDEBUG sophrosyne 8285 prepper Server started\n```\n\n## Why write another logger?\nWe didn't. Prepper is a sequence of event handlers/emitters hidden behind an api that makes it look like a logger. The emitters decorate or filter attributes of the log event as it bubbles to the finally handler, which should invoke the logging framework of your choice.\n\n## Why write an event emitter that looks like a logger, but doesn't log anything?\nWe ran into problems with our [ELK stack](https://www.elastic.co/webinars/introduction-elk-stack). With the default configuration ElasticSearch dynamically creates a schema the first time it encounters a property. This means the first service to execute\n```js\nlog.debug('Bad request', { user: 'cressie176' })\n```\nDefines the schema and any subsequent log statments along the lines of\n```js\nlog.debug('Bad request', { user: { id: 'cressie176' }})\n```\nwill cause ElasticSearch to throw an error and the message to be dropped. When ElasticSearch throws an error it slows down. A noisy service, logging messages in high volumes can cause a delay in logs.\n\nAs second problem we encountered is that developers were careless with what they logged. We've had binary messages, web pages, emails and some very large json documents transmitted to our logging infrastructure, degrading performance, incurring cost and most significantly risking information leak (thankfully none of the content was or a financial or personal nature).\n\nWe felt it infeasible for multiple development teams working on different sets of microservices to keep track of a common logging schema without tooling, and experience has taught us that education isn't sufficient to prevent these problems happening again, so we built a logger that supports a centrally managed, easy to update schema. We could have done this with a static ElasticSearch schema or with logstash groks, but thought this would have been harder to maintain and update. It's also preferable to filter documents prior to transmission.\n\n## What other features does Prepper have?\n\n### A helpful API\nThe api was designed to support common use cases for logging messages and errors with context\n```js\nlogger.log([message], [error], [context])\n```\nFor example:\n```js\n// Emits { message: 'some message', level: 'info' }\nlogger.log('some message')\n\n// Emits { message: 'some message', level: 'info', user: { ... }}\nlogger.log('some message', { user: user })\n\n// Emits { message: 'On Noes', level: 'error', error: { message: 'On Noes', stack: ... }}\nlogger.log(new Error('Oh Noes'))\n\n// Emits { message: 'An error occurred', level: 'error', error: { message: 'On Noes', stack: ... }}\nlogger.log('An error occurred', new Error('Oh Noes'))\n```\n\n### Support for common log levels\nThe supported log levels are trace, debug, info, warn, error and fatal.\n```js\n// Emits { message: 'some message', level: 'trace' }\nlogger.trace('some message')\n\n// Emits { message: 'On Noes', level: 'warn', error: { message: 'On Noes', stack: ... }}\nlogger.warn(new Error('Oh Noes'))\n```\n\n### Custom log levels\nYou can add additional log levels by extending the Logger class, e.g.\n```\nfunction MyLogger() {\n    var self = this\n    this.catastrophe = function(event) {\n        var args = Array.prototype.slice.apply(arguments)\n        self._logEvent.apply(self, args.concat('catastrophe'))\n        return self\n    }\n    Logger.call(this);\n}\nutil.inherits(MyLogger, Logger)\n\nmodule.exports = MyLogger\n```\n### Request scoped loggers\nBecause loggers are event emitters too you can wire them together. By defining an application scoped logger, referenced by ```app.locals.logger``` and a request scoped logger referenced by ```response.locals.logger```, and connecting them together you can automatically decorate log events with request details. See the express tests for an example.\n\n### Thoughtful error treatment\nSometimes you need to yield and error, but don't want it logged as one. If you decorate the error object with a ```level``` attribute set to the desired level and log the error with ```logger.log(err)``` instead of ```logger.error(err)```, prepper will use the specified level rather than the default of error. Any other attributes (e.g. ```code```) added to an error will be included in the event too.\n\n### A few helpful handlers\n\n#### Merge\nMerges the given object into the event. You can optionally supply a key and whether to merge left or right.\n```js\n// Merges package json into the event under the package sub-document, overwriting existing properties\nnew prepper.handlers.Merge(packageJson, { key: 'package', invert: true })\n```\n#### Env\nDecorates the event with all key value pairs from ```process.env``` under the ```env``` sub-document\n```js\nnew prepper.handlers.Env()\n```\n#### Process\nDecorates the event with various process properties (title, version, pid, memory usage, etc) under the ```process``` sub-document\n```js\nnew prepper.handlers.Process()\n```\n#### System\nDecorates the event with various os properties (hostname, load average, memory usage, platform, release, etc) under the ```system``` sub-document\n```js\nnew prepper.handlers.System()\n```\n#### Timestamp\nDecorates the event with a timestamp\n```js\nnew prepper.handlers.Timestamp()\n```\n### Tracer\nDecorates the event with a new uuid or existing tracer if specified. Especially useful for web applications when the logger is scoped to the request\n```js\nnew prepper.handlers.Tracer({ tracer: req.headers.tracer })\n```\n### KeyFilter\nIncludes or excludes properties based on keyname. Useful for removing password fields and limiting output to a fixed schema\n```js\n// Restrict the log even to a reasonable set of values, explicitly excluding potentially sensitve fields\n// and package dependencies that might creep in accidentally because of names like 'leveldb'\nnew prepper.handlers.KeyFilter({\n    include: [\n        'tracer',\n        'timestamp',\n        'level',\n        'message',\n        'error.message',\n        'error.code',\n        'error.stack',\n        'request.url',\n        'request.headers',\n        'request.params',\n        'request.method',\n        'response.statusCode',\n        'response.time',\n        'env.NODE_ENV',\n        'env.USER',\n        'env.PWD',\n        'process',\n        'system',\n        'package.name',\n        'user.id',\n        'user.username'\n    ],\n    exclude: [\n        'password',\n        'secret',\n        'token',\n        'request.headers.cookie',\n        'dependencies',\n        'devDependencies'\n    ]\n})\n```\nThe KeyFilter assumes that you have flattened the event object previously in the sequence, and that you will unflatten it at some point prior to logging. The KeyFilter deliberately doesn't flatten the event object internally as it's likely you might want to filter based on keys or values in another handler. We do provide handlers for Flattening and Unflattening the event, so usage is likely to be:\n```js\nnew prepper.handlers.Sequence([\n    new prepper.handlers.Flatten(),\n    new prepper.handlers.KeyFilter( includes: [...], excludes: [...] ),\n    new prepper.handlers.ValueFilter( excludes: [creditCards, etc] ), // Not implemented yet\n    new prepper.handlers.Unflatten()\n])\n```\n#### Oversized\nDecorates the event with an ```prepper.violation.oversized``` property if the stringified version of the event is larger than the given size. Circular references are ignored, as are toJSON functions and property getters which throw exceptions.\n```js\nnew prepper.handlers.Oversized({ size: 20000 })\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonebeyond%2Fprepper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonebeyond%2Fprepper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonebeyond%2Fprepper/lists"}