{"id":27259546,"url":"https://github.com/pnxtech/bunext","last_synced_at":"2025-04-11T04:06:10.456Z","repository":{"id":22425557,"uuid":"25763419","full_name":"pnxtech/bunext","owner":"pnxtech","description":"Bunyan Extractor - extract data from bunyan log files","archived":false,"fork":false,"pushed_at":"2020-10-19T00:31:13.000Z","size":22,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-11T04:06:03.598Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pnxtech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-26T08:21:13.000Z","updated_at":"2020-10-19T00:31:15.000Z","dependencies_parsed_at":"2022-08-20T17:00:57.236Z","dependency_job_id":null,"html_url":"https://github.com/pnxtech/bunext","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pnxtech%2Fbunext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pnxtech%2Fbunext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pnxtech%2Fbunext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pnxtech%2Fbunext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pnxtech","download_url":"https://codeload.github.com/pnxtech/bunext/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248339281,"owners_count":21087215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-11T04:06:09.849Z","updated_at":"2025-04-11T04:06:10.446Z","avatar_url":"https://github.com/pnxtech.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bunext\n\nBunyan is a logging [library](https://github.com/trentm/node-bunyan) for node.js applications which writes log entries in the JSON format.  Bunext (Bunyan Extractor), is a commandline tool for extracting data and working with Bunyan log files and the JSON they contain.\n\n\u003e **Bonus Tip:** bunext output is compatible with the excellent [jq](https://stedolan.github.io/jq/) tool for advanced JSON queries and syntax highlighting. \n\n## Usage\n\nBunext is a commandline tool so you'll use it at your shell prompt. It requires a bunyan log file to operate on. A typical bunyan log file consists of text containing concatinated stringified JSON.\n\n```\n{\"name\":\"siteserver\",\"hostname\":\"phoenix.local\",\"pid\":23798,\"level\":30,\"msg\":\"SiteServer (v.0.2.0) server listening on port 8123\",\"time\":\"2014-10-18T19:37:55.283Z\",\"v\":0}\n{\"name\":\"siteserver\",\"hostname\":\"phoenix.local\",\"pid\":23798,\"level\":30,\"job\":{\"scheduled\":\"monitor_redis:0 * * * *\"},\"msg\":\"\",\"time\":\"2014-10-18T19:37:55.907Z\",\"v\":0}\n{\"name\":\"siteserver\",\"hostname\":\"phoenix.local\",\"pid\":23798,\"level\":30,\"job\":{\"scheduled\":\"node_status:0 * * * *\"},\"msg\":\"\",\"time\":\"2014-10-18T19:37:55.907Z\",\"v\":0}\n{\"name\":\"siteserver\",\"hostname\":\"phoenix.local\",\"pid\":23798,\"level\":30,\"node_status\":{\"architecture\":\"x64\",\"platform\":\"darwin\",\"nodeVersion\":\"v0.10.29\",\"memory\":{\"rss\":52682752,\"heapTotal\":40378624,\"heapUsed\":26613240},\"uptime\":\"2 seconds\"},\"msg\":\"\",\"time\":\"2014-10-18T19:37:55.910Z\",\"v\":0}\n```\n\nGiven a log file called app.log you can invoke bunext to see the file's JSON data. This allows you to view the JSON in a pretty print format.  While easier to read, this format is of limited value when your log file consists of thousands of entries.\n\n\t$ bunext app.log\n\nUsing bunext options you can specify a date range and expression predicates which help you quickly filter and extract the information you're looking for.\n\nYou can also invoke JavaScript expressions against the file's JSON data.\n\n\t$ bunext -d '10/10/14 - 10/30/14' -e 'redis-status.memory \u003e 256000' app.log\n\t\nYou can learn more about those and other options later in this doc, hopefully this gives you a sense of how useful bunext can be.\t\n\n## Install\n\nYou can install bunext from its GitHub source, using using:\n\n\t$ ./install.sh\n\t\nAlternatively, you can install bunext via NPM:\n\n\t$ [sudo] npm install bunext -g\n\nThe first install method above is useful when you want to work with the bunext source code and potentially contribute pull requests.  The second method is useful when you need to quickly install bunext on machines where you need to debug the source of an application problem by reviewing your log files.\n\n## Options\n\nExecuting bunext without option flags or specifying the `-h` or `--help` options you'll see a list of options.  Some options will only work when paired with other options - such cases are highligted in this section.\n\n\n```\n$ bunext\n\n  Usage: bunext [options] bunyan.log\n\n  Options:\n\n    -h, --help                     output usage information\n    -V, --version                  output the version number\n    -a, --array                    Output results in array format\n    -c, --compare [pattern]        Used with -e to filter by sub pattern\n    -d, --dates [start,end]        Specify a date range\n    -e, --expression [expression]  Filter using expression predicate\n    -r, --raw                      Output raw bunyan data\n    -s, --source                   Extract source (JSON) object\n    -t, --timestamp                Show timestamp in results\n    -p, --prettydate               Show pretty timestamp in results\n```\n\n### -V, --version\nOutput bunext version number\n\n\t$ bunext -V\n\t1.0.4\n\n### -a, --array\nOutput data in an array format. This is useful when you want to collect specific data an into an array that you'll use in code.\n\n```shell\n$ bunext -a -e 'request_info.xForwardedFor' test.log\n\n[\n  \"199.116.73.139, \n  \"172.31.8.142\",\n  \"107.178.200.192,\n]\n```\n\nand...\n\n```\n$ bunext -ate 'redis_status.used_memory' temp.log\n\n[\n\"2014-10-25T18:37:01.454Z | 3081232\",\n\"2014-10-25T18:47:31.376Z | 3081232\",\n\"2014-10-25T19:47:20.964Z | 3081232\",\n\"2014-10-25T19:50:18.845Z | 3081232\",\n\"2014-10-25T19:55:16.202Z | 3081232\",\n\"2014-10-25T19:55:43.546Z | 3373328\",\n\"2014-10-25T20:01:47.410Z | 617488\",\n]\n```\n\nYou may need to cleanup the resulting output in order for the output to be useable.  Bunext uses a simple and rather naive method of building an array. For example, if your JSON has double quotes they won't be escaped. User beware.\n\n### -c, --compare\nThe compare flag is used with the expression flag.\nFirst the express identifies a search pattern in the JSON entry.  This is typically a json path which resolves to a string value.  The compare flag takes the output of the expression against the presence of a substring.  The JavaScript indexOf method is used to find a sub string.\n\n```shell\n$ bunext -e \"body.serviceName\" -c \"log\" test.log\n```\n\n### -d, --dates\nFiltering by a date range is one of the most important things you can do when narrowing in on a server error. Monitoring tools will tell you when an issue occurred and you can use that information with bunext.\n\nHere we ask bunex to search the test.log file for entries who's level is set to 60 (Fatal error) between October 10 and October 30th.\n\n```shell\n$ bunext -d '10/10/14 - 10/30/14' -e 'level===60' test.log\n```\n\n### -e, --expression\nThe ability to use JavaScript expressions to query bunyan log data is one of the most useful features of bunext. When combined with a date range you have a powerful query engine in your arsinal.\n\nBunext expects a JavaScript expression that results in a boolean value or a primitive type.\n\n```shell\n$ bunext -e 'level===60' test.log\n```\n\nIn the example above, we used the expression 'level===60'.  This will return true as the log file in question does have entries where level is indeed 60.  However, we're probably not expecting to see the following as output:\n\n\ttrue\n\ttrue\n\ttrue\n\ttrue\n\ttrue\n\ttrue\n\nThe reason is ofcourse that the expression evalutes to true in some cases.  If we want to see the associated JSON object we need to specify the -s, --source option flag.\n\n```shell\n$ bunext -s -e 'level===60' test.log\n```\n\nThen we might see something like this:\n\n```json\n{\n  \"name\": \"siteserver\",\n  \"hostname\": \"ip-172-31-40-196\",\n  \"pid\": 12438,\n  \"level\": 60,\n  \"err\": {\n    \"message\": \"Object true has no method 'then'\",\n    \"name\": \"TypeError\",\n  },\n  \"msg\": \"Object true has no method 'then'\",\n  \"time\": \"2014-10-09T22:33:41.920Z\",\n  \"v\": 0\n}\n```\n\nIn this next example lets consider the following log entry:\n\n```shell\n{\n  \"name\": \"siteserver\",\n  \"hostname\": \"ip-172-31-40-161\",\n  \"pid\": 1677,\n  \"level\": 30,\n  \"event\": \"monitor\",\n  \"redis_status\": {\n    \"redis_version\": \"2.8.4\",\n    \"uptime_in_seconds\": \"857892\",\n    \"uptime_in_days\": \"9\",\n    \"used_memory\": \"3081232\",\n    \"db2\": \"keys=4,expires=0,avg_ttl=0\",\n    \"db3\": \"keys=1,expires=0,avg_ttl=0\"\n  },\n  \"msg\": \"\",\n  \"time\": \"2014-10-25T18:37:01.454Z\",\n  \"v\": 0\n}\n```\n\nLets say we're interested in exporting the memory usage of Redis over a period of time.  The `used_memory` field is actually stored inside of the redis_status object.  To reference it, we simply use an expression of `redis_status.used_memory`\n\n```shell\n$ bunext -e 'redis_status.used_memory' siteserver.log\n\n3081232\n3081232\n3081232\n3081232\n3081232\n3373328\n617488\n```\n\nThat's cool.  However, not that useful in its current form.  Let's use the `-t` option to view the time stamp for each entry:\n\n\t$ bunext -te 'redis_status.used_memory' siteserver.log\n\n\t2014-10-25T18:37:01.454Z | 3081232\n\t2014-10-25T18:47:31.376Z | 3081232\n\t2014-10-25T19:47:20.964Z | 3081232\n\t2014-10-25T19:50:18.845Z | 3081232\n\t2014-10-25T19:55:16.202Z | 3081232\n\t2014-10-25T19:55:43.546Z | 3373328\n\t2014-10-25T20:01:47.410Z | 617488\n\nNow that's better!\n\n\u003e Note that we used `-te` above.  That's just a shorthand for `-t -e`\n\nThis gets better as we use conditional expressions to further filter the output\n\n```shell\n$ bunext -te 'redis_status.used_memory \u003c 3081232' siteserver.log\n\n2014-10-25T20:01:47.410Z | 617488\n```\n\n### -r, --raw\nThe -r and --raw flags tell bunext to output raw bunyan entries.  This is useful in a number of use cases, such as performing multi-staged queries and concatinating entries from multiple servers prior to further querying the resulting data.\n\nConsider this sample bunyan entry:\n\n\n\u003e {\"name\":\"siteserver\",\"hostname\":\"ip-172-31-40-161\",\"pid\":1811,\"level\":30,\"event\":\"monitor\",\"node_status\":{\"architecture\":\"x64\",\"platform\":\"linux\",\"nodeVersion\":\"v0.10.25\",\"memory\":{\"rss\":46936064,\"heapTotal\":54096128,\"heapUsed\":20113944},\"uptime\":\"1 minute, 30.171894613999932 seconds\"},\"msg\":\"\",\"time\":\"2014-10-25T18:49:00.346Z\",\"v\":0}\n\nLet's format it a bit to have an easier look:\n\n```json\n{\n  \"name\": \"siteserver\",\n  \"hostname\": \"ip-172-31-40-161\",\n  \"pid\": 1811,\n  \"level\": 30,\n  \"event\": \"monitor\",\n  \"node_status\": {\n    \"architecture\": \"x64\",\n    \"platform\": \"linux\",\n    \"nodeVersion\": \"v0.10.25\",\n    \"memory\": {\n      \"rss\": 46936064,\n      \"heapTotal\": 54096128,\n      \"heapUsed\": 20113944\n    },\n    \"uptime\": \"1 minute, 30.171894613999932 seconds\"\n  },\n  \"msg\": \"\",\n  \"time\": \"2014-10-25T18:49:00.346Z\",\n  \"v\": 0\n}\n```\n\nHere we're interested in extracting memory usage for a node server. First we want to filter all JSON entries where `event===monitor` then where `node_status.memory.heapUsed` is greater than some amount of bytes.\n\nUsing the raw flag we can extract all of the raw bunyan JSON entries which contain an event of monitor and save that in a temp file.\n\n```shell\n$ bunext -re 'event===\"monitor\"' siteserver.log \u003e temp.log\n```\n\nThen we can operate on the temp file to extract the heapUsed data.\n\n```shell\n$ bunext -te 'node_status.memory.heapUsed \u003e 20113944' temp.log\n\n2014-10-25T19:52:00.819Z | 20140080\n2014-10-25T19:53:00.818Z | 20185648\n2014-10-25T19:54:00.818Z | 20220424\n2014-10-25T19:55:00.819Z | 20253272\n2014-10-25T20:04:00.370Z | 20152888\n2014-10-25T20:05:00.383Z | 20187744\n2014-10-25T20:06:00.387Z | 20220672\n2014-10-25T20:07:00.387Z | 20254424\n```\n\n### -s, --source\nThe source option flag instructs bunext to return the source JSON data for a matching query. Unlike the raw option the source JSON is extracted and formatted for readability.\n\n\t$ bunext -s -e 'level===60' test.log\n\n### -t, --timestamp\nAll bunyan log entries include a time field in each JSON entry. That field indicates when bunyan wrote the log entry. Because the time field is guarenteed to exists we can request it to be included in our queries.\n\n\t$ bunext -t -d '10/10/14 - 10/30/14' -e 'redis-status.memory \u003e 256000' app.log\n\n### -p, --prettydate\nThe time entry in our last section can be a bit difficult to read quickly.  Using the prettydate format you can request a more human readable format.\n\n\t$ bunext -p -d '10/10/14 - 10/30/14' -e 'redis-status.memory \u003e 256000' app.log\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpnxtech%2Fbunext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpnxtech%2Fbunext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpnxtech%2Fbunext/lists"}