{"id":24506181,"url":"https://github.com/pvdb/ntail","last_synced_at":"2025-04-14T07:09:10.995Z","repository":{"id":1238825,"uuid":"1177084","full_name":"pvdb/ntail","owner":"pvdb","description":"A tail(1)-like utility for nginx log files that supports parsing, filtering and formatting individual log lines.","archived":false,"fork":false,"pushed_at":"2014-03-14T10:04:38.000Z","size":456,"stargazers_count":39,"open_issues_count":2,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-14T07:08:46.692Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pvdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-12-17T12:44:56.000Z","updated_at":"2024-02-11T11:16:19.000Z","dependencies_parsed_at":"2022-08-16T12:40:37.471Z","dependency_job_id":null,"html_url":"https://github.com/pvdb/ntail","commit_stats":null,"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdb%2Fntail","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdb%2Fntail/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdb%2Fntail/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdb%2Fntail/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pvdb","download_url":"https://codeload.github.com/pvdb/ntail/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248837280,"owners_count":21169374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-21T23:33:25.809Z","updated_at":"2025-04-14T07:09:10.964Z","avatar_url":"https://github.com/pvdb.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"ntail\n=====\n\nA `tail(1)`-like utility for nginx log files that supports parsing, filtering and formatting of individual\nlog lines (in nginx's so-called [\"combined\" log format](http://wiki.nginx.org/NginxHttpLogModule#log_format)).\n\n\u003ca name=\"intro\"/\u003e\n\nCheck it out, yo!\n-----------------\n\nInstead of this...\n\n\u003cpre style=\"background-color: black; color: white; padding: 15px; width: 1100px; overflow: hidden; text-overflow: ellipsis;\"\u003e\n\u003cspan style=\"color:white;\"\u003e$ tail -f /var/log/nginx/access.log\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e192.0.32.10 - - [21/Jan/2011:14:07:34 +0000] \"GET / HTTP/1.1\" 200 3700 \"-\" \"Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.237 Safari/534.10\" \"-\"\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e192.0.32.10 - - [21/Jan/2011:14:07:34 +0000] \"GET /nginx-logo.png HTTP/1.1\" 200 370 \"http://localhost/\" \"Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.237 Safari/534.10\" \"-\"\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e192.0.32.10 - - [21/Jan/2011:14:07:34 +0000] \"GET /poweredby.png HTTP/1.1\" 200 3034 \"http://localhost/\" \"Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.237 Safari/534.10\" \"-\"\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e192.0.32.10 - - [21/Jan/2011:14:07:34 +0000] \"GET /favicon.ico HTTP/1.1\" 404 3650 \"-\" \"Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.237 Safari/534.10\" \"-\"\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e192.0.32.10 - - [21/Jan/2011:14:19:04 +0000] \"GET /nginx-logo.png HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.237 Safari/534.10\" \"-\"\u003c/span\u003e\n\u003cspan style=\"color:white;\"\u003e$ _\u003c/span\u003e\n\u003c/pre\u003e\n\n... you get this:\n\n\u003cpre style=\"background-color: black; padding: 15px; width: 800px;\"\u003e\n\u003cspan style=\"color:white;\"\u003e$ tail -f /var/log/nginx/access.log \u003cstrong\u003e| ntail\u003c/strong\u003e\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e2011-01-21 14:07:34 -       192.0.32.10 - 200 - GET / - (Chrome, Linux) - -\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e2011-01-21 14:07:34 -       192.0.32.10 - 200 - GET /nginx-logo.png - (Chrome, Linux) - localhost\u003c/span\u003e\n\u003cspan style=\"color: green;\"\u003e2011-01-21 14:07:34 -       192.0.32.10 - 200 - GET /spanoweredby.png - (Chrome, Linux) - localhost\u003c/span\u003e\n\u003cspan style=\"color: red;\"\u003e2011-01-21 14:07:34 -       192.0.32.10 - 404 - GET /favicon.ico - (Chrome, Linux) - -\u003c/span\u003e\n\u003cspan style=\"color: orange;\"\u003e2011-01-21 14:19:04 -       192.0.32.10 - 304 - GET /nginx-logo.png - (Chrome, Linux) - -\u003c/span\u003e\n\u003cspan style=\"color:white;\"\u003e$ _\u003c/span\u003e\n\u003c/pre\u003e\n\n\u003ca name=\"installation\"/\u003e\n\nInstallation\n------------\n\nInstalling the gem also installs the `ntail` executable, typically as `/usr/bin/ntail` or `/usr/local/bin/ntail`:\n\n    $ gem install ntail\n\nTo ensure easy execution of the `ntail` script, add the actual installation directory to your shell's `$PATH` variable.\n\n\u003ca name=\"basic\"/\u003e\n\nBasic Usage\n-----------\n\n* process an entire nginx log file and print each parsed and formatted line to STDOUT\n\n        $ ntail /var/log/nginx/access.log\n\n* process an entire nginx log file and pipe each parsed and formatted line into a browser _(depends on the optional `bcat` gem)_\n\n        $ ntail /var/log/nginx/access.log | bcat\n\n* tail an \"active\" nginx log file and print each new line to STDOUT _(stop with ^C)_\n\n        $ tail -f /var/log/nginx/access.log | ntail\n\n* tail an \"active\" nginx log file and pipe each new line into a browser _(stop with ^C)_\n\n        $ tail -f /var/log/nginx/access.log | ntail | bcat\n\n\u003ca name=\"advanced\"/\u003e\n\nAdvanced Examples\n-----------------\n\n* read from STDIN and print each line to STDOUT _(stop with ^D)_\n\n        $ ntail\n\n* read from STDIN and print out the length of each line _(to illustrate -e option)_\n\n        $ ntail -e 'puts size'\n\n* read from STDIN but only print out non-empty lines _(to illustrate -f option)_\n\n        $ ntail -f 'size != 0'\n\n* the following invocations behave exactly the same _(to illustrate -e and -f options)_\n\n        $ ntail\n        $ ntail -f 'true' -e 'puts self'\n\n* print out all HTTP requests that are coming from a given IP address\n\n        $ ntail -f 'remote_address == \"208.67.222.222\"' /var/log/nginx/access.log\n\n* find all HTTP requests that resulted in a '5xx' HTTP error/status code _(e.g. Rails 500 errors)_\n\n        $ gunzip -S .gz -c access.log-20101216.gz | ntail -f 'server_error_status?'\n\n* generate a summary report of HTTP status codes, for all non-200 HTTP requests\n\n        $ ntail -f 'status != \"200\"' -e 'puts status' access.log | sort | uniq -c\n        76 301\n        16 302\n         2 304\n         1 406\n\n* print out GeoIP country and city information for each HTTP request _(depends on the optional `geoip` gem)_\n\n        $ ntail -e 'puts [to_country_s, to_city_s].join(\"\\t\")' /var/log/nginx/access.log\n        United States   Los Angeles\n        United States   Houston\n        Germany         Berlin\n        United Kingdom  London\n\n* print out the IP address and the corresponding host name for each HTTP request _(slows things down considerably, due to `nslookup` call)_\n\n        $ ntail -e 'puts [remote_address, to_host_s].join(\"\\t\")' /var/log/nginx/access.log\n        66.249.72.196   crawl-66-249-72-196.googlebot.com\n        67.192.120.134  s402.pingdom.com\n        75.31.109.144   adsl-75-31-109-144.dsl.irvnca.sbcglobal.net\n\n* parse an access log file, and pipe its raw output (indirectly - via the `parsed.log` file) into the `gltail` realtime logfile visualizer\n\n        $ ntail -v --raw --sleep 0.1 /var/log/nginx/access.log \u003e parsed.log\n\n\u003ca name=\"todo\"/\u003e\n\nTODO\n----\n\n* implement a native `\"-f\"` option for ntail, similar to that of `tail(1)`, using e.g. flori's [file-tail gem](https://github.com/flori/file-tail)\n* implement a `\"-i\"` option (\"ignore exceptions\"/\"continue processing\"), if handling a single line raises an exception\n* or indeed a reverse `\"-r\"` option (\"re-raise exception\"), to immediately stop processing and raising the exception for investigation\n* implement (better) support for custom nginx log formats, in addition to [nginx's default \"combined\" log format](http://wiki.nginx.org/NginxHttpLogModule#log_format) (as well as Apache's various log formats)\n\n\u003ca name=\"acknowledgements\"/\u003e\n\nAcknowledgements\n----------------\n\n* ntail's parsing feature is inspired by an nginx log parser written by [Richard Taylor (moomerman)](https://github.com/moomerman)\n* parsing and expanding ntail's formatting string is done using nathansobo's quite brilliant [treetop gem](https://github.com/nathansobo/treetop)\n* ntail's raw line output is compatible with Fudge's fun and useful [gltail gem](https://github.com/Fudge/gltail), as well as the quite amazing [logstalgia](http://code.google.com/p/logstalgia/)\n* kudos to [Ed James (edjames)](https://github.com/edjames) for recommending the use of [instance_eval][eval] to clean up the DSL\n* ntail's HTML formatting was requested/suggested by [Phil Berryman](http://twitter.com/#!/philberryman) and implemented whilst listening to [Leftism](http://en.wikipedia.org/wiki/Leftism_\\(album\\)), the Leftfield album recommended to me by [Alastair Douglas](http://twitter.com/#!/alastairdouglas)\n\n[eval]: https://github.com/pvdb/ntail/commit/b0f40522012b9858c433808cd1f5c21cb455fadd \"use instance_eval to simplify the DSL\"\n\n\u003ca name=\"contributing\"/\u003e\n\nContributing to ntail\n---------------------\n\n* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet\n* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it\n* Fork the project\n* Start a feature/bugfix branch\n* Commit and push until you are happy with your contribution\n* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.\n* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.\n\n\u003ca name=\"copyright\"/\u003e\n\nCopyright\n---------\n\nCopyright (c) 2011 Peter Vandenberk. See LICENSE.txt for further details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpvdb%2Fntail","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpvdb%2Fntail","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpvdb%2Fntail/lists"}