Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/sysread/app-jsonlogutils

Command line utilities for dealing with JSON-formatted log files
https://github.com/sysread/app-jsonlogutils

cat cli column command-line command-line-tool grep json log perl tail

Last synced: about 1 month ago
JSON representation

Command line utilities for dealing with JSON-formatted log files

Awesome Lists containing this project

README 

        
=pod



=encoding UTF-8



=head1 NAME



App::JsonLogUtils - Command line utilities for dealing with JSON-formatted log files



=head1 VERSION



version 0.03



=head1 SYNOPSIS



  # From the command line

  tail -f /path/to/log/file.log \

    | jgrep -m message="some pattern" \

    | jcut -f "timestamp priority message" \

    | cols -c "timestamp priority message" -s '|' \

    | column -t -s '|'



  # From code

  use App::JsonLogUtils qw(tail json_log);



  my $log = json_log tail '/path/to/file.log';



  while (my $entry = <$log>) {

    my ($json, $line) = @$entry;

    ...

  }



  # Grepping JSON logs

  use App::JsonLogUtils qw(lines json_log);

  use Iterator::Simple qw(igrep imap);



  my $entries = igrep{ $_->{foo} =~ /bar/ } # filter objects

                imap{ $_->[0] }             # select the object

                json_log                    # parse

                lines '/path/to/file.log';  # read



=head1 DESCRIPTION



Writing logs in JSON, one object per line, makes them very easily machine

readable. Wonderful. Unfortunately, it also makes it unfuriating to deal with

them using the standard unix command line tools. This package provides a few

tools to salve the burn.



=head1 COMMAND LINE TOOLS



=head2 L



Greps patterns in individual object fields.



=head2 L



Filter the fields included in objects.



=head2 L



Display fields in a format suitable for C.



=head2 L



An interactive shell for monitoring JSON log files.



=head1 EXPORTABLE ROUTINES



If desired, the iterators used to implement the tools above are optionally

exported by the main module.



=head1 lines



Accepts a file path or opened file handle and returns an iterator which yields

the chomped lines from the file.



  my $log = lines '/path/to/file.log';



  while (my $line = <$log>) {

    ...

  }



=head1 tail



Accepts a file path or opened file handle and returns an iterator while yields

chomped lines from the file as they are appended, starting from the end of the

file. Lines already written to the file when this routine is first called are

ignored (that is, there is no equivalent to C at this time).



  my $log = tail '/path/to/file.log';



  while (my $line = <$log>) { # sleeps until lines appended to file

    ...

  }



=head1 json_log



Accepts a file iterator (see L and L) and returns an iterator

yielding an array ref holding two items, a hash ref of the parsed JSON log

entry, and the original log entry string. Empty lines are skipped with a

warning. JSON decoding errors are ignored with a warning.



  my $lines = json_log tail '/path/to/file.log';



  while (my $entry = <$lines>) {

    my ($object, $line) = @_;

    ...

  }



=head2 json_cols



Accepts a list of fields (as a space-separared string or array ref of strings),

a string separator, and an iterator over JSON object strings, and returns a new

iterator. The returned iterator will first yield a string of column names

joined by the separator string. Subsequent calls will iterate over the JSON

object strings, return the value of each of the selected fields joined by the

separator string.



  # File $input

  {"a": 1, "b": 2, "c": 3}

  {"a": 4, "b": 5, "c": 6}

  {"a": 7, "b": 8, "c": 9}



  # Select columns a and c, separated by a pipe

  my $cols = json_cols "a c", "|" , lines $input;



  # ...yields the following strings:

  "a|c"

  "1|3"

  "4|6"

  "7|9"



=head2 json_cut



Accepts a space-separated string or array ref of C<$fields>, boolean

C<$inverse>, and an iterator of JSON log lines. Returns an iterator yielding

objects containing only the fields selected in C<$fields>. If C<$inverse> is

true, instead yields objects containing only the fields I contained in

C<$fields>.



Give the same input as the L:



  my $cut = json_cut "a c", 0, lines $input;



  # ...yields the following hash refs:

  {a => 1, c => 3}

  {a => 4, c => 6}

  {a => 7, c => 9}



  # Inverted

  my $cut = json_cut "a c", 1, lines $input;



  # ...yields:

  {b => 2}

  {b => 5}

  {b => 8}



=head2 json_grep



Accepts a hash ref where keys are field names and values are arrays of

regular expressions, a boolean C<$inverse>, and an iterator of JSON object

strings. Returns an iterator yielding array refs of the parsed JSON object

hash and the original string (just like L). Only those entries

for which all fields' patterns match are returned. If C<$inverse> is set,

the logic is negated and only those entries for which all patterns test

false are returned.



  # File $input

  {"foo": "bar"}

  {"foo": "baz"}

  {"foo": "bat"}

  {"foo": "BAR"}



  # Code

  my $entries = json_grep { foo => [qw/bar/i, qr/baz/] }, 0, lines $input;



  # ...yields the following:

  [ {foo => "bar"}, '{"foo": "bar"}' ]

  [ {foo => "baz"}, '{"foo": "baz"}' ]

  [ {foo => "BAR"}, '{"foo": "BAR"}' ]



=head1 FUTURE PLANS



None, but will happily consider requests and patches.



=head1 AUTHOR



Jeff Ober 



=head1 COPYRIGHT AND LICENSE



This software is copyright (c) 2018 by Jeff Ober.



This is free software; you can redistribute it and/or modify it under

the same terms as the Perl 5 programming language system itself.



=cut