{"id":13761470,"url":"https://github.com/benbernard/RecordStream","last_synced_at":"2025-05-10T12:32:38.582Z","repository":{"id":1086849,"uuid":"937398","full_name":"benbernard/RecordStream","owner":"benbernard","description":"commandline tools for slicing and dicing JSON records.","archived":false,"fork":false,"pushed_at":"2020-06-29T18:59:35.000Z","size":19728,"stargazers_count":303,"open_issues_count":10,"forks_count":31,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-05-03T10:47:56.168Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benbernard.png","metadata":{"files":{"readme":"README.md","changelog":"Changes","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-09-24T22:46:34.000Z","updated_at":"2025-01-18T20:17:02.000Z","dependencies_parsed_at":"2022-08-16T12:00:30.685Z","dependency_job_id":null,"html_url":"https://github.com/benbernard/RecordStream","commit_stats":null,"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benbernard%2FRecordStream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benbernard%2FRecordStream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benbernard%2FRecordStream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benbernard%2FRecordStream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benbernard","download_url":"https://codeload.github.com/benbernard/RecordStream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253415826,"owners_count":21904932,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T13:01:56.868Z","updated_at":"2025-05-10T12:32:33.614Z","avatar_url":"https://github.com/benbernard.png","language":"Perl","funding_links":[],"categories":["Perl","\u003ca name=\"data-management-json\"\u003e\u003c/a\u003eData management - JSON/YAML/etc."],"sub_categories":[],"readme":"\n\u003cimg src=\"https://cdn.rawgit.com/benbernard/RecordStream/master/logos/logo-small.svg\" align=\"right\"\u003e\n\n\n[![CPAN version](https://badge.fury.io/pl/App-RecordStream.png)](https://metacpan.org/release/App-RecordStream)\n[![Build Status](https://travis-ci.org/benbernard/RecordStream.svg?branch=master)](https://travis-ci.org/benbernard/RecordStream)\n\n# NAME\n\nApp::RecordStream - recs - A system for command-line analysis of data\n\n# SYNOPSIS\n\nA set of programs for creating, manipulating, and outputting a stream of\nRecords, or JSON hashes.  Inspired by Monad.\n\n# INSTALLATION\n\n## Quick, standalone bundle\n\nThe quickest way to start using recs is via the minimal, standalone bundle:\n\n    curl -fsSL https://recs.pl \u003e recs\n    chmod +x recs\n    ./recs --help\n\nThis is also known as the \"fatpacked\" recs.\n\n## From CPAN\n\nYou can also install recs from [CPAN](http://cpan.org) as App::RecordStream:\n\n    cpanm --interactive App::RecordStream\n\nUsing [cpanm](https://metacpan.org/pod/cpanm) in interactive mode will prompt you for optional feature\nsupport.  Other CPAN clients such as [cpan](https://metacpan.org/pod/cpan) and [cpanp](https://metacpan.org/pod/cpanp) also work fine, but\nyou can't opt to use any optional features (just like cpanm in non-interactive\nmode).  A kitchen-sink install of App::RecordStream looks like:\n\n    cpanm --with-recommends --with-all-features App::RecordStream\n\nIf you don't have [cpanm](https://metacpan.org/pod/cpanm) itself, you can install it easily with:\n\n    curl -fsSL https://cpanmin.us | perl - App::cpanminus\n\n# DESCRIPTION\n\nThe recs system consists of three basic sets of commands:\n\n- _Input_ commands responsible for generating streams of record objects\n- _Manipulation_ commands responsible for analyzing, selecting, and manipulating records\n- _Output_ commands responsible for taking record streams and producing output for humans or other programs\n\nThese commands can interface with other systems to retrieve data, parse existing\nfiles, or just regex out some values from a text stream.\n\nCommands are run using `recs command [options and arguments]`.  If you're using\na CPAN-based install, you may also run commands directly as `recs-command`,\nthough this is no longer recommended for forwards compatibility.  Both\n[installation methods](#installation) provide a top-level `recs` executable\nwhich dispatches to commands, so this is the preferred invocation style.\n\nThe core recs commands are briefly summarized below, and you can list all\navailable commands by running `recs --list`.\n\nTo read more about each command, run `recs command --help`.  Longer\ndocumentation is available as `recs command --help-all` or `perldoc recs-command`.\nFor example, to read more about [\"fromcsv\"](#fromcsv), you might run any of the\nfollowing:\n\n    recs fromcsv --help\n    recs fromcsv --help-all\n    perldoc recs-fromcsv\n\n# COMMANDS\n\n## Input Generation\n\n- fromcsv\n\n    Produces records from a csv file/stream\n\n- fromdb\n\n    Produces records for a db table, or from a SELECT statement into a db.\n\n- fromre\n\n    Matches input streams against a regex, puts capture groups into hashes\n\n- frommongo\n\n    Generate a record stream from a MongoDB query.\n\n- frommultire\n\n    Matches input streams against several regexes, puts capture groups into the record\n\n- fromsplit\n\n    Splits input stream on a delimiter\n\n- fromps\n\n    Generate records from the process tree\n\n- fromatomfeed\n\n    Produces records for an optionally paginated atom feed.\n\n- fromxml\n\n    Produces records for an XML document.\n\n- fromkv\n\n    Produces records from input streams containing loosely formed key/value pairs\n\n- fromtcpdump\n\n    Produces records from packet capture files (.pcap) as made by tcpdump\n\n## Stream Manipulation\n\n- annotate\n\n    Annotate records with common fields, will memoize additions to speed up common\n    annotations\n\n- collate\n\n    Perforce aggregation operations on records.  Group by a field, get an average,\n    sum, correlation, etc.  Very powerful\n\n- delta\n\n    Transform values into deltas between adjacent records\n\n- eval\n\n    Eval a string of Perl against each record\n\n- flatten\n\n    Flatten records of input to one level\n\n- grep\n\n    Select records for which a string of Perl evaluates to true.\n\n- multiplex\n\n    Take records, grouped by keys, and run a separate recs command for each group.\n\n- normalizetime\n\n    Based on a time field, tag records with a normalized time, i.e. every 5 minute buckets\n\n- join\n\n    Perform an inner join of two record streams.  Associate records in one stream\n    with another stream.\n\n- substream\n\n    Filter to a range of matching records with paired Perl snippets `--start` and `--end`.\n\n- sort\n\n    Sort records based on keys, may specify multiple levels of sorting, as well as\n    numerical or lexical sort ordering\n\n- topn\n\n    Outputs the top _n_ records. You may segment the input based on a list of keys\n    such that unique values of keys are treated as distinct input streams. This\n    enables top _n_ listings per value groupings.\n\n- xform\n\n    Perform a block of Perl on each record, which may modify the record, Record is\n    then output\n\n- generate\n\n    Perform a block of Perl on each record to generate a record stream, which is\n    then output with a chain link back to the original record.\n\n## Output Generation\n\n- todb\n\n    Inserts records into a DBI supported SQL database.  Will create a local SQLite\n    database by default\n\n- tocsv\n\n    Generates correctly quoted CSV files from record streams.\n\n- tognuplot\n\n    Create a graph of field values in a record using GNU Plot.\n\n- totable\n\n    Pretty prints a table of results.\n\n- tohtml\n\n    Prints out an HTML table of the record stream\n\n- toprettyprint\n\n    Prettily prints records, one key to a line, great for making sense of very large records\n\n- toptable\n\n    Prints a multi-dimensional (pivot) table of values.  Very powerful.\n\n# KEY SPECS\n\nMany of the commands above take key arguments to specify or assign to a key in a\nrecord. Almost all of the places where you can specify a key (which normally\nmeans a first level key in the record), you can instead specify a key spec.\n\nA key spec may be nested, and may index into arrays.  Use a `/` to nest into a\nhash and a `#NUM` to index into an array (i.e. `#2`)\n\nAn example is in order, take a record like this:\n\n    {\"biz\":[\"a\",\"b\",\"c\"],\"foo\":{\"bar 1\":1},\"zap\":\"blah1\"}\n    {\"biz\":[\"a\",\"b\",\"c\"],\"foo\":{\"bar 1\":2},\"zap\":\"blah2\"}\n    {\"biz\":[\"a\",\"b\",\"c\"],\"foo\":{\"bar 1\":3},\"zap\":\"blah3\"}\n\nIn this case a key spec of `foo/bar 1` would have the values 1, 2, and 3\nrespectively.\n\nSimilarly, `biz/#0` would have the value of `a` for all 3 records\n\n## Fuzzy matching\n\nYou can also prefix key specs with `@` to engage the fuzzy matching logic.\nMatching is tried like this, in order, with the first key to match winning:\n\n- 1. Exact match (`eq`)\n- 2. Prefix match (`m/^/`)\n- 3. Match anywhere in the key (`m//`)\n\nGiven the above example data and the fuzzy key spec `@b/#2`, the `b` portion\nwould expand to `biz` and `2` would be the index into the array, so all\nrecords would have the value of `c`.\n\nSimiarly, `@f/b` would have values 1, 2, and 3.\n\n# WRITING YOUR OWN COMMANDS\n\nThe data stream format of the recs commands is JSON hashes separated by new\nlines.  If you wish to write your own recs command in your own language, just\nget a JSON parser and you should be good to go.  The recs commands use\n[JSON::MaybeXS](https://metacpan.org/pod/JSON::MaybeXS).\n\nIf you name your command as `recs-mycommand` and put it somewhere in your\n`PATH` environment variable, the `recs` command will dispatch to it when\ncalled as `recs mycommand`.  It will also be included in `recs --list`\noutput.\n\nIf you want to write your new command in Perl, you can use the same Perl API\nthat the standard recs toolkit uses.  See the various\n[App::RecordStream::Operation](https://metacpan.org/pod/App::RecordStream::Operation) subclasses.  Once your new operation class is\ninstalled in perl's library paths, `recs` will find it automatically without\nthe need for any executable command shim.\n\n# EXAMPLES\n\n    # look in the custom access log for all accesses with greater than 5 seconds,\n    # display in a table\n    cat access.log \\\n      | recs fromre --fields ip,time '^(\\d+).*TIME: (\\d+)' \\\n      | recs grep '$r-\u003e{time} \u003e 5' \\\n      | recs totable\n\n# SEE ALSO\n\nEach of the commands discussed have a `--help` mode available to print out\nusage and examples for the particular command.  See that documentation for\ndetailed information on the operation of each of the commands.  Also see some\nother man pages:\n\n- Run `recs examples` or see [App::RecordStream::Manual::Examples](https://metacpan.org/pod/App::RecordStream::Manual::Examples) for a set of simple recs examples\n- Run `recs story` or see [App::RecordStream::Manual::Story](https://metacpan.org/pod/App::RecordStream::Manual::Story) for a humorous introduction to RecordStream\n\n# AUTHORS\n\nBenjamin Bernard \u003cperlhacker@benjaminbernard.com\u003e\n\nKeith Amling \u003ckeith.amling@gmail.com\u003e\n\nThomas Sibley \u003ctsibley@cpan.org\u003e\n\n# COPYRIGHT AND LICENSE\n\nCopyright 2007–2017 by the AUTHORS\n\nThis software is released under the MIT and Artistic 1.0 licenses.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenbernard%2FRecordStream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenbernard%2FRecordStream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenbernard%2FRecordStream/lists"}