{"id":19795580,"url":"https://github.com/dkogan/vnlog","last_synced_at":"2025-04-05T00:09:17.105Z","repository":{"id":41337486,"uuid":"115182073","full_name":"dkogan/vnlog","owner":"dkogan","description":"Process labelled tabular ASCII data using normal UNIX tools","archived":false,"fork":false,"pushed_at":"2025-02-15T03:49:14.000Z","size":856,"stargazers_count":161,"open_issues_count":0,"forks_count":6,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-28T23:08:40.111Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dkogan.png","metadata":{"files":{"readme":"README.org","changelog":"Changes","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-12-23T08:25:47.000Z","updated_at":"2025-02-15T03:49:13.000Z","dependencies_parsed_at":"2024-04-25T17:42:55.108Z","dependency_job_id":"c4f8ccc4-a50d-488b-92fd-d8322772f6ba","html_url":"https://github.com/dkogan/vnlog","commit_stats":null,"previous_names":[],"tags_count":54,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fvnlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fvnlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fvnlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dkogan%2Fvnlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dkogan","download_url":"https://codeload.github.com/dkogan/vnlog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266564,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T07:16:46.146Z","updated_at":"2025-04-05T00:09:17.081Z","avatar_url":"https://github.com/dkogan.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"* Talk\n\nI just gave a talk about this at [[https://www.socallinuxexpo.org/scale/17x][SCaLE 17x]]. Here are the [[https://www.youtube.com/watch?v=Qvb_uNkFGNQ\u0026t=12830s][video of the talk]] and\nthe [[https://github.com/dkogan/talk-feedgnuplot-vnlog/blob/master/feedgnuplot-vnlog.org][\"slides\"]].\n\n* Summary\n\nVnlog (\"vanilla-log\") is a toolkit for manipulating tabular ASCII data with\nlabelled fields using normal UNIX tools. If you regularly use =awk= and =sort=\nand =uniq= and others, these tools will make you infinitely more powerful. The\nvnlog tools /extend/, rather than replace the standard tooling, so minimal\neffort is required to learn and use these tools.\n\nEverything assumes a trivially simple log format:\n\n- A whitespace-separated table of ASCII human-readable text\n- A =#= character starts a comment that runs to the end of the line (like in\n many scripting languages)\n- The first line that begins with a single =#= (not =##= or =#!=) is a /legend/,\n naming each column. This is required, and the field names that appear here are\n referenced by all the tools.\n- Empty fields reported as =-=\n\nThis describes 99% of the format, with some extra details [[#format-details][below]]. Example:\n\n#+BEGIN_EXAMPLE\n#!/usr/bin/whatever\n# a b c\n1 2 3\n## comment\n4 5 6\n#+END_EXAMPLE\n\nSuch data can be processed directly with almost any existing tool, and /this/\ntoolkit allows the user to manipulate this data in a nicer way by relying on\nstandard UNIX tools. The core philosophy is to avoid creating new knowledge as\nmuch as possible. Consequently, the vnlog toolkit relies /heavily/ on existing\n(and familiar!) tools and workflows. As such, the toolkit is small, light, and\nhas a /very/ friendly learning curve.\n\n* Synopsis\n\nI have [[https://raw.githubusercontent.com/dkogan/vnlog/master/dji-tsla.tar.gz][two sets of historical stock data]], from the start of 2018 until now\n(2018/11):\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl head -n 4\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume\n: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840\n: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000\n: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000\n\nAnd\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c tsla.vnl head -n 4\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume\n: 2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339\n: 2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300\n: 2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600\n\nI can add whitespace to make the headers more legible by humans:\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl head -n 4 | vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume \n: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840\n: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000\n: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000\n\nI can pull out the closing prices:\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl vnl-filter -p Close | head -n4\n#+END_SRC\n\n#+RESULTS:\n: # Close\n: 25289.27\n: 25080.50\n: 25286.49\n\n=vnl-filter= is primarily a wrapper around =awk= or =perl=, allowing the user to\nreference columns by name. I can then plot the closing prices:\n\n#+BEGIN_SRC sh :results file link :exports both\n\u003c dji.vnl vnl-filter -p Close |\n feedgnuplot --lines --unset grid\n#+END_SRC\n\n#+RESULTS:\n[[file:guide-1.svg]]\n\nHere I kept /only/ the closing price column, so the x-axis is just the row\nindex. The data was in reverse chronological order, so this plot is also in\nreverse chronological order. Let's fix that:\n\n#+BEGIN_SRC sh :results file link :exports both\n\u003c dji.vnl vnl-sort -k Date |\n vnl-filter -p Close |\n feedgnuplot --lines --unset grid\n#+END_SRC\n\n#+RESULTS:\n[[file:guide-2.svg]]\n\nThe =vnl-sort= tool (and most of the other =vnl-xxx= tools) are wrappers around\nthe core tools already available on the system (such as =sort=, in this case).\nWith the primary difference being reading/writing vnlog, and referring to\ncolumns by name.\n\nWe now have the data in the correct order, but it'd be nice to see the actual\ndates on the x-axis. While we're at it, let's label the axes too:\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl vnl-filter -p Date,Close | head -n4\n#+END_SRC\n\n#+RESULTS:\n: # Date Close\n: 2018-11-15 25289.27\n: 2018-11-14 25080.50\n: 2018-11-13 25286.49\n\n#+BEGIN_SRC sh :results file link :exports both\n\u003c dji.vnl vnl-sort -k Date |\n vnl-filter -p Date,Close |\n feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain \\\n --xlabel 'Date' --ylabel 'Price ($)'\n#+END_SRC\n\n#+RESULTS:\n[[file:guide-3.svg]]\n\nWhat was the highest value of the Dow-Jones index, and when did it happen?\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl vnl-sort -rgk Close |\n head -n2 |\n vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume \n: 2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000\n\nAlrighty. Looks like the high was in October. Let's zoom in on that month:\n\n#+BEGIN_SRC sh :results file link :exports both\n\u003c dji.vnl vnl-sort -k Date |\n vnl-filter 'Date ~ /2018-10/' -p Date,Close |\n feedgnuplot --lines --unset grid --timefmt %Y-%m-%d --domain \\\n --xlabel 'Date' --ylabel 'Price ($)'\n#+END_SRC\n\n#+RESULTS:\n[[file:guide-4.svg]]\n\nOK. Is this thing volatile? What was the largest single-day gain?\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl vnl-filter -p '.,d=diff(Close)' |\n head -n4 |\n vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume d \n: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 - \n: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 -208.77\n: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 205.99\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl vnl-filter -p '.,d=diff(Close)' |\n vnl-sort -rgk d |\n head -n2 |\n vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume d \n: 2018-02-02 26061.79 26061.79 25490.66 25520.96 25520.96 522880000 1175.21\n\nWhoa. So the best single-gain day was 2018-02-02: the dow gained 1175.21 points\nbetween closing on Feb 1 and Feb 2. But it actually lost ground that day! What\nif I looked at the difference between the opening and closing in a single day?\n\n#+BEGIN_SRC sh :results output :exports both\n\u003c dji.vnl vnl-filter -p '.,d=Close-Open' |\n vnl-sort -rgk d |\n head -n2 |\n vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Date Open High Low Close AdjClose Volume d \n: 2018-02-06 24085.17 24946.23 23778.74 24912.77 24912.77 823940000 827.6\n\nI guess by that metric 2018-02-06 was better. Let's join the Dow-jones index\ndata and the TSLA data, and let's look at them together:\n\n#+BEGIN_SRC sh :results output :exports both\nvnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |\n head -n4 |\n vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Date Open_dji High_dji Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla\n: 2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840 342.33 348.58 339.04 348.44 348.44 4486339 \n: 2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 342.70 347.11 337.15 344.00 344.00 5036300 \n: 2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000 333.16 344.70 332.20 338.73 338.73 5448600 \n\n#+BEGIN_SRC sh :results output :exports both\nvnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |\n vnl-filter -p '^Close' |\n head -n4 |\n vnl-align\n#+END_SRC\n\n#+RESULTS:\n: # Close_dji Close_tsla\n: 25289.27 348.44 \n: 25080.50 344.00 \n: 25286.49 338.73 \n\n#+BEGIN_SRC sh :results file link :exports both\nvnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |\n vnl-filter -p '^Close' |\n feedgnuplot --domain --points --unset grid \\\n --xlabel 'DJI price ($)' --ylabel 'TSLA price ($)'\n#+END_SRC\n\n#+RESULTS:\n[[file:guide-5.svg]]\n\nHuh. Apparently there's no obvious, strong correlation between TSLA and\nDow-Jones closing prices. And we saw that with just a few shell commands,\nwithout dropping down into a dedicated analysis system.\n\n* Build and installation\nvnlog is a part of Debian/buster and Ubuntu/cosmic (18.10) and later. On those\nboxes you can simply\n\n#+BEGIN_EXAMPLE\n$ sudo apt install vnlog libvnlog-dev libvnlog-perl python3-vnlog\n#+END_EXAMPLE\n\nto get the binary tools, the C API, the perl and python3 interfaces\nrespectively.\n\n** Install on non-Debian boxes\nMost of this is written in an interpreted language, so there's nothing to build\nor install, and you can run the tools directly from the source tree:\n\n#+BEGIN_EXAMPLE\n$ git clone https://github.com/dkogan/vnlog.git\n$ cd vnlog\n$ ./vnl-filter .....\n#+END_EXAMPLE\n\nThe python and perl libraries can be run from the tree by setting the\n=PYTHONPATH= and =PERL5LIB= environment variables respectively. For the C\nlibrary, you should =make=, and then point your =CFLAGS= and =LDLIBS= and\n=LD_LIBRARY_PATH= to the local tree.\n\nIf you do want to install to some arbitrary location to simplify the paths, do\nthis:\n\n#+BEGIN_EXAMPLE\n$ make\n$ PREFIX=/usr/local make install\n#+END_EXAMPLE\n\nThis will install /all/ the components into =/usr/local=.\n\n* Description\nVnlog data is nicely readable by both humans and machines. Any time your\napplication invokes =printf()= for either diagnostics or logging, consider\nwriting out vnlog-formatted data. You retain human readability, but gain the\npower all the =vnl-...= tools provide.\n\nVnlog tools are designed to be very simple and light. There's an ever-growing\nlist of other tools that do vaguely the same thing. Some of these:\n\n- https://github.com/BurntSushi/xsv\n- https://csvkit.readthedocs.io/\n- https://github.com/johnkerl/miller\n- https://github.com/jqnatividad/qsv\n- https://github.com/greymd/teip\n- https://github.com/eBay/tsv-utils-dlang\n- https://www.gnu.org/software/datamash/\n- https://stedolan.github.io/jq/\n- https://github.com/benbernard/RecordStream\n- https://github.com/dinedal/textql\n- https://www.visidata.org/\n- http://harelba.github.io/q/\n- https://github.com/BatchLabs/charlatan\n- https://github.com/dbohdan/sqawk\n\nMany of these provide facilities to run various analyses, and others focus on\ndata types that aren't just a table (json for instance). Vnlog by contrast\ndoesn't analyze anything, and targets the most trivial possible data format.\nThis makes it very easy to run any analysis you like in any tool you like. The\nmain envisioned use case is one-liners, and the tools are geared for that\npurpose. The above mentioned tools are much more powerful than vnlog, so they\ncould be a better fit for some use cases. I claim that\n\n- 90% of the time you want to do simple things, and vnlog is a great fit for the\n task\n- If you really do need to do something complex, you shouldn't be in the shell\n writing oneliners anymore, and a fully-fledged analysis system (numpy, etc) is\n more appropriate\n\nIn the spirit of doing as little as possible, the provided tools are wrappers\naround tools you already have and are familiar with. The provided tools are:\n\n- =vnl-filter= is a tool to select a subset of the rows/columns in a vnlog\n and/or to manipulate the contents. This is an =awk= wrapper where the fields\n can be referenced by name instead of index. 20-second tutorial:\n\n#+BEGIN_SRC sh :results none :exports code\nvnl-filter -p col1,col2,colx=col3+col4 'col5 \u003e 10' --has col6\n#+END_SRC\n\n will read the input, and produce a vnlog with 3 columns: =col1= and =col2=\n from the input, and a column =colx= that's the sum of =col3= and =col4= in the\n input. Only those rows for which /both/ =col5 \u003e 10= is true /and/ that have a\n non-null value for =col6= will be output. A null entry is signified by a\n single =-= character.\n\n#+BEGIN_SRC sh :results none :exports code\nvnl-filter --eval '{s += x} END {print s}'\n#+END_SRC\n\n#+RESULTS:\n\n will evaluate the given awk program on the input, but the column names work as\n you would hope they do: if the input has a column named =x=, this would\n produce the sum of all values in this column.\n\n- =vnl-sort=, =vnl-uniq=, =vnl-join=, =vnl-tail=, =vnl-ts= are wrappers around\n the corresponding commandline tools. These work exactly as you would expect\n also: the columns can be referenced by name, and the legend comment is handled\n properly. These are wrappers, so all the commandline options those tools have\n \"just work\" (except options that don't make sense in the context of vnlog). As\n an example, =vnl-tail -f= will follow a log: data will be read by =vnl-tail=\n as it is written into the log (just like =tail -f=, but handling the legend\n properly). And you already know how to use these tools without even reading\n the manpages! Note: I use the Linux kernel and the tools from GNU Coreutils\n exclusively, but this all has been successfully tested on FreeBSD and OSX\n also. Please let me know if something doesn't work.\n\n- =vnl-align= aligns vnlog columns for easy interpretation by humans. The\n meaning is unaffected\n\n- =Vnlog::Parser= is a simple perl library to read a vnlog\n\n- =vnlog= is a simple python library to read a vnlog. Both python2 and python3\n are supported\n\n- =libvnlog= is a C library to simplify reading and writing a vnlog. Clearly all\n you /really/ need for writing is =printf()=, but this is useful if we have\n lots of columns, many containing null values in any given row, and/or if we\n have parallel threads writing to a log. In my usage I have hundreds of columns\n of sparse data, so this is handy\n\n- =vnl-make-matrix= converts a one-point-per-line vnlog to a matrix of data.\n I.e.\n\n#+BEGIN_EXAMPLE\n$ cat dat.vnl\n# i j x\n0 0 1\n0 1 2\n0 2 3\n1 0 4\n1 1 5\n1 2 6\n2 0 7\n2 1 8\n2 2 9\n3 0 10\n3 1 11\n3 2 12\n\n$ \u003c dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp\nWriting to '/tmp/x.matrix'\n\n$ cat /tmp/x.matrix\n1 2 3\n4 5 6\n7 8 9\n10 11 12\n#+END_EXAMPLE\n\nAll the tools have manpages that contain more detail. And more tools will\nprobably be added with time.\n\n* Format details\nThe high-level description of the vnlog format from [[#Summary][above]] is sufficient to\nread/write \"normal\" vnlog data, but there are a few corner cases that should be\nmentioned. To reiterate, the format description from above describes vnlog as:\n\n- A whitespace-separated table of ASCII human-readable text\n- A =#= character starts a comment that runs to the end of the line (like in\n many scripting languages)\n- The first line that begins with a single =#= (not =##= or =#!=) is a /legend/,\n naming each column. This is required, and the field names that appear here are\n referenced by all the tools.\n- Empty fields reported as =-=\n\nFor a few years now I've been using these tools myself, and supporting others as\nthey were passing vnlog data around. In the process I've encountered some\nslightly-weird data, and patched the tools to accept it. So today the included\nvnlog tools are /very/ permissive, and accept any vnlog data that can possibly\nbe accepted. Other vnlog tools may not be quite as permissive, and may not be\nable to interpret \"weird\" data. Points of note, describing the included vnlog\ntools:\n\n- Leading and trailing whitespace is ignored. Everywhere. So this data file will\n be read properly, with the =x= column containing 1 and 3:\n\n #+begin_example\n # x y\n1 2\n 3 4\n #+end_example\n\n- Empty (or whitespace-only) lines anywhere are ignored, and treated as a\n comment\n- An initial =#= comment without field names is treated as a comment, and we\n continue looking for the legend in the following lines. So this data file will\n be read properly:\n\n #+begin_example\n## comment\n#\n# x y\n1 2\n3 4\n #+end_example\n\n- Trailing comments are supported, like in most scripting languages. So this\n data file will be read properly:\n\n #+begin_example\n# x y\n1 2 # comment\n3 4\n #+end_example\n\n- Field names are /very/ permissive: anything that isn't whitespace is\n supported. So this data file will be read properly:\n\n #+begin_example\n# x y # 1+ -\n1 2 3 4 5\n11 12 13 14 15\n #+end_example\n\n We can pull out the =#= and =1+= and =-= columns:\n\n #+begin_src sh\nvnl-filter -p '#,1+,-'\n #+end_src\n\n And we can even operate on them, if we use whitespace to indicate field\n boundaries:\n\n #+begin_src sh\nvnl-filter -p 'x=1+ + 5'\n #+end_src\n\n Note that this implies that trailing comments in a legend line are /not/\n supported: the extra =#= characters will be used for field names. Field names\n containing =,= or === are currently not accepted by =vnl-filter=, but /are/\n accepted by the other tools (=vnl-sort= and such). I'll make =vnl-filter= able\n to work with those field names too, eventually, but as a user, the simplest\n thing to do is to not pass around data with such field names.\n\n- Duplicated labels are supported whenever possible. So\n\n #+begin_example\n# x y z z\n1 2 3 4\n11 12 13 14\n #+end_example\n\n will work just fine, unless we're operating on =z=. With this data, both of\n these commands work:\n\n #+begin_src sh\nvnl-filter -p x\nvnl-filter -p z\n #+end_src\n\n Picking =z= selects both of the =z= columns. But neither of these commands can\n work with the non-unique =z= column:\n\n #+begin_src sh\nvnl-filter -p s=z+1\nvnl-sort -k z\n #+end_src\n\n* Workflows and recipes\n** Storing disjoint data\n\nA common use case is a complex application that produces several semi-related\nsubsets of data at once. Example: a moving vehicle is reporting both its own\nposition and the observed positions of other vehicles; at any given time any\nnumber of other vehicles may be observed. Two equivalent workflows are possible:\n\n- a single unified vnlog stream for /all/ the data\n- several discrete vnlog streams for each data subset\n\nBoth are valid approaches\n\n*** One unified vnlog stream\nHere the application produces a /single/ vnlog that contains /all/ the columns,\nfrom /all/ the data subsets. In any given row, many of the columns will be empty\n(i.e. contain only =-= ). For instance, a row describing a vehicle own position\nwill not have data about any observations, and vice versa. It is inefficient to\nstore all the extra =-= but it makes many things much nicer, so it's often worth\nit. =vnl-filter= can be used to pull out the different subsets. Sample\n=joint.vnl=:\n\n#+BEGIN_EXAMPLE\n# time x_self x_observation\n1 10 -\n2 20 -\n2 - 100\n3 30 -\n3 - 200\n3 - 300\n#+END_EXAMPLE\n\nHere we have 3 instances in time. We have no observations at =time= 1, one\nobservation at =time= 2, and two observations at =time= 3. We can use\n=vnl-filter= to pull out the data we want:\n\n#+BEGIN_EXAMPLE\n$ \u003c joint.vnl vnl-filter -p time,self\n\n# time x_self\n1 10\n2 20\n2 -\n3 30\n3 -\n3 -\n#+END_EXAMPLE\n\nIf we only care about our own positions, the =+= modifier in picked columns in\n=vnl-filter= is very useful here:\n\n#+BEGIN_EXAMPLE\n$ \u003c joint.vnl vnl-filter -p time,+self\n\n# time x_self\n1 10\n2 20\n3 30\n\n\n$ \u003c joint.vnl vnl-filter -p time,+observation\n\n# time x_observation\n2 100\n3 200\n3 300\n#+END_EXAMPLE\n\nNote that the default is =--skipempty=, so if we're /only/ looking at =x_self=\nfor instance, then we don't even need to =+= modifier:\n\n#+begin_example\n$ \u003c joint.vnl vnl-filter -p self\n\n# x_self\n10\n20\n30\n#+end_example\n\nAlso, note that the =vnlog= C interface works very nicely to produce these\ndatafiles:\n\n- You can define lots and lots of columns, but only fill some of them before\n calling =vnlog_emit_record()=. The rest will be set to =-=.\n- You can create multiple contexts for each type of data, and you can populate\n them with data independently. And when calling =vnlog_emit_record_ctx()=,\n you'll get a record with data for just that context.\n\n*** Several discrete vnlog streams\n\nConversely, the application can produce /separate/ vnlog streams for /each/\nsubset of data. Depending on what is desired, exactly, =vnl-join= can be used to\nre-join them:\n\n#+BEGIN_EXAMPLE\n$ cat self.vnl\n\n# time x_self\n1 10\n2 20\n3 30\n\n\n$ cat observations.vnl\n\n# time x_observation\n2 100\n3 200\n3 300\n\n\n$ vnl-join -j time -a- self.vnl observations.vnl\n\n# time x_self x_observation\n1 10 -\n2 20 100\n3 30 200\n3 30 300\n#+END_EXAMPLE\n\n** Data statistics\n\nA common need is to compute basic statistics from your data. Many of the\nalternative toolkits listed above provide built-in facilities to do this, but\nvnlog does not: it's meant to be unixy, where each tool has very limited scope.\nThus you can either do this with =awk= like you would normally, or you can use\nother standalone tools to perform the needed computations. For instance, I can\ngenerate some data:\n\n#+BEGIN_EXAMPLE\n$ seq 2 100 | awk 'BEGIN {print \"# x\"} {print log($1)}' \u003e /tmp/log.vnl\n#+END_EXAMPLE\n\nThen I can compute the mean with =awk=:\n\n#+BEGIN_EXAMPLE\n$ \u003c /tmp/log.vnl vnl-filter --eval '{sum += x} END {print sum/NR}'\n3.67414\n#+END_EXAMPLE\n\nOr I can compute the mean (and other stuff) with a separate standalone tool:\n\n#+BEGIN_EXAMPLE\n$ \u003c /tmp/log.vnl ministat\nx \u003cstdin\u003e\n+----------------------------------------------------------------------------+\n| xx |\n| x xxxxxxx |\n| xx xxxxxxxxxxxx|\n| x x xxxxxxxxxxxxxxxxxxxxxxx|\n|x x x x x x x x x xx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|\n| |_______________A____M___________| |\n+----------------------------------------------------------------------------+\n N Min Max Median Avg Stddev\nx 99 0.693147 4.60517 3.93183 3.6741353 0.85656382\n#+END_EXAMPLE\n\n=ministat= is not a part of the vnlog toolkit, but the vnlog format is generic\nso it works just fine.\n\n** Powershell-style filtering of common shell commands\n\nEverything about vnlog is generic and simple, so it's easy to use it to process\ndata that wasn't originally meant to be used this way. For instance filtering\nthe output of =ls -l= to report only file names and sizes, skipping directories,\nand sorting by file sizes:\n\n#+BEGIN_EXAMPLE\n$ ls -l\n\ntotal 320\n-rw-r--r-- 1 dima dima 5044 Aug 25 15:04 Changes\n-rw-r--r-- 1 dima dima 12749 Aug 25 15:04 Makefile\n-rw-r--r-- 1 dima dima 69789 Aug 25 15:04 README.org\n-rw-r--r-- 1 dima dima 33781 Aug 25 15:04 README.template.org\n-rw-r--r-- 1 dima dima 5359 Aug 25 15:04 b64_cencode.c\ndrwxr-xr-x 4 dima dima 4096 Aug 25 15:04 completions\ndrwxr-xr-x 3 dima dima 4096 Aug 25 15:04 lib\ndrwxr-xr-x 3 dima dima 4096 Aug 25 15:04 packaging\ndrwxr-xr-x 2 dima dima 4096 Aug 25 15:04 test\n-rwxr-xr-x 1 dima dima 5008 Aug 25 15:04 vnl-align\n-rwxr-xr-x 1 dima dima 56637 Aug 25 15:04 vnl-filter\n-rwxr-xr-x 1 dima dima 5678 Aug 25 15:04 vnl-gen-header\n-rwxr-xr-x 1 dima dima 29815 Aug 25 15:04 vnl-join\n-rwxr-xr-x 1 dima dima 3631 Aug 25 15:04 vnl-make-matrix\n-rwxr-xr-x 1 dima dima 8372 Aug 25 15:04 vnl-sort\n-rwxr-xr-x 1 dima dima 5822 Aug 25 15:04 vnl-tail\n-rwxr-xr-x 1 dima dima 4439 Aug 25 15:04 vnl-ts\n-rw-r--r-- 1 dima dima 559 Aug 25 15:04 vnlog-base64.h\n-rw-r--r-- 1 dima dima 8169 Aug 25 15:04 vnlog.c\n-rw-r--r-- 1 dima dima 12677 Aug 25 15:04 vnlog.h\n\n\n$ (echo '# permissions num_links user group size month day time name';\n ls -l | tail -n +2) |\n vnl-filter 'permissions !~ \"^d\"' -p name,size |\n vnl-sort -gk size |\n vnl-align\n\n# name size\nvnlog-base64.h 559\nvnl-make-matrix 3631\nvnl-ts 4439\nvnl-align 5008\nChanges 5044\nb64_cencode.c 5359\nvnl-gen-header 5678\nvnl-tail 5822\nvnlog.c 8169\nvnl-sort 8372\nvnlog.h 12677\nMakefile 12749\nvnl-join 29815\nREADME.template.org 33781\nvnl-filter 56637\nREADME.org 69789\n#+END_EXAMPLE\n\nWith a bit of shell manipulation, these tools can be applied to a whole lot of\ndifferent data streams that know nothing of vnlog.\n\n* C interface\n** Writing vnlog files\n*** Basic usage\nFor most uses, vnlog files are simple enough to be generated with plain prints.\nBut then each print statement has to know which numeric column we're populating,\nwhich becomes effortful with many columns. In my usage it's common to have a\nlarge parallelized C program that's writing logs with hundreds of columns where\nany one record would contain only a subset of the columns. In such a case, it's\nhelpful to have a library that can output the log files. This is available.\nBasic usage looks like this:\n\nIn a shell:\n\n#+BEGIN_SRC sh :results none :exports code\nvnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' \u003e vnlog_fields_generated.h\n#+END_SRC\n\n#+RESULTS:\n\nIn a C program test.c:\n\n#+BEGIN_SRC C\n#include \"vnlog_fields_generated.h\"\n\nint main()\n{\n vnlog_emit_legend();\n\n vnlog_set_field_value__w(-10);\n vnlog_set_field_value__x(40);\n vnlog_set_field_value__y(\"asdf\");\n vnlog_emit_record();\n\n vnlog_set_field_value__z(0.3);\n vnlog_set_field_value__x(50);\n vnlog_set_field_value__w(-20);\n vnlog_set_field_value__binary(\"\\x01\\x02\\x03\", 3);\n vnlog_emit_record();\n\n vnlog_set_field_value__w(-30);\n vnlog_set_field_value__x(10);\n vnlog_set_field_value__y(\"whoa\");\n vnlog_set_field_value__z(0.5);\n vnlog_emit_record();\n\n return 0;\n}\n#+END_SRC\n\nThen we build and run, and we get\n\n#+BEGIN_EXAMPLE\n$ cc -o test test.c -lvnlog\n\n$ ./test\n\n# w x y z binary\n-10 40 asdf - -\n-20 50 - 0.2999999999999999889 AQID\n-30 10 whoa 0.5 -\n#+END_EXAMPLE\n\nThe binary field in base64-encoded. This is a rarely-used feature, but sometimes\nyou really need to log binary data for later processing, and this makes it\npossible.\n\nSo you\n\n1. Generate the header to define your columns\n\n2. Call =vnlog_emit_legend()=\n\n3. Call =vnlog_set_field_value__...()= for each field you want to set in that\n row.\n\n4. Call =vnlog_emit_record()= to write the row and to reset all fields for the\n next row. Any fields unset with a =vnlog_set_field_value__...()= call are\n written as null: =-=\n\nThis is enough for 99% of the use cases. Things get a bit more complex if we\nhave have threading or if we have multiple vnlog ouput streams in the same\nprogram. For both of these we use vnlog /contexts/.\n\n*** Contexts\n\nTo support independent writing into the same vnlog (possibly by multiple\nthreads; this is reentrant), each log-writer should create a context, and use it\nwhen talking to vnlog. The context functions will make sure that the fields in\neach context are independent and that the output records won't clobber each\nother:\n\n#+BEGIN_SRC C\nvoid child_writer( // the parent context also writes to this vnlog. Pass NULL to\n // use the global one\n struct vnlog_context_t* ctx_parent )\n{\n struct vnlog_context_t ctx;\n vnlog_init_child_ctx(\u0026ctx, ctx_parent);\n\n while(records)\n {\n vnlog_set_field_value_ctx__xxx(\u0026ctx, ...);\n vnlog_set_field_value_ctx__yyy(\u0026ctx, ...);\n vnlog_set_field_value_ctx__zzz(\u0026ctx, ...);\n vnlog_emit_record_ctx(\u0026ctx);\n }\n\n vnlog_free_ctx(\u0026ctx); // required only if we have any binary fields\n}\n#+END_SRC\n\nIf we want to have multiple independent vnlog writers to /different/ streams\n(with different columns and legends), we do this instead:\n\n=file1.c=:\n#+BEGIN_SRC C\n#include \"vnlog_fields_generated1.h\"\n\nvoid f(void)\n{\n // Write some data out to the default context and default output (STDOUT)\n vnlog_emit_legend();\n ...\n vnlog_set_field_value__xxx(...);\n vnlog_set_field_value__yyy(...);\n ...\n vnlog_emit_record();\n}\n#+END_SRC\n\n=file2.c=:\n#+BEGIN_SRC C\n#include \"vnlog_fields_generated2.h\"\n\nvoid g(void)\n{\n // Make a new session context, send output to a different file, write\n // out legend, and send out the data\n struct vnlog_context_t ctx;\n vnlog_init_session_ctx(\u0026ctx);\n FILE* fp = fopen(...);\n vnlog_set_output_FILE(\u0026ctx, fp);\n vnlog_emit_legend_ctx(\u0026ctx);\n ...\n vnlog_set_field_value__a(...);\n vnlog_set_field_value__b(...);\n ...\n vnlog_free_ctx(\u0026ctx); // required only if we have any binary fields\n vnlog_emit_record();\n}\n#+END_SRC\n\nNote that it's the user's responsibility to make sure the new sessions go to a\ndifferent =FILE= by invoking =vnlog_set_output_FILE()=. Furthermore, note that\nthe included =vnlog_fields_....h= file defines the fields we're writing to; and\nif we have multiple different vnlog field definitions in the same program (as in\nthis example), then the different writers /must/ live in different source files.\nThe compiler will barf if you try to =#include= two different\n=vnlog_fields_....h= files in the same source.\n\n*** Remaining APIs\n\n- =vnlog_printf(...)= and =vnlog_printf_ctx(ctx, ...)= write to a pipe like\n=printf()= does. This exists primarily for comments.\n\n- =vnlog_clear_fields_ctx(ctx, do_free_binary)= clears out the data in a context\nand makes it ready to be used for the next record. It is rare for the user to\nhave to call this manually. The most common case is handled automatically\n(clearing out a context after emitting a record). One area where this is useful\nis when making a copy of a context:\n\n#+BEGIN_SRC C\nstruct vnlog_context_t ctx1;\n// .... do stuff with ctx1 ... add data to it ...\n\nstruct vnlog_context_t ctx2 = ctx1;\n// ctx1 and ctx2 now both have the same data, and the same pointers to\n// binary data. I need to get rid of the pointer references in ctx1\n\nvnlog_clear_fields_ctx(\u0026ctx1, false);\n#+END_SRC\n\n- =vnlog_free_ctx(ctx)= frees memory for an vnlog context. Do this before\nthrowing the context away. Currently this is only needed for context that have\nbinary fields, but this should be called for all contexts anyway, in case this\nchanges in a later revision\n\n** Reading vnlog files\nThe basic usage goes like this:\n\n#+begin_src c\n#include \u003cstdio.h\u003e\n#include \u003cstdbool.h\u003e\n#include \u003cvnlog/vnlog-parser.h\u003e\nbool parse_vnlog(const char* filename)\n{\n FILE* fp = fopen(filename);\n if(fp == NULL)\n return false;\n\n vnlog_parser_t ctx;\n if(VNL_OK != vnlog_parser_init(\u0026ctx, fp))\n return false;\n\n // String in the \"time\" column for the most-recently-parsed row\n const char*const* time_record = vnlog_parser_record_from_key(\u0026ctx, \"time\");\n if(time_record == NULL)\n {\n vnlog_parser_free(\u0026ctx);\n return false;\n }\n\n int i_record = 0;\n vnlog_parser_result_t result;\n while(VNL_OK == (result = vnlog_parser_read_record(\u0026ctx, fp)))\n {\n for(int i=0; i\u003cctx.Ncolumns; i++)\n printf(\"Record %d: %s = %s\\n\",\n i_record,\n ctx.record[i].key, ctx.record[i].value);\n printf(\"Record %d: time = %s\\n\",\n i_record, *time_record);\n i_record++;\n }\n\n vnlog_parser_free(\u0026ctx);\n return true;\n}\n#+end_src\n\nThe usage should be clear from this example. See =vnlog-parser.h= for details.\n\n** Base64 interface\nThe C interface supports writing base64-encoded binary data using Chris Venter's\nlibb64. The base64-encoder used here was slightly modified: the output appears\nall on one line, making is suitable to appear in a vnlog field. If we're writing\na vnlog with =printf()= directly without using the =vnlog.h= interface described\nabove, we allow this modified base64 encoder to be invoked by itself. Usage:\n\n#+BEGIN_SRC C\nvoid* binary_buffer = ...;\nint binary_buffer_len = ...;\n\nchar base64_buffer[vnlog_base64_dstlen_to_encode(binary_buffer_len)];\nvnlog_base64_encode( base64_buffer, sizeof(base64_buffer),\n binary_buffer, binary_buffer_len );\n#+END_SRC\n\nClearly the above example allocates the base64 buffer on the stack, so it's only\nsuitable for small-ish data chunks. But if you have lots and lots of data,\nprobably writing it as base64 into a vnlog isn't the best thing to do.\n* Python interface\nReading vnlog data into a python program is simple. The =vnlog= Python module\nprovides three different ways to do that:\n\n1. slurp the whole thing into a numpy array using the =slurp()= function. Basic\n usage:\n\n #+begin_src python\nimport vnlog\narr,list_keys,dict_key_index = \\\n vnlog.slurp(filename_or_fileobject)\n #+end_src\n\n This parses out the legend, and then calls =numpy.loadtxt()=. Null data values\n (=-=) are not supported\n\n2. Iterate through the records: =vnlog= class, used as an iterator. Basic usage:\n\n #+begin_src python\nimport vnlog\nfor d in vnlog.vnlog(filename_or_fileobject):\n print(d['time'],d['height'])\n #+end_src\n\n Null data values are represented as =None=\n\n3. Parse incoming lines individually: =vnlog= class, using the =parse()= method.\n Basic usage:\n\n #+begin_src python\nimport vnlog\nparser = vnlog.vnlog()\nfor l in file:\n parser.parse(l)\n d = parser.values_dict()\n if not d:\n continue\n print(d['time'],d['height'])\n #+end_src\n\nMost of the time you'd use options 1 or 2 above. Option 3 is the most general,\nbut also the most verbose and slowest.\n\n** Structured dtypes in slurp()\n\nThe =slurp()= example from above is simple: it doesn't specify a =dtype= (data\ntype), so =float= (64-bit IEEE-754 float) is used by default, for /all/ the data\nin the array. You can specify another =dtype=, for instance:\n\n #+begin_src python\narr,list_keys,dict_key_index = \\\n vnlog.slurp(filename_or_fileobject,\n dtype = int)\n #+end_src\n\nYou can also specify a [[https://numpy.org/doc/stable/user/basics.rec.html][/structured/ dtype]] to pull out different columns, with\ndifferent individual dtypes. Most notably, this allows interpreting some columns\nas strings. Let's say you have a very common vnlog data, such as this \"data.vnl\":\n\n#+begin_example\n# image x y z temperature\nimage1.png 1 2 5 34\nimage2.png 3 4 1 35\n#+end_example\n\nReading this with a simple =vnlog.slurp()= will fail: the filenames are not\nparseable as numerical values. We can instead do this:\n\n#+begin_src python\ndtype = np.dtype([ ('image', 'U16'),\n ('x y z', int, (3,)),\n ('temperature', float), ])\narr = vnlog.slurp(\"data.vnl\", dtype=dtype)\n#+end_src\n\nThis will read the image filename, the xyz points and the temperature into\ndifferent sub-arrays, with different types each. Accessing the result looks\nlike this:\n\n#+begin_example\nprint(arr['image'])\n---\u003e array(['image1.png', 'image2.png'], dtype='\u003cU16')\n\nprint(arr['x y z'])\n---\u003e array([[1, 2, 5],\n [3, 4, 1]])\n\nprint(arr['temperature'])\n---\u003e array([34., 35.])\n#+end_example\n\nNotes:\n\n- The given structured dtype defines both how to organize the data, and which\n data to extract. So it can be used to read in only a subset of the available\n columns. Here I could have omitted the 'temperature' column, for instance\n\n- Sub-arrays are allowed. In the example I could say either\n\n #+begin_src python\n dtype = np.dtype([ ('image', 'U16'),\n ('x y z', int, (3,)),\n ('temperature', float), ])\n #+end_src\n\n or\n\n #+begin_src python\n dtype = np.dtype([ ('image', 'U16'),\n ('x', int),\n ('y', int),\n ('z', int),\n ('temperature', float), ])\n #+end_src\n\n The latter would read =x=, =y=, =z= into separate, individual arrays. Sometime\n we want this, sometimes not.\n\n- Nested structured dtypes are not allowed. Fields inside other fields are not\n supported, since it's not clear how to map that to a flat vnlog legend\n\n- If a structured dtype is given, =slurp()= returns the array only, since the\n field names are already available in the dtype\n\n* numpy interface\nIf we need to read data into numpy specifically, nicer tools are available than\nthe generic =vnlog= Python module. The built-in =numpy.loadtxt= =numpy.savetxt=\nfunctions work well (with the caveat that =numpy.loadtxt()= should be followed\nby =numpysane.atleast_dims(..., -2)= to make sure that a data array of shape\n=(Nrows,Ncols)= is returned even if =Nrows==1=. For example to write to standard\noutput a vnlog with fields =a=, =b= and =c=:\n\n#+BEGIN_SRC python\nnumpy.savetxt(sys.stdout, array, fmt=\"%g\", header=\"a b c\")\n#+END_SRC\n\nNote that numpy automatically adds the =#= to the header. To read a vnlog from a\nfile on disk, do something like\n\n#+BEGIN_SRC python\narray = numpysane.atleast_dims(numpy.loadtxt('data.vnl'), -2)\n#+END_SRC\n\nThese functions know that =#= lines are comments, but don't interpret anything\nas field headers. That's easy to do, so I'm not providing any helper libraries.\nI might do that at some point, but in the meantime, patches are welcome.\n\n* Compatibility\n\nI use GNU/Linux-based systems exclusively, but everything has been tested\nfunctional on FreeBSD and OSX in addition to Debian, Ubuntu and CentOS. I can\nimagine there's something I missed when testing on non-Linux systems, so please\nlet me know if you find any issues.\n\n* Caveats and bugs\n\nThese tools are meant to be simple, so some things are hard requirements. A big\none is that columns are whitespace-separated. There is /no/ mechanism for\nescaping or quoting whitespace into a single field. I think supporting something\nlike that is more trouble than it's worth.\n\n* Manpages\n** vnl-filter\n#+BEGIN_EXAMPLE\nNAME\n vnl-filter - filters vnlogs to select particular rows, fields\n\nSYNOPSIS\n $ cat run.vnl\n\n # time x y z temperature\n 3 1 2.3 4.8 30\n 4 1.1 2.2 4.7 31\n 6 1 2.0 4.0 35\n 7 1 1.6 3.1 42\n\n\n $ \u003crun.vnl vnl-filter -p x,y,z | vnl-align\n\n # x y z\n 1 2.3 4.8\n 1.1 2.2 4.7\n 1 2.0 4.0\n 1 1.6 3.1\n\n\n $ \u003crun.vnl vnl-filter -p i=NR,time,'dist=sqrt(x*x + y*y + z*z)' | vnl-align\n\n # i time dist\n 1 3 5.41572\n 2 4 5.30471\n 3 6 4.58258\n 4 7 3.62905\n\n\n $ \u003crun.vnl vnl-filter 'temperature \u003e= 35' | vnl-align\n\n # time x y z temperature\n 6 1 2.0 4.0 35\n 7 1 1.6 3.1 42\n\n\n\n $ \u003crun.vnl vnl-filter --eval '{s += temperature} END { print \"mean temp: \" s/NR}'\n\n mean temp: 34.5\n\n\n $ \u003crun.vnl vnl-filter -p x,y | feedgnuplot --terminal 'dumb 80,30' --unset grid --domain --lines --exit\n\n 2.3 +---------------------------------------------------------------------+\n | + + *************** + |\n | ************** |\n | *******|\n 2.2 |-+ ************|\n | ******** |\n | ******** |\n 2.1 |-+ ********* +-|\n | ******** |\n | ******** |\n | **** |\n 2 |-+ * +-|\n | * |\n | * |\n | * |\n 1.9 |-+ * +-|\n | * |\n | * |\n | * |\n 1.8 |-+ * +-|\n | * |\n | * |\n 1.7 |-+ * +-|\n | * |\n | * |\n | * + + + + |\n 1.6 +---------------------------------------------------------------------+\n 0.98 1 1.02 1.04 1.06 1.08 1.1\n\nDESCRIPTION\n This tool is largely a frontend for awk to operate on vnlog files. Vnlog\n is both an input and an output. This tool makes it very simple to select\n specific rows and columns for output and to manipulate the data in\n various ways.\n\n This is a UNIX-style tool, so the input/output of this tool is strictly\n STDIN/STDOUT. Furthermore, in its usual form this tool is a filter, so\n the format of the output is *exactly* the same as the format of the\n input. The exception to this is when using \"--eval\", in which the output\n is dependent on whatever expression we're evaluating.\n\n This tool is convenient to process both stored data or live data; in the\n latter case, it's very useful to pipe the streaming output to\n \"feedgnuplot --stream\" to get a realtime visualization of the incoming\n data.\n\n This tool reads enough of the input file to get a legend, at which point\n it constructs an awk program to do the main work, and execs to awk (it's\n possible to use perl as well, but this isn't as fast).\n\n Input/output data format\n The input/output data is vnlog: a plain-text table of values. Any lines\n beginning with \"#\" are treated as comments, and are passed through. The\n first line that begins with \"#\" but not \"##\" or \"#!\" is a *legend* line.\n After the \"#\", follow whitespace-separated field names. Each subsequent\n line is whitespace-separated values matching this legend. For instance,\n this is a valid vnlog file:\n\n #!/usr/bin/something\n ## more comments\n # x y z\n -0.016107 0.004362 0.005369\n -0.017449 0.006711 0.006711\n -0.018456 0.014093 0.006711\n -0.017449 0.018791 0.006376\n\n \"vnl-filter\" uses this format for both the input and the output. The\n comments are preserved, but the legend is updated to reflect the fields\n in the output file.\n\n A string \"-\" is used to indicate an undefined value, so this is also a\n valid vnlog file:\n\n # x y z\n 1 2 3\n 4 - 6\n - - 7\n\n Filtering\n To select specific *columns*, pass their names to the \"-p\" option (short\n for \"--print\" or \"--pick\", which are synonyms). In its simplest form, to\n grab only columns \"x\" and \"y\", do\n\n vnl-filter -p x,y\n\n See the detailed description of \"-p\" below for more detail.\n\n To select specific *rows*, we use *matches* expressions. Anything on the\n \"vnl-filter\" commandline and not attached to any \"--xxx\" option is such\n an expression. For instance\n\n vnl-filter 'size \u003e 10'\n\n would select only those rows whose \"size\" column contains a value \u003e 10.\n See the detailed description of matches expressions below for more\n detail.\n\n Context lines\n \"vnl-filter\" supports the context output options (\"-A\", \"-B\" and \"-C\")\n exactly like the \"grep\" tool. I.e to print out all rows whose \"size\"\n column contains a value \u003e 10 *but also* include the 3 rows immediately\n before *and* after such matching rows, do this:\n\n vnl-filter -C3 'size \u003e 10'\n\n \"-B\" reports the rows *before* matching ones and \"-A\" the rows *after*\n matching ones. \"-C\" reports both. Note that this applies *only* to\n *matches* expressions: records skipped because they fail \"--has\" or\n \"--skipempty\" are *not* included in contextual output.\n\n Backend choice\n By default, the parsing of arguments and the legend happens in perl,\n which then constructs a simple awk script, and invokes \"mawk\" to\n actually read the data and to process it. This is done because awk is\n lighter weight and runs faster, which is important because our data sets\n could be quite large. We default to \"mawk\" specifically, since this is a\n simpler implementation than \"gawk\", and runs much faster. If for\n whatever reason we want to do everything with perl, this can be\n requested with the \"--perl\" option.\n\n Special functions\n For convenience we support several special functions in any expression\n passed on to awk or perl (named expressions, matches expressions,\n \"--eval\" strings). These generally maintain some internal state, and\n vnl-filter makes sure that this state is consistent. Note that these are\n evaluated *after* \"--skipcomments\" and \"--has\". So any record skipped\n because of a \"--has\" expression, for instance, will *not* be considered\n in prev(), diff() and so on.\n\n * rel(x) returns value of \"x\" relative to the first value of \"x\". For\n instance we might want to see the time or position relative to the\n start, not relative to some absolute beginning. Example:\n\n $ cat tst.vnl\n\n # time x\n 100 200\n 101 212\n 102 209\n\n\n $ \u003ctst.vnl vnl-filter -p 't=rel(time),x=rel(x)\n\n # t x\n 0 0\n 1 12\n 2 9\n\n * diff(x) returns the difference between the current value of \"x\" and\n the previous value of \"x\". The first row will always be \"-\".\n Example:\n\n $ \u003ctst.vnl vnl-filter -p x,'d1=diff(x),d2=diff(diff(x))' | vnl-align\n\n # x d1 d2\n 1 - -\n 8 7 7\n 27 19 12\n 64 37 18\n 125 61 24\n\n * sum(x) returns the cumulative sum of \"x\". As diff(x) can be thought\n of as a derivative, sum(x) can be thought of as an integral. So\n \"diff(sum(x))\" would return the same value as \"x\" (except for the\n first row; diff() always returns \"-\" for the first row).\n\n Example:\n\n $ \u003ctst.vnl vnl-filter -p 'x,s=sum(x),ds=diff(sum(x))' | vnl-align\n\n # x s ds\n 1 1 -\n 8 9 8\n 27 36 27\n 64 100 64\n 125 225 125\n\n * prev(x) returns the previous value of \"x\". One could construct sum()\n and rel() using this, if they weren't already available.\n\n * latestdefined(x) returns the most recent value of \"x\" that isn't\n \"-\". If \"x\" isn't \"-\", this simply returns \"x\".\n\nARGUMENTS\n Matches expressions\n Anything on the commandline not attached to any \"--xxx\" option is a\n *matches* expression. These are used to select particular records (rows)\n in a data file. For each row, we evaluate all the expressions. If *all*\n the expressions evaluate to true, that row is output. This expression is\n passed directly to the awk (or perl) backend.\n\n Example: to select all rows that have valid data in column \"a\" *or*\n column \"b\" *or* column \"c\" you can\n\n vnl-filter 'a != \"-\" || b != \"-\" || c != \"-\"'\n\n or\n\n vnl-filter --perl 'defined a || defined b || defined c'\n\n As with the named expressions given to \"-p\" (described above), these are\n passed directly to awk, so anything that can be done with awk is\n supported here.\n\n -p|--print|--pick expr\n These option provide the mechanism to select specific columns for\n output. For instance to pull out columns called \"lat\", \"lon\", and any\n column whose name contains the string \"feature_\", do\n\n vnl-filter -p lat,lon,'feature_.*'\n\n or, equivalently\n\n vnl-filter --print lat --print lon --print 'feature_.*'\n\n We look for exact column name matches first, and if none are found, we\n try a regex. If there was no column called exactly \"feature_\", then the\n above would be equivalent to\n\n vnl-filter -p lat,lon,feature_\n\n This mechanism is much more powerful than just selecting columns. First\n off, we can rename chosen fields:\n\n vnl-filter -p w=feature_width\n\n would pick the \"feature_width\" field, but the resulting column in the\n output would be named \"w\". When renaming a column in this way regexen\n are *not* supported, and exact field names must be given. But the string\n to the right of the \"=\" is passed on directly to awk (after replacing\n field names with column indices), so any awk expression can be used\n here. For instance to compute the length of a vector in separate columns\n \"x\", \"y\", and \"z\" you can do:\n\n vnl-filter -p 'l=sqrt(x*x + y*y + z*z)'\n\n A single column called \"l\" would be produced.\n\n We can also *exclude* columns by preceding their name with \"!\". This\n works like you expect. Rules:\n\n * The pick/exclude directives are processed in order given to produce\n the output picked-column list\n\n * If the first \"-p\" item is an exclusion, we implicitly pick *all* the\n columns prior to processing the \"-p\".\n\n * The exclusion expressions match the *output* column names, not the\n *input* names.\n\n * We match the exact column names first. If that fails, we match as a\n regex\n\n Example. To grab all the columns *except* the temperature(s) do this:\n\n vnl-filter -p !temperature\n\n To grab all the columns that describe *something* about a robot (columns\n whose names have the string \"robot_\" in them), but *not* its temperature\n (i.e. *not* \"robot_temperature\"), do this:\n\n vnl-filter -p robot_,!temperature\n\n --has a,b,c,...\n Used to select records (rows) that have a non-empty value in a\n particular field (column). A *null* value in a column is designated with\n a single \"-\". If we want to select only records that have a value in the\n \"x\" column, we pass \"--has x\". To select records that have data for\n *all* of a given set of columns, the \"--has\" option can be repeated, or\n these multiple columns can be given in a whitespace-less comma-separated\n list. For instance if we want only records that have data in *both*\n columns \"x\" *and* \"y\" we can pass in \"--has x,y\" or \"--has x --has y\".\n If we want to combine multiple columns in an *or* (select rows that have\n data in *any* of a given set of columns), use a matches expression, as\n documented below.\n\n If we want to select a column *and* pick only rows that have a value in\n this column, a shorthand syntax exists:\n\n vnl-filter --has col -p col\n\n is equivalent to\n\n vnl-filter -p +col\n\n Note that just like the column specifications in \"-p\" the columns given\n to \"--has\" must match exactly *or* as a regex. In either case, a unique\n matching column must be found.\n\n -l|list-columns\n Instead of doing any processing, parse the input to get the available\n columns, print those out, and exit\n\n -A N|--after-context N\n Output N lines following each *matches* expression, even those lines\n that do not themselves match. This works just like the \"grep\" options of\n the same name. See \"Context lines\"\n\n -B N|--before-context N\n Output N lines preceding each *matches* expression, even those lines\n that do not themselves match. This works just like the \"grep\" options of\n the same name. See \"Context lines\"\n\n -C N|--context N\n Output N lines preceding and following each *matches* expression, even\n those lines that do not themselves match. This works just like the\n \"grep\" options of the same name. See \"Context lines\"\n\n --eval expr\n Instead of printing out all matching records and picked columns, just\n run the given chunk of awk (or perl). In this mode of operation,\n \"vnl-filter\" acts just like a glorified awk, that allows fields to be\n accessed by name instead of by number, as it would be in raw awk.\n\n Since the expression may print *anything* or nothing at all, the output\n in this mode is not necessarily itself a valid vnlog stream. And no\n column-selecting arguments should be given, since they make no sense in\n this mode.\n\n In awk the expr is a full set of pattern/action statements. So to print\n the sum of columns \"a\" and \"b\" in each row, and at the end, print the\n sum of all values in the \"a\" column\n\n vnl-filter --eval '{print a+b; suma += a} END {print suma}'\n\n In perl the arbitrary expression fits in like this:\n\n sub evalexpr\n {\n eval expression; # evaluate the arbitrary expression\n }\n while(\u003c\u003e) # read each line\n {\n chomp;\n next unless matches; # skip non-matching lines\n evalexpr();\n }\n\n --function|--sub\n Evaluates the given expression as a function that can be used in other\n expressions. This is most useful when you want to print something that\n can't trivially be written as a simple expression. For instance:\n\n $ cat tst.vnl\n # s\n 1-2\n 3-4\n 5-6\n\n $ \u003c tst.vnl\n vnl-filter --function 'before(x) { sub(\"-.*\",\"\",x); return x }' \\\n --function 'after(x) { sub(\".*-\",\"\",x); return x }' \\\n -p 'b=before(s),a=after(s)'\n # b a\n 1 2\n 3 4\n 5 6\n\n See the CAVEATS section below if you're doing something\n sufficiently-complicated where you need this.\n\n --function-abs|--sub-abs\n Convenience option to add an absolute-value abs() function. This is only\n useful for awk programs (the default, no \"--perl\" given) since perl\n already provides abs() by default.\n\n --begin|--BEGIN\n Evaluates the given expression in the BEGIN {} block of the generated\n awk (or perl) program.\n\n --end|--END\n Evaluates the given expression in the END {} block of the generated awk\n (or perl) program.\n\n --[no]skipempty\n Do [not] skip records where all fields are blank. By default we *do*\n skip all empty records; to include them, pass \"--noskipempty\"\n\n --skipcomments\n Don't output non-legend comments\n\n --perl\n By default all procesing is performed by \"mawk\", but if for whatever\n reason we want perl instead, pass \"--perl\". Both modes work, but \"mawk\"\n is noticeably faster. \"--perl\" could be useful because it is more\n powerful, which could be important since a number of things pass\n commandline strings directly to the underlying language (named\n expressions, matches expressions, \"--eval\" strings). Note that while\n variables in perl use sigils, column references should *not* use sigils.\n To print the sum of all values in column \"a\" you'd do this in awk\n\n vnl-filter --eval '{suma += a} END {print suma}'\n\n and this in perl\n\n vnl-filter --perl --eval '{$suma += a} END {say $suma}'\n\n The perl strings are evaluated without \"use strict\" or \"use warnings\" so\n I didn't have to declare $suma in the example.\n\n With \"--perl\", empty strings (\"-\" in the vnlog file) are converted to\n \"undef\".\n\n --dumpexprs\n Used for debugging. This spits out all the final awk (or perl) program\n we run for the given commandline options and given input. This is the\n final program, with the column references resolved to numeric indices,\n so one can figure out what went wrong.\n\n --unbuffered\n Flushes each line after each print. This makes sure each line is output\n as soon as it is available, which is crucial for realtime output and\n streaming plots.\n\n --stream\n Synonym for \"--unbuffered\"\n\nCAVEATS\n This tool is very lax in its input validation (on purpose). As a result,\n columns with names like %CPU and \"TIME+\" do work (i.e. you can more or\n less feed in output from \"top -b\"). The downside is that shooting\n yourself in the foot is possible. This tradeoff is currently tuned to be\n very permissive, which works well for my use cases. I'd be interested in\n hearing other people's experiences. Potential pitfalls/unexpected\n behaviors:\n\n * All column names are replaced in all eval strings without regard to\n context. The earlier example that reports the sum of values in a\n column: vnl-filter --eval '{suma += a} END {print suma}' will work\n fine if we *do* have a column named \"a\" and do *not* have a column\n named \"suma\". But will not do the right thing if any of those are\n violated. For instance, if a column \"a\" doesn't exist, then \"awk\"\n would see \"suma += a\" instead of something like \"suma += $5\". \"a\"\n would be an uninitialized variable, which evaluates to 0, so the\n full \"vnl-filter\" command would not fail, but would print 0 instead.\n It's the user's responsibility to make sure we're talking about the\n right columns. The focus here was one-liners so hopefully nobody has\n so many columns, they can't keep track of all of them in their head.\n I don't see any way to resolve this without seriously impacting the\n scope of the tool, so I'm leaving this alone.\n\n * It is natural to use vnlog as a database. You can run queries with\n something like\n\n vnl-filter 'key == 5'\n\n This works. But unlike a real database this is clearly a linear\n lookup. With large data files, this would be significantly slower\n than the logarithmic searches provided by a real database. The\n meaning of \"large\" and \"significant\" varies, and you should test it.\n In my experience vnlog \"databases\" scale surprisingly well. But at\n some point, importing your data to something like sqlite is well\n worth it.\n\n * When substituting column names I match *either* a word-nonword\n transition (\"\\b\") *or* a whitespace-nonword transition. The word\n boundaries is what would be used 99% of the time. But the keys may\n have special characters in them, which don't work with \"\\b\". This\n means that whitespace becomes important: \"1+%CPU\" will not be parsed\n as expected, which is correct since \"+%CPU\" is also a valid field\n name. But \"1+ %CPU\" will be parsed correctly, so if you have weird\n field names, put the whitespace into your expressions. It'll make\n them more readable anyway.\n\n * Strings passed to \"-p\" are split on \",\" *except* if the \",\" is\n inside balanced \"()\". This makes it possible to say things like\n vnl-filter --function 'f(a,b) { ... }' -p 'c=f(a,b)'. This is\n probably the right behavior, although some questionable looking\n field names become potentially impossible: \"f(a\" and \"b)\" *could*\n otherwise be legal field names, but you're probably asking for\n trouble if you do that.\n\n * Currently there're two modes: a pick/print mode and an \"--eval\"\n mode. Then there's also \"--function\", which adds bits of \"--eval\" to\n the pick/print mode, but it feels maybe insufficient. I don't yet\n have strong feelings about what this should become. Comments welcome\n\n\n#+END_EXAMPLE\n\n** vnl-align\n#+BEGIN_EXAMPLE\nNAME\n vnl-align - aligns vnlog columns for easy interpretation by humans\n\nSYNOPSIS\n $ cat tst.vnl\n\n # w x y z\n -10 40 asdf -\n -20 50 - 0.300000\n -30 10 whoa 0.500000\n\n\n $ vnl-align tst.vnl\n\n # w x y z\n -10 40 asdf -\n -20 50 - 0.300000\n -30 10 whoa 0.500000\n\nDESCRIPTION\n The basic usage is\n\n vnl-align logfile\n\n The arguments are assumed to be the vnlog files. If no arguments are\n given, the input comes from STDIN.\n\n This is very similar to \"column -t\", but handles \"#\" lines properly:\n\n 1. The first \"#\" line is the legend. For the purposes of alignment, the\n leading \"#\" character and the first column label are treated as one\n column\n\n 2. All other \"#\" lines are output verbatim.\n\n\n#+END_EXAMPLE\n\n** vnl-sort\n#+BEGIN_EXAMPLE\nNAME\n vnl-sort - sorts an vnlog file, preserving the legend\n\nSYNOPSIS\n $ cat a.vnl\n # a b\n AA 11\n bb 12\n CC 13\n dd 14\n dd 123\n\n Sort lexically by a:\n $ \u003ca.vnl vnl-sort -k a\n # a b\n AA 11\n CC 13\n bb 12\n dd 123\n dd 14\n\n Sort lexically by a, ignoring case:\n $ \u003ca.vnl vnl-sort -k a --ignore-case\n # a b\n AA 11\n bb 12\n CC 13\n dd 123\n dd 14\n\n Sort lexically by a, then numerically by b:\n $ \u003ca.vnl vnl-sort -k a -k b.n\n # a b\n AA 11\n CC 13\n bb 12\n dd 14\n dd 123\n\n Sort lexically by a, then numerically by b in reverse:\n $ \u003ca.vnl vnl-sort -k a -k b.nr\n # a b\n AA 11\n CC 13\n bb 12\n dd 123\n dd 14\n\n\n Sort by month and then day:\n $ cat dat.vnl\n # month day\n March 5\n Jan 2\n Feb 1\n March 30\n Jan 21\n\n $ \u003cdat.vnl vnl-sort -k month.M -k day.n\n # month day\n Jan 2\n Jan 21\n Feb 1\n March 5\n March 30\n\nDESCRIPTION\n Usage: vnl-sort [options] logfile logfile logfile ... \u003c logfile\n\n This tool sorts given vnlog files in various ways. \"vnl-sort\" is a\n wrapper around the GNU coreutils \"sort\" tool. Since this is a wrapper,\n most commandline options and behaviors of the \"sort\" tool are present;\n consult the sort(1) manpage for detail. The differences from GNU\n coreutils \"sort\" are\n\n * The input and output to this tool are vnlog files, complete with a\n legend\n\n * The columns are referenced by name, not index. So instead of saying\n\n sort -k1\n\n to sort by the first column, you say\n\n sort -k time\n\n to sort by column \"time\".\n\n * The fancy \"KEYDEF\" spec from \"sort\" is only partially supported. I\n only allow us to sort by full *fields*, so the start/stop positions\n don't make sense. I *do* support the \"OPTS\" to change the type of\n sorting in a given particular column. For instance, to sort by month\n and then by day, do this (see example above):\n\n vnl-sort -k month.M -k day.n\n\n * \"--files0-from\" is not supported due to lack of time. If somebody\n really needs it, talk to me.\n\n * \"--output\" is not supported due to an uninteresting technical\n limitation. The output always goes to standard out.\n\n * \"--field-separator\" is not supported because vnlog assumes\n whitespace-separated fields\n\n * \"--zero-terminated\" is not supported because vnlog assumes\n newline-separated records\n\n * By default we call the \"sort\" tool to do the actual work. If the\n underlying tool has a different name or lives in an odd path, this\n can be specified by passing \"--vnl-tool TOOL\"\n\n Past that, everything \"sort\" does is supported, so see that man page for\n detailed documentation. Note that all non-legend comments are stripped\n out, since it's not obvious where they should end up.\n\nCOMPATIBILITY\n I use GNU/Linux-based systems exclusively, but everything has been\n tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and\n CentOS. I can imagine there's something I missed when testing on\n non-Linux systems, so please let me know if you find any issues.\n\nSEE ALSO\n sort(1)\n\n\n#+END_EXAMPLE\n\n** vnl-join\n#+BEGIN_EXAMPLE\nNAME\n vnl-join - joins two log files on a particular field\n\nSYNOPSIS\n $ cat a.vnl\n # a b\n AA 11\n bb 12\n CC 13\n dd 14\n dd 123\n\n $ cat b.vnl\n # a c\n aa 1\n cc 3\n bb 4\n ee 5\n - 23\n\n Try to join unsorted data on field 'a':\n $ vnl-join -j a a.vnl b.vnl\n # a b c\n join: /dev/fd/5:3: is not sorted: CC 13\n join: /dev/fd/6:3: is not sorted: bb 4\n\n Sort the data, and join on 'a':\n $ vnl-join --vnl-sort - -j a a.vnl b.vnl | vnl-align\n # a b c\n bb 12 4\n\n Sort the data, and join on 'a', ignoring case:\n $ vnl-join -i --vnl-sort - -j a a.vnl b.vnl | vnl-align\n # a b c\n AA 11 1\n bb 12 4\n CC 13 3\n\n Sort the data, and join on 'a'. Also print the unmatched lines from both files:\n $ vnl-join -a1 -a2 --vnl-sort - -j a a.vnl b.vnl | vnl-align\n # a b c\n - - 23\n AA 11 -\n CC 13 -\n aa - 1\n bb 12 4\n cc - 3\n dd 123 -\n dd 14 -\n ee - 5\n\n Sort the data, and join on 'a'. Print the unmatched lines from both files,\n Output ONLY column 'c' from the 2nd input:\n $ vnl-join -a1 -a2 -o 2.c --vnl-sort - -j a a.vnl b.vnl | vnl-align\n # c\n 23\n -\n -\n 1\n 4\n 3\n -\n -\n 5\n\nDESCRIPTION\n Usage: vnl-join [join options]\n [--vnl-sort -|[sdfgiMhnRrV]+]\n [ --vnl-[pre|suf]fix[1|2] xxx |\n --vnl-[pre|suf]fix xxx,yyy,zzz |\n --vnl-autoprefix |\n --vnl-autosuffix ]\n logfile1 logfile2\n\n This tool joins two vnlog files on a given field. \"vnl-join\" is a\n wrapper around the GNU coreutils \"join\" tool. Since this is a wrapper,\n most commandline options and behaviors of the \"join\" tool are present;\n consult the join(1) manpage for detail. The differences from GNU\n coreutils \"join\" are\n\n * The input and output to this tool are vnlog files, complete with a\n legend\n\n * The columns are referenced by name, not index. So instead of saying\n\n join -j1\n\n to join on the first column, you say\n\n join -j time\n\n to join on column \"time\".\n\n * -1 and -2 are supported, but *must* refer to the same field. Since\n vnlog knows the identify of each field, it makes no sense for -1 and\n -2 to be different. So pass \"-j\" instead, it makes more sense in\n this context.\n\n * \"-a-\" is available as a shorthand for \"-a1 -a2\": this is a full\n outer join, printing unmatched records from both of the inputs.\n Similarly, \"-v-\" is available as a shorthand for \"-v1 -v2\": this\n will output *only* the unique records in both of the inputs.\n\n * \"vnl-join\"-specific options are available to adjust the field-naming\n in the output:\n\n --vnl-prefix1\n --vnl-suffix1\n --vnl-prefix2\n --vnl-suffix2\n --vnl-prefix\n --vnl-suffix\n --vnl-autoprefix\n --vnl-autosuffix\n\n See \"Field names in the output\" below for details.\n\n * A \"vnl-join\"-specific option \"--vnl-sort\" is available to sort the\n input and/or output. See below for details.\n\n * By default we call the \"join\" tool to do the actual work. If the\n underlying tool has a different name or lives in an odd path, this\n can be specified by passing \"--vnl-tool TOOL\"\n\n * If no \"-o\" is given, we output the join field, the remaining fields\n in logfile1, the remaining fields in logfile2, .... This is what \"-o\n auto\" does, except we also handle empty vnlogs correctly.\n\n * \"-e\" is not supported because vnlog uses \"-\" to represent undefined\n fields.\n\n * \"--header\" is not supported because vnlog assumes a specific header\n structure, and \"vnl-join\" makes sure that this header is handled\n properly\n\n * \"-t\" is not supported because vnlog assumes whitespace-separated\n fields\n\n * \"--zero-terminated\" is not supported because vnlog assumes\n newline-separated records\n\n * Rather than only 2-way joins, this tool supports N-way joins for any\n N \u003e 2. See below for details.\n\n Past that, everything \"join\" does is supported, so see that man page for\n detailed documentation. Note that all non-legend comments are stripped\n out, since it's not obvious where they should end up.\n\n Field names in the output\n By default, the field names in the output match those in the input. This\n is what you want most of the time. It is possible, however that a column\n name adjustment is needed. One common use case for this is if the files\n being joined have identically-named columns, which would produce\n duplicate columns in the output. Example: we fixed a bug in a program,\n and want to compare the results before and after the fix. The program\n produces an x-y trajectory as a function of time, so both the bugged and\n the bug-fixed programs produce a vnlog with a legend\n\n # time x y\n\n Joining this on \"time\" will produce a vnlog with a legend\n\n # time x y x y\n\n which is confusing, and *not* what you want. Instead, we invoke\n \"vnl-join\" as\n\n vnl-join --vnl-suffix1 _buggy --vnl-suffix2 _fixed -j time buggy.vnl fixed.vnl\n\n And in the output we get a legend\n\n # time x_buggy y_buggy x_fixed y_fixed\n\n Much better.\n\n Note that \"vnl-join\" provides several ways of specifying this. The above\n works *only* for 2-way joins. An alternate syntax is available for N-way\n joins, a comma-separated list. The same could be expressed like this:\n\n vnl-join -a- --vnl-suffix _buggy,_fixed -j time buggy.vnl fixed.vnl\n\n Finally, if passing in structured filenames, \"vnl-join\" can infer the\n desired syntax from the filenames. The same as above could be expressed\n even simpler:\n\n vnl-join --vnl-autosuffix -j time buggy.vnl fixed.vnl\n\n This works by looking at the set of passed in filenames, and stripping\n out the common leading and trailing strings.\n\n Sorting of input and output\n The GNU coreutils \"join\" tool expects sorted columns because it can then\n take only a single pass through the data. If the input isn't sorted,\n then we can use normal shell substitutions to sort it:\n\n $ vnl-join -j key \u003c(vnl-sort -s -k key a.vnl) \u003c(vnl-sort -s -k key b.vnl)\n\n For convenience \"vnl-join\" provides a \"--vnl-sort\" option. This allows\n the above to be equivalently expressed as\n\n $ vnl-join -j key --vnl-sort - a.vnl b.vnl\n\n The \"-\" after the \"--vnl-sort\" indicates that we want to sort the\n *input* only. If we also want to sort the output, pass the short codes\n \"sort\" accepts instead of the \"-\". For instance, to sort the input for\n \"join\" and to sort the output numerically, in reverse, do this:\n\n $ vnl-join -j key --vnl-sort rg a.vnl b.vnl\n\n The reason this shorthand exists is to work around a quirk of \"join\".\n The sort order is *assumed* by \"join\" to be lexicographical, without any\n way to change this. For \"sort\", this is the default sort order, but\n \"sort\" has many options to change the sort order, options which are\n sorely missing from \"join\". A real-world example affected by this is the\n joining of numerical data. If you have \"a.vnl\":\n\n # time a\n 8 a\n 9 b\n 10 c\n\n and \"b.vnl\":\n\n # time b\n 9 d\n 10 e\n\n Then you cannot use \"vnl-join\" directly to join the data on time:\n\n $ vnl-join -j time a.vnl b.vnl\n # time a b\n join: /dev/fd/4:3: is not sorted: 10 c\n join: /dev/fd/5:2: is not sorted: 10 e\n 9 b d\n 10 c e\n\n Instead you must re-sort both files lexicographically, *and* then\n (because you almost certainly want to) sort it back into numerical\n order:\n\n $ vnl-join -j time \u003c(vnl-sort -s -k time a.vnl) \u003c(vnl-sort -s -k time b.vnl) |\n vnl-sort -s -n -k time\n # time a b\n 9 b d\n 10 c e\n\n Yuck. The shorthand described earlier makes the interface part of this\n palatable:\n\n $ vnl-join -j time --vnl-sort n a.vnl b.vnl\n # time a b\n 9 b d\n 10 c e\n\n Note that the input sort is stable: \"vnl-join\" will invoke \"vnl-sort\n -s\". If you want a stable post-sort, you need to ask for it with\n \"--vnl-sort s...\".\n\n N-way joins\n The GNU coreutils \"join\" tool is inherently designed to join *exactly*\n two files. \"vnl-join\" extends this capability by chaining together a\n number of \"join\" invocations to produce a generic N-way join. This works\n exactly how you would expect with the following caveats:\n\n * Full outer joins are supported by passing \"-a-\", but no other \"-a\"\n option is supported. This is possible, but wasn't obviously worth\n the trouble.\n\n * \"-v\" is not supported. Again, this is possible, but wasn't obviously\n worth the trouble.\n\n * Similarly, \"-o\" is not supported. This is possible, but wasn't\n obviously worth the trouble, especially since the desired behavior\n can be obtained by post-processing with \"vnl-filter\".\n\nBUGS AND CAVEATS\n The underlying \"sort\" tool assumes lexicographic ordering, and matches\n fields purely based on their textual contents. This means that for the\n purposes of joining, 10, 10.0 and 1.0e1 are all considered different. If\n needed, you can normalize your keys with something like this:\n\n vnl-filter -p x='sprintf(\"%f\",x)'\n\nCOMPATIBILITY\n I use GNU/Linux-based systems exclusively, but everything has been\n tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and\n CentOS. I can imagine there's something I missed when testing on\n non-Linux systems, so please let me know if you find any issues.\n\nSEE ALSO\n join(1)\n\n\n#+END_EXAMPLE\n\n** vnl-tail\n#+BEGIN_EXAMPLE\nNAME\n vnl-tail - tail a log file, preserving the legend\n\nSYNOPSIS\n $ read_temperature | tee temp.vnl\n # temperature\n 29.5\n 30.4\n 28.3\n 22.1\n ... continually produces data\n\n ... at the same time, in another terminal\n $ vnl-tail -f temp.vnl\n # temperature\n 28.3\n 22.1\n ... outputs data as it comes in\n\nDESCRIPTION\n Usage: vnl-tail [options] logfile logfile logfile ... \u003c logfile\n\n This tool runs \"tail\" on given vnlog files in various ways. \"vnl-tail\"\n is a wrapper around the GNU coreutils \"tail\" tool. Since this is a\n wrapper, most commandline options and behaviors of the \"tail\" tool are\n present; consult the tail(1) manpage for detail. The differences from\n GNU coreutils \"tail\" are\n\n * The input and output to this tool are vnlog files, complete with a\n legend\n\n * \"-c\" is not supported because vnlog really doesn't want to break up\n lines\n\n * \"--zero-terminated\" is not supported because vnlog assumes\n newline-separated records\n\n * By default we call the \"tail\" tool to do the actual work. If the\n underlying tool has a different name or lives in an odd path, this\n can be specified by passing \"--vnl-tool TOOL\"\n\n Past that, everything \"tail\" does is supported, so see that man page for\n detailed documentation.\n\nCOMPATIBILITY\n I use GNU/Linux-based systems exclusively, but everything has been\n tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and\n CentOS. I can imagine there's something I missed when testing on\n non-Linux systems, so please let me know if you find any issues.\n\nSEE ALSO\n tail(1)\n\n\n#+END_EXAMPLE\n\n** vnl-ts\n#+BEGIN_EXAMPLE\nNAME\n vnl-ts - add a timestamp to a vnlog stream\n\nSYNOPSIS\n $ read_temperature\n # temperature\n 29.5\n 30.4\n 28.3\n 22.1\n ... continually produces data at 1Hz\n\n $ read_temperature | vnl-ts -s %.s\n # time-rel temperature\n 0.013893 30.2\n 1.048695 28.6\n 2.105592 29.3\n 3.162873 22.0\n ...\n\nDESCRIPTION\n Usage: vnl-ts [-i | -s] [-m] [--vnl-field t] format \u003c pipe\n\n This tool runs \"ts\" on given vnlog streams. \"vnl-ts\" is a wrapper around\n the \"ts\" tool from Joey Hess's moreutils\n \u003chttps://joeyh.name/code/moreutils/\u003e toolkit. Since this is a wrapper,\n most commandline options and behaviors of the \"ts\" tool are present;\n consult the ts(1) manpage for details. The differences from \"ts\" are\n\n * The input and output to this tool are vnlog files, complete with a\n legend\n\n * The format *must* be passed-in by the user; no default is assumed.\n\n * The given format *must not* contain whitespace, so that it fits a\n single vnlog field.\n\n * \"-r\" is not supported: it assumes input timestamps with whitespace,\n which is incompatible with vnlog\n\n * A \"vnl-ts\"-specific option \"--vnl-field\" is available to set the\n name of the new field. If omitted, a reasonable default will be\n used.\n\n * By default we call the \"ts\" tool to do the actual work. If the\n underlying tool has a different name or lives in an odd path, this\n can be specified by passing \"--vnl-tool TOOL\"\n\n Past that, everything \"ts\" does is supported, so see that man page for\n detailed documentation.\n\nCOMPATIBILITY\n By default this calls the tool named \"ts\". At least on FreeBSD, it's\n called \"moreutils-ts\", so on such systems you should invoke \"vnl-ts\n --vnl-tool moreutils-ts ...\"\n\n I use GNU/Linux-based systems exclusively, but everything has been\n tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and\n CentOS. I can imagine there's something I missed when testing on\n non-Linux systems, so please let me know if you find any issues.\n\nSEE ALSO\n ts(1)\n\n\n#+END_EXAMPLE\n\n** vnl-uniq\n#+BEGIN_EXAMPLE\nNAME\n vnl-uniq - uniq a log file, preserving the legend\n\nSYNOPSIS\n $ cat colors.vnl\n # color\n blue\n yellow\n yellow\n blue\n yellow\n orange\n orange\n\n $ \u003c colors.vnl | vnl-sort | vnl-uniq -c\n # count color\n 2 blue\n 2 orange\n 3 yellow\n\nDESCRIPTION\n Usage: vnl-uniq [options] \u003c logfile\n\n This tool runs \"uniq\" on a given vnlog dataset. \"vnl-uniq\" is a wrapper\n around the GNU coreutils \"uniq\" tool. Since this is a wrapper, most\n commandline options and behaviors of the \"uniq\" tool are present;\n consult the uniq(1) manpage for detail. The differences from GNU\n coreutils \"uniq\" are\n\n * The input and output to this tool are vnlog files, complete with a\n legend\n\n * \"--zero-terminated\" is not supported because vnlog assumes\n newline-separated records\n\n * Only *one* input is supported (a file on the cmdline or data on\n standard input), and the output *always* goes to standard output.\n Specifying the output as a file on the commandline is not supported.\n\n * \"--vnl-count NAME\" can be given to name the \"count\" column. \"-c\" is\n still supported to add the default new column named \"count\", but if\n another name is wanted, \"--vnl-count\" does that. \"--vnl-count\"\n implies \"-c\"\n\n * In addition to the normal behavior of skipping fields at the start,\n \"-f\" and \"--skip-fields\" can take a negative argument to skip the\n *all but the last* N fields. For instance, to use only the one last\n field, pass \"-f -1\" or \"--skip-fields=-1\".\n\n * By default we call the \"uniq\" tool to do the actual work. If the\n underlying tool has a different name or lives in an odd path, this\n can be specified by passing \"--vnl-tool TOOL\"\n\n Past that, everything \"uniq\" does is supported, so see that man page for\n detailed documentation.\n\nCOMPATIBILITY\n I use GNU/Linux-based systems exclusively, but everything has been\n tested functional on FreeBSD and OSX in addition to Debian, Ubuntu and\n CentOS. I can imagine there's something I missed when testing on\n non-Linux systems, so please let me know if you find any issues.\n\nSEE ALSO\n uniq(1)\n\n\n#+END_EXAMPLE\n\n** vnl-gen-header\n#+BEGIN_EXAMPLE\nNAME\n vnl-gen-header - create definition for vnlog output from C\n\nSYNOPSIS\n $ vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' \u003e vnlog_fields_generated.h\n\nDESCRIPTION\n We provide a simple C library to produce vnlog output. The fields this\n library outputs must be known at compile time, and are specified in a\n header created by this tool. Please see the vnlog documentation for\n instructions on how to use the library\n\nARGUMENTS\n This tool needs to be given a list of field definitions. First we look\n at the commandline, and if the definitions are not available there, we\n look on STDIN. Each definition is a string \"type name\" (one def per\n argument on the commandline or per line on STDIN). If reading from\n STDIN, we ignore blank lines, and treat any line starting with \"#\" as a\n comment.\n\n Each def represents a single output field. Each such field spec in a\n C-style variable declaration with a type followed by a name. Note that\n these field specs contain whitespace, so each one must be quoted before\n being passed to the shell.\n\n The types can be basic scalars, possibly with set widths (\"char\",\n \"double\", \"int\", \"uint32_t\", \"unsigned int\", ...), a NULL-terminated\n string (\"char*\") or a generic chunk of binary data (\"void*\").\n\n The names must consist entirely of letters, numbers or \"_\", like\n variables in C.\n\n\n#+END_EXAMPLE\n\n** vnl-make-matrix\n#+BEGIN_EXAMPLE\nNAME\n vnl-make-matrix - create a matrix from a one-point-per-record vnlog\n\nSYNOPSIS\n $ cat /tmp/dat.vnl\n # i j x\n 0 0 1\n 0 1 2\n 0 2 3\n 1 0 4\n 1 1 5\n 1 2 6\n 2 0 7\n 2 1 8\n 2 2 9\n 3 0 10\n 3 1 11\n 3 2 12\n\n $ \u003c/tmp/dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp --prefix test_\n Writing to '/tmp/test_x.matrix'\n\n $ cat /tmp/test_x.matrix\n 1 2 3\n 4 5 6\n 7 8 9\n 10 11 12\n\nDESCRIPTION\n Vnlog represents each \"data item\" as a line of text. This is very often\n what one wants, but at times it isn't. One example of this is matrix\n data: we want each line to contain a whole row of a matrix. This script\n exists for convenience, to bridge this gap.\n\n The input is an vnlog, coming in on STDIN and/or in files given on the\n commandline. This vnlog must have at least two fields: the\n least-significant (slowest-changing) index of each point (must be the\n *first* field), and as many value fields as desired. These points must\n be written out in order, and it is assumed that all entries in the\n matrix are specified. The output is a set of (non-vnlog) matrix files in\n the directory given in the \"--outdir\" argument. These files are named\n \"PREFIX_XXX.matrix\" where \"PREFIX\" comes from --prefix (or empty) and\n \"XXX\" is the field name. These matrices can be loaded into any analysis\n tool (numpy for instance), or plotted directly with gnuplot:\n\n set size ratio -1\n plot \"/tmp/test_x.matrix\" matrix with image\n pause -1\n\n\n#+END_EXAMPLE\n\n* Repository\n\nhttps://github.com/dkogan/vnlog/\n\n* Authors\n\nDima Kogan (=dima@secretsauce.net=) wrote this toolkit for his work at the Jet\nPropulsion Laboratory, and is delighted to have been able to release it\npublically\n\nChris Venter (=chris.venter@gmail.com=) wrote the base64 encoder\n\n* License and copyright\n\nThis library is free software; you can redistribute it and/or modify it under\nthe terms of the GNU Lesser General Public License as published by the Free\nSoftware Foundation; either version 2.1 of the License, or (at your option) any\nlater version.\n\nCopyright 2016-2017 California Institute of Technology\n\nCopyright 2017-2018 Dima Kogan (=dima@secretsauce.net=)\n\n=b64_cencode.c= comes from =cencode.c= in the =libb64= project. It is written by\nChris Venter (=chris.venter@gmail.com=) who placed it in the public domain. The\nfull text of the license is in that file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkogan%2Fvnlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdkogan%2Fvnlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdkogan%2Fvnlog/lists"}