{"id":20188011,"url":"https://github.com/arran4/pimtrace","last_synced_at":"2026-06-06T22:31:56.503Z","repository":{"id":62774583,"uuid":"547670958","full_name":"arran4/pimtrace","owner":"arran4","description":"A CLI tool for preforming queries on ical, maildir, mbox and csv files","archived":false,"fork":false,"pushed_at":"2024-06-26T19:33:55.000Z","size":393,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-13T18:24:37.207Z","etag":null,"topics":["cli","csv","ical","query","tool"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arran4.png","metadata":{"files":{"readme":"readme.MD","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-08T04:22:32.000Z","updated_at":"2024-08-17T09:13:42.000Z","dependencies_parsed_at":"2024-11-14T03:27:45.763Z","dependency_job_id":"e21f40c5-3bd4-4ad2-89cc-2b2a10d73bb7","html_url":"https://github.com/arran4/pimtrace","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arran4%2Fpimtrace","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arran4%2Fpimtrace/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arran4%2Fpimtrace/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arran4%2Fpimtrace/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arran4","download_url":"https://codeload.github.com/arran4/pimtrace/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241622613,"owners_count":19992504,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","csv","ical","query","tool"],"created_at":"2024-11-14T03:26:59.059Z","updated_at":"2025-03-03T07:12:35.597Z","avatar_url":"https://github.com/arran4.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PIMTrace tools\n\nThese are a series of tools that allow the querying, filtering and summarizing\nof data on the command line in one-liners. \n\nCurrent supported formats:\n* ICal\n* Mbox\n* Mail file\n* CSV\n\nMore might be coming.\n\nThis is a work in progress (I need it for something else) however I will develop\nit to meet my itches. I am happy to accept PRs of reasonable addition and changes.\nI am a \"batteries included\" person so additions are appreciated.\n\n## Uses?\n\n* Filtering and sorting CSV files\n* Querying ical and mail files for specific or statistical information\n\n# Usage\n\nCurrently, there are 3 main tools provided by this:\n* `icaltrace` - for ical. This was thrown in quickly\n* `mailtrace` - For mailfiles and mbox files at the moment\n* `csvtrace` - For tabular data, namely CSV files\n\n# Query Language\n\n## Basic\n\nThere are a couple of things you can do:\n* Filter out entries\n* Sort the entries\n* Output it in\n  * The original format\n  * A new format, such as\n    * Tabular (csv)\n* Group by columns\n* Convert columns using a function\n\nThe ways you do it are with the following constructs:\n* `sort ...`\n* `filter ...` \u003c-- Filters out rows\n* `into table ...` \u003c-- Converts to tabular data and/or filters out columns and/or transforms columns using functions \n* `into summary ...` \u003c-- All of `into table` but must also group records\n\nSee the syntax section for a better guide but works mostly as you would expect from a SQL like query such as:\n* `sort h.numberrange`\n* `filter h.numberrange eq .1`\n* `into table h.numberrange f.year[h.date] f.month[h.date]`\n* `into summary h.numberrange calculate f.count`\n\nYou can repeat elements for interesting effects:\n```\nfilter h.email icontains .gmail into summary h.numberrange calculate f.count filter c.count eq .1\n```\nWhich is similar to the sql `HAVING` after a `GROUP BY`\n\n### Functions\n\nSee [functions](functions.md) for a complete list of functions. They include both scalar and aggregate\nfunctions. A sample: \n\n| Function Def       | Description                                                              |\n|--------------------|--------------------------------------------------------------------------|\n| `f.as[Any,String]` | Renames the column to a specific name                                    |\n| `f.count[]`        | Returns a count of lines represented by this                             |\n| `f.count[Any]`     | Returns the number of truthy elements returned                           |\n| `f.month[String]`  | Converts time string to a date and returns the month number of that date |\n| `f.month[Integer]` | Converts Unix time to a date and returns the month number of that date   |\n| `f.sum[]`          | Returns a sum of lines represented by this                               |\n| `f.sum[Any]`       | Returns the number of truthy elements returned                           |\n| `f.year[String]`   | Converts time string to a date and returns the year number of that date  |\n| `f.year[Integer]`  | Converts Unix time to a date and returns the year number of that date    |\n\nWhen specifying multiple arguments currently white space isn't supported, so the `as` function\nlooks like this when used:\n```bash\n% go run ./cmd/csvtrace -input ast/testdata/data10.csv -input-type csv -output plot.png -output-type plot.bar -parser=basic into table h.name h.numberrange f.as\\[h.numberrange,.nr2\\]\n```\n\n### Syntax\n\nThe query language supports any combination of the following \"syntax\".\n\nRoots:\n* `sort` ` ` expressions...\n* `filter` ` ` boolean-expressions...\n* `into` ` ` `table` ` ` expressions...\n* `into` ` ` `summary` ` ` expressions... ` ` `calculate` expressions...\n\nBoolean-expressions:\n* boolean-expression\n* boolean-expression ` ` boolean-expressions\n\nBoolean-expression:\n* `not` ` ` Boolean-expression\n* expression `eq` expression\n* expression `contains` expression\n* expression `icontains` expression\n\nExpression:\n* `f.` functionname \n* `func.` functionname \n* `f.` functionname `[` function-arguments... `]`\n* `func.` functionname `[` function-arguments... `]`\n* `h.` headername\n* `header.` headername\n* `p.` propertyname\n* `property.` propertyname\n* `c.` columnname\n* `column.` columnname\n* string\n\nheadername:\n* TEXT\n\nfunctionname:\n* TEXT\n\npropertyname:\n* TEXT\n\nstring:\n* `.` TEXT\n\n## Notes:\n* Certain \"expressions\" are only available for certain data types:\n  * `column` CSV, table, group, data\n  * `header` Mail\n  * `property` ICal\n\nMore is definitely required, in most cases it shouldn't be too hard to add. That or create an issue, or if it's too hard\ntell me why in an issue / discussion.\n\n## Examples:\n\n```\nGolandProjects/pimtrace 1698 % go run ./cmd/csvtrace -input ast/testdata/data10.csv -input-type=csv -output-type table -parser basic \n+-------------------+----------------+----------------------------------+------------------------------+----------+-------------+\n|       NAME        |     PHONE      |              EMAIL               |           ADDRESS            | CURRENCY | NUMBERRANGE |\n+-------------------+----------------+----------------------------------+------------------------------+----------+-------------+\n| Jasper Joseph     | (125) 832-4826 | mauris.vestibulum@protonmail.edu | Ap #783-8034 Nunc Street     | $73.44   |           4 |\n| Rogan Hopkins     | 1-764-710-2172 | dolor.fusce.mi@protonmail.couk   | Ap #758-5121 Suspendisse Ave | $70.93   |           9 |\n| Shay Cleveland    | (637) 964-7108 | lorem@icloud.net                 | 456-1057 Libero. St.         | $7.62    |           9 |\n| Maite Weaver      | (739) 463-1949 | adipiscing.mauris@hotmail.net    | Ap #255-7815 Erat Avenue     | $32.88   |           7 |\n| Adria Herring     | (337) 415-5957 | integer.eu@yahoo.com             | Ap #669-2053 Amet St.        | $13.79   |           6 |\n| Laurel Gonzalez   | (895) 846-2962 | porttitor.vulputate@google.org   | Ap #767-7502 In, Road        | $98.11   |           9 |\n| Jane Bender       | 1-659-923-1774 | duis.gravida@protonmail.couk     | 618-8403 Aliquam Av.         | $40.56   |           9 |\n| Melinda Barton    | 1-576-782-5035 | donec@aol.ca                     | 878-2869 Purus. Av.          | $63.92   |           9 |\n| Colorado Sandoval | (737) 534-1143 | vel.vulputate@aol.net            | 8766 Nonummy St.             | $82.73   |           9 |\n| Felix Sutton      | (915) 275-7510 | justo.faucibus@hotmail.com       | 978-3821 Odio. Rd.           | $30.35   |           1 |\n+-------------------+----------------+----------------------------------+------------------------------+----------+-------------+\nGolandProjects/pimtrace 1699 % go run ./cmd/csvtrace -input ast/testdata/data10.csv -input-type=csv -output-type table -parser basic filter h.numberrange eq .1\n+--------------+----------------+----------------------------+--------------------+----------+-------------+\n|     NAME     |     PHONE      |           EMAIL            |      ADDRESS       | CURRENCY | NUMBERRANGE |\n+--------------+----------------+----------------------------+--------------------+----------+-------------+\n| Felix Sutton | (915) 275-7510 | justo.faucibus@hotmail.com | 978-3821 Odio. Rd. | $30.35   |           1 |\n+--------------+----------------+----------------------------+--------------------+----------+-------------+\nGolandProjects/pimtrace 1700 % go run ./cmd/csvtrace -input ast/testdata/data10.csv -input-type=csv -output-type table -parser basic into summary h.numberrange calculate f.count\n+-------------+-------+\n| NUMBERRANGE | COUNT |\n+-------------+-------+\n|           4 |     1 |\n|           9 |     6 |\n|           7 |     1 |\n|           6 |     1 |\n|           1 |     1 |\n+-------------+-------+\n\n```\n\n# Output types\n\n## Tabular: CSV and Table\n\nMost input formats can be converted to CSV and Table with different conditions\n\n## Plot\n\nAny tabular data can be converted into a bar provided the format is correct.\n\n### Bar\n\nThe data:\n```csv\ncountry,number1,number2,number3\nPH,3,6,4\nSE,7,1,6\n```\n\nCan be converted into a plot like:\n\n![](barplot.png)\n\nGiven the following input:\n```bash\n% go run ./cmd/csvtrace -input ast/testdata/countrynums.csv -input-type csv -output barplot.png -output-type plot.bar -parser=basic\n```\n\nWith this output type you ***must*** specify an output filename and the first column must be \na string, and all subsequent columns must contain a number on each row.\n\n# FAQ\n\n## Why\n\nI needed this, I wanted to make one tool oneliners easier and more \"query\" like. \nHowever, not so much you have to pass it as a string or use a lot of \"escape\" codes.\n\n## `*Trace`?\n\nIt's an illusion towards `dtrace` although it's not quite the same thing, it was\nchosen for uniqueness.\n\n## Will you be developing it more\n\nI would like to. Maybe?\n\n## Future data formats\n\nI would like to include other formats such as:\n* JSONLines\n* JSON / YAML (using jsonpath to denote different entities)\n* TSV\n* \"Mail dir\" format - Directory walking\n* Archived and compressed files\n* PST files\n\netc.\n\n## Basic parser\n\nI know it's a bit more than a basic parser however I was hoping to make the syntax\nmore complex with an LR parser however I didn't get around to it. I was hoping\nto expand on the concept of \"args\" as tokens, and do more sub-arg tokenization.\n\nI am requiring the use of `-parser basic` so that future scripts don't get broken.\nAs a \"least I could do\" approach.\n\n# License \n\nAGPL and \"custom\" to request. All contributions will have to be compatible with that.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farran4%2Fpimtrace","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farran4%2Fpimtrace","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farran4%2Fpimtrace/lists"}