{"id":21660306,"url":"https://github.com/slub/jsonl-fstats","last_synced_at":"2025-04-11T22:41:47.480Z","repository":{"id":41281071,"uuid":"104871463","full_name":"slub/jsonl-fstats","owner":"slub","description":"get field statistic of a line delimited JSON document","archived":false,"fork":false,"pushed_at":"2018-03-21T11:01:27.000Z","size":55,"stargazers_count":3,"open_issues_count":0,"forks_count":2,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-25T18:45:07.454Z","etag":null,"topics":["line-delimited-json","marc","python"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/slub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-26T10:33:54.000Z","updated_at":"2023-12-16T01:47:23.000Z","dependencies_parsed_at":"2022-09-21T00:41:16.246Z","dependency_job_id":null,"html_url":"https://github.com/slub/jsonl-fstats","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fjsonl-fstats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fjsonl-fstats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fjsonl-fstats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/slub%2Fjsonl-fstats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/slub","download_url":"https://codeload.github.com/slub/jsonl-fstats/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248493022,"owners_count":21113159,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["line-delimited-json","marc","python"],"created_at":"2024-11-25T09:32:47.765Z","updated_at":"2025-04-11T22:41:47.455Z","avatar_url":"https://github.com/slub.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# jsonl-fstats - line delimited json fields statistics\n\njsonl-fstats is a Python3 program  that extracts some statistics regarding field coverage in a line delimited JSON document.\n\nIt reads from stdin and prints to stdout.\n\n## Usage\n\n```\njsonl-fstats.py \n        -h, --help                      show this help message and exit\n        -marc                           Ignore Marc Indicator\n        -help                           print more help\n        -headless                       don't print header\n        -len_val LEN_VAL                specify the length for the values of \"max-value\" and \"min-value\"\n        -no_whitespace NO_WHITESPACE    don't count values only including whitespaces\n        -delimiter DELIMITER            delimiter to use\n        -csv-output                     prints the output as pure CSV data (all values are quoted)\n```\n\n* example with input and output file:\n    ```\n    jsonl-fstats.py \u003c [INPUT LINE-DELIMITED JSON DOCUMENT] \u003e [OUTPUT STATISTICS DOCUMENT]\n    ```\n\n## Requirements\n\n[numpy](http://www.numpy.org/)\n\ne.g. \n```\napt-get install python3-numpy\n```\n\n## Run\n\n* install numpy\n* clone this git repo or just download the [jsonl_fstats.py](jsonl_fstats/jsonl_fstats.py) file\n* run ./jsonl_fstats.py\n* for a hackish way to use jsonl-fstats system-wide, copy to /usr/local/bin\n\n\n\n### Install system-wide via pip\n\n* via pip:\n    ```\n    sudo -H pip3 install --upgrade [ABSOLUTE PATH TO YOUR LOCAL GIT REPOSITORY OF JSONL-FSTATS]\n    ```\n    (which provides you ```jsonl-fstats``` as a system-wide commandline command)\n    \n\n## Description\n\n(of the column headers of a resulting statistic)\n\n### ... in English\n\n#### Total Records:\n* total record count of the input\n\n#### existing\n* number of records that contain this field (path), i.e., field coverage\n\n#### occurrence\n* total count of the occurrence of this field (path) over all records, i.e., an indicator for field where multiple values are allowed\n\n#### %\n* ^ percentage of 'existing'\n* (existing / Total Records * 100)\n\n#### !%\n* ^ percentage of 'notexisting'\n* (not existing / Total Records * 100)\n\n#### notexisting\n* number of records that do not contain this field (path)\n\n#### unique\n* number of unique/distinct values of this field (path), i.e., cardinality\n\n#### avg\n* average occurrence of a value/values of this field (path) over all records\n\n#### std\n* standard variation of a value/values of this field (path) over all records\n\n#### max\n* maximum count of a (unique/distinct) value of this field (path) over all records\n\n#### min\n* minimum count of a (unique/distinct) value of this field (path) over all records\n\n#### max-value\n* the concrete value of 'max', i.e., the most frequently used value\n* the output is trimmed per default to max 17 character; you can change this string length with the option -var_len\n\n#### min-value\n* the concrete value of 'min', i.e., the least frequently used value\n* the output is trimmed per default to max 17 character; you can change this string length with the option -var_len\n\n#### max-len\n* the number of characters of the longest string of this field (path)\n\n#### min-len\n* the number of characters of the shortest string of this field (path)\n\n#### field name\n* the field (path) of this statistic line\n\n### ... in German\n\nErklärung der Spaltenköpfe\n\n#### Total Records:\n* gibt an, über wieviele Rekords (=Zeilen) die Datei verfügt\n\n#### existing\n* gibt an, wieviele Felder diesen Pfades existieren.\n\n#### occurrence\n* gibt an, wieviele Werte diesen Pfades vorhanden sind. (Mehrfachbelegung)\n\n#### %\n* existing in Prozent\n* existing / Total Records * 100\n\n#### !%\n* notexisting in Prozent\n* notexisting / Total Records * 100)\n\n#### notexisting\n* gibt an, wieviele Rekords nicht über diesen Pfad verfügen\n\n#### unique\n* gibt an, wieviele einzigartige Werte man in diesem Pfad findet\n\n#### avg\n* gibt an, wie oft man im Schnitt einen Wert in diesem Pfad findet\n\n#### std\n* Standardabweichung der Häufigkeit der Werte in diesem Pfad\n\n#### max\n* Maximale Anzahl eines Wertes in diesem Pfad\n\n#### min\n* Minimale Anzahl eines Wertes in diesem Pfad\n\n#### max-value\n* Der Wert, welcher am häufigsten in diesem Pfad vorkommt\n* Ausgabe ist auf 17 Stellen gekürzt, kann mit dem Schalter -var_len angepasst werden\n\n#### min-value\n* Der Wert, welcher am wenigsten in diesem Pfad vorkommt\n* Ausgabe ist auf 17 Stellen gekürzt, kann mit dem Schalter -var_len angepasst werden\n\n#### max-len\n* Die Zeichenanzahl des längsten Wertes in diesem Pfad\n\n#### min-len\n* Die Zeichenanzahl des kürzesten Wertes in diesem Pfad\n\n#### field name\n* Der Pfad zu den analysierten Werten\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslub%2Fjsonl-fstats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fslub%2Fjsonl-fstats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fslub%2Fjsonl-fstats/lists"}