{"id":15960565,"url":"https://github.com/lucaswerkmeister/ja2l","last_synced_at":"2026-05-16T11:02:54.230Z","repository":{"id":148665178,"uuid":"98239109","full_name":"lucaswerkmeister/ja2l","owner":"lucaswerkmeister","description":"JSON array to lines","archived":false,"fork":false,"pushed_at":"2023-05-31T16:55:51.000Z","size":39,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-27T02:40:11.087Z","etag":null,"topics":["dgsh","json"],"latest_commit_sha":null,"homepage":null,"language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lucaswerkmeister.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-24T22:09:04.000Z","updated_at":"2023-05-31T16:54:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"b5f84abb-6e59-4590-9746-b24814cb1790","html_url":"https://github.com/lucaswerkmeister/ja2l","commit_stats":{"total_commits":41,"total_committers":1,"mean_commits":41.0,"dds":0.0,"last_synced_commit":"de1c7abef5fc7479b02b2b4f8a04e9f03c43a3c6"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lucaswerkmeister/ja2l","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaswerkmeister%2Fja2l","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaswerkmeister%2Fja2l/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaswerkmeister%2Fja2l/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaswerkmeister%2Fja2l/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lucaswerkmeister","download_url":"https://codeload.github.com/lucaswerkmeister/ja2l/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lucaswerkmeister%2Fja2l/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33100319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dgsh","json"],"created_at":"2024-10-07T15:20:31.577Z","updated_at":"2026-05-16T11:02:54.197Z","avatar_url":"https://github.com/lucaswerkmeister.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ja2l\n\n[![Build Status](https://travis-ci.org/lucaswerkmeister/ja2l.svg?branch=master)](https://travis-ci.org/lucaswerkmeister/ja2l)\n\nJSON array to lines.\nExpects a JSON array like this on standard input:\n\n```json\n[\n{\"foo\":\"bar\"},\n{\"abc\":\"xyz\"},\n{\"whatever\":0}\n]\n```\n\nand converts it into a stream of JSON values on standard output:\n\n```json\n{\"foo\":\"bar\"}\n{\"abc\":\"xyz\"}\n{\"whatever\":0}\n```\n\nThis stream is suitable for processing with [jq],\nand can also be split up for parallelized processing.\n(This format is also variously known as\nline delimited JSON (ldjson),\n[newline delimited JSON (ndjson)][ndjson],\nor [JSON lines (jsonl)][jsonl].)\n\n## Project status\n\nFunctional, but more tests and various other goodies would be nice.\n\n## Build instructions\n\n`make \u0026\u0026 sudo make install`. Needs glibc and GCC.\n\nThe Makefile follows standard GNU conventions;\nfor instance, packagers can use something like\n`make DESTDIR=\"$pkgdir\" prefix=/usr install`.\n(Note that standard `$(bindir)`, `$(man1dir)`, etc. directories\nare expected to exist,\nbecause the GNU Make manual says to not use the nonstandard `mkdir -p`\nand I don’t know how to create the directories without it.)\n\n## Requirements\n\n* glibc (`error`, `getopt_long`)\n* Linux (`/proc/sys/fs/pipe-max-size`, `fcntl(F_SETPIPE_SZ)`)\n* GCC (`-fanalyzer`; if you remove that from `CFLAGS`, clang may or may not work)\n* optional: [dgsh]\n\nThe reason lists in parentheses are probably not exhaustive.\n(The most likely reason for that is that\nI’ll probably forget to update them as I update the program.)\n\n## dgsh support\n\nja2l can be built with [dgsh] support.\nWhen used in a dgsh pipeline,\nit accepts zero or one inputs\n(depending on whether or not a file name was specified on the command line)\nand scatters the JSON values across any (nonzero) number of outputs.\n\nTo build ja2l with dgsh support,\nadd `-DUSE_DGSH` to the `CPPFLAGS`\nand `-ldgsh` to the `LDLIBS`,\ne. g. like this:\n\n```sh\nmake CPPFLAGS=-DUSE_DGSH LDLIBS=-ldgsh clean all\n```\n\nThis can be used to speed up processing of the JSON data with [jq],\nsimilar to this script:\n\n```sh\nfunction countElements {\n    jq -r '\n      .elements |\n      .[]\n    ' | awk '\n      {\n        a[$0]++\n      }\n      END {\n        for (k in a)\n          print a[k] \"\\t\" k\n      }\n    '\n}\n\nfunction summarizeElements {\n    awk -F'\\t' '\n      {\n        a[$2] += $1\n      }\n      END {\n          for (k in a)\n            if (a[k] \u003e= 1000)\n              print a[k] \"\\t\" k\n      }\n    ' |\n    sort -nr\n}\n\nja2l | {{\n        countElements \u0026\n        countElements \u0026\n        countElements \u0026\n        countElements \u0026\n    }} |\n    cat |\n    summarizeElements\n```\n\nThis prints the most common “elements” in the JSON input,\nparallelizing the extraction and counting of elements across four `countElements` invocations.\nThe results of those invocations are then aggregated into a single result list again.\nIf the processing is CPU-bound,\n`jq` is the expensive part,\nand you have four processors or processor cores (without counting hyper-threading),\nthis should speed up processing by about a factor of four.\n\n## Attribution\n\nThe `cleanup.h` header file is based on systemd header files,\nwhich are published under the LGPL2.1+.\n\nThe install-related variables in the `Makefile` are copied from the GNU Make manual,\nwhich is published under the FDL1.3+.\n\n## License\n\nThe content of this repository is released under the AGPL3+\nas provided in the LICENSE file that accompanied this code.\n\nBy submitting a “pull request” or otherwise contributing to\nthis repository, you agree to license your contribution under\nthe license mentioned above.\n\n[dgsh]: https://www.spinellis.gr/sw/dgsh/\n[jq]: https://stedolan.github.io/jq/\n[ndjson]: http://ndjson.org/\n[jsonl]: http://jsonlines.org/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucaswerkmeister%2Fja2l","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flucaswerkmeister%2Fja2l","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flucaswerkmeister%2Fja2l/lists"}