{"id":21156209,"url":"https://github.com/aphorica/json-processor","last_synced_at":"2025-03-14T15:20:52.967Z","repository":{"id":57098142,"uuid":"181334068","full_name":"Aphorica/json-processor","owner":"Aphorica","description":"Build a JSON object from several JSON files, remove comments from JSON files","archived":false,"fork":false,"pushed_at":"2019-04-16T19:01:12.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-21T13:39:34.222Z","etag":null,"topics":["json","pre-processor"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aphorica.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-14T16:02:55.000Z","updated_at":"2019-04-16T19:01:14.000Z","dependencies_parsed_at":"2022-08-20T16:50:59.508Z","dependency_job_id":null,"html_url":"https://github.com/Aphorica/json-processor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aphorica%2Fjson-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aphorica%2Fjson-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aphorica%2Fjson-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aphorica%2Fjson-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aphorica","download_url":"https://codeload.github.com/Aphorica/json-processor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243597798,"owners_count":20316845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","pre-processor"],"created_at":"2024-11-20T11:39:19.318Z","updated_at":"2025-03-14T15:20:52.946Z","avatar_url":"https://github.com/Aphorica.png","language":"JavaScript","readme":"# @aphorica/json-processor\n\ngithub: https://aphorica.github.io\u003cbr/\u003e\nWeb: https://aphorica.com\n\nEnvironment: Node.js\n\n## Motivation\n\nA recent project I worked on implemented a sophisticated process\nthat was configuration driven, the configuration being specified\nin JSON files \u0026ndash; a good idea, but the configuration\nfile was getting unwieldy.  I needed a way to\nbreak the configuration into more manageable pieces.\n\nAnother driving factor was that this application could have several\nkinds of inputs and outputs, each having their own partial configuration\nsets.  These needed to be combined according options chosen in the UI.\nBy breaking the configuration into multiple files, I could also address\nthis feature.\n\nThere are other solutions for providing segmented configuration files,\nbut they implement YAL (yet another language), and provide kitchen\nsink functionality.\n\nWhat I was looking for was simple:\n\n - Stay within the JSON format \u0026mdash; this should be doable\n   with canonical JSON syntax, without having to extend the language.\n\n - Providing a commenting form might be a nice add-on, since I have\n   to process the entire file, anyway.\n\n - I don't need or want anything more than this.\n\n - I want it nice and tiny.\n\nI couldn't really find anything else on _npm_ or otherwise, so\nI wrote this.\n\nEnjoy, if you find it useful.\n\n## Installing\n\n`npm install @aphorica/json-processor`\n\nor\n\n`yarn add @aphorica/json-processor`\n\n## Overview\n\nThe utility reads JSON files and looks for directives in the keys it encounters\nduring traversal.  Possible keys are:\n\n\u003cdl\u003e\n\u003cdt\u003e\"file\"\u003c/dt\u003e\n\u003cdd\u003eThe value specifies a file that contains further JSON (or perhaps other input type \u0026ndash; see below)\nof information.  The file is read and traversal continues into the file (allowing\nnesting.)\u003c/dd\u003e\n\u003cdt\u003e\"!!\" (anything following the bangs is ok)\u003c/dt\u003e\n\u003cdd\u003eA comment.  Comments are simply removed. The comment-prefix can be changed\nin the passed-in options (see below.)\u003c/dd\u003e\n\u003c/dt\u003e\n\nThis allows two things:\n\n1. Assembly of a large configuration from many smaller, more easily managed and\ncontainable configuration files.\n\n2. Meaningful commenting in a JSON file (Crockford indicates commenting was explicitly left out of the JSON specification, but I think they're\n    useful.)  Without _json-processor_, you can certainly add a comment\n    prefix key in your objects, if you want, but in your program you'll\n    have to handle them, yourself.\n    \n    _json-processor_ provides a canonical form and implementation, so\n    you don't have to think about it.\n\n## Content\n\nHere is an example JSON file with both a comment and file directives:\n\n```\n{\n  \"!!\": \"The summary sheet retabulates the 'Clockify' input into its final form.  Note the individual tab sheets are created in another process (driven by 'by_contractor.json')\",\n\n  \"links-actor\":\"target\",\n  \"sheetStyle\": {\n    \"font\": {\n      \"name\":\"Calibri\",\n      \"size\":\"10\"\n    }\n  },\n\t\"groups\": [\n    {\"file\": \"{output}level_0.json\"},\n    {\"file\": \"{output}level_10.json\"},\n    {\"file\": \"{output}level_18.json\"},\n    {\"file\": \"{input}level_20.json\"}\n\t],\n  \"lookups\": {\n    \"contractors\": {\n      \"file\": \"[contractors]\"\n    }\n  }\n}\n```\nNotes on the file content:\n\n- The object specifying a file will be replaced in its entirety by\n  the content in the file.\n\n- An array of items can be specified.  If an array item is an\n  object specifying a file the entire item will be replaced with\n  the content in the specified file.\n\n- the {...} and [...] delimit substitution keys.  See the _Substitution_ topic,\n  below.\n\nIn this example, the comment is long, but if you view it in an editor\nwith soft wraps set (_vscode_, in my case), it looks fine.\n\nAlternatively, you can break up comments into sections, if you like,\nusing normal JSON syntax:\n\n```\n{\n  \"!!\": [\n     \"Here is point 1\",\n     \"Here is point 2\",\n     {\"furthermore\": \"ad nauseum}\n    ]\n  ,\n  \"actual-data\": {\n      ...\n  }\n}\n```\n\nAny children under the comment key will be removed (you don't\nhave to provide the comment key in child objects.)\n\nIf a comment is an array item, the item will be removed from\nthe array and the array rebuilt, sans comment item (order will\nbe maintained)\n\n## Substitution\n\nSubstitutions happen in the value field only for objects\ncontaining _\"file\"_ keys. Substitution paths are listed in the\n_\"options/paths\"_ argument on invocation (see next).\n\nThe substitution delimiters are _\"{\u0026nbsp;}\"_.\n\nWhen a delimited substitution value is encountered in a value, it will be\nreplaced with the corresponding path, and the base path prepended.\n\nif the completed path is absolute (per `path.absolute()` test), the\nbase path will _not_ be prepended.\n\n## Invoking\n```\n  processed = processJSON([base-path], file, [options])\n```\nArgs:\n\u003cdl\u003e\n\u003cdt\u003ebase-path (optional - needed if 'paths' are provided [see below])\n\u003cdd\u003e\nThe root path of the collection of json files.\u003c/dd\u003e\n\u003cdt\u003efile\u003c/dt\u003e\n\u003cdd\u003e\nThe top-level json filename\u003c/dd\u003e\n\u003cdt\u003eoptions (optional)\u003c/dt\u003e\n\u003cdd\u003e\nIf provided, options are:\n\u003cdl\u003e\n\u003cdt\u003epaths:\u003c/dt\u003e\n\u003cdd\u003eAn object of replacement keys and paths.  When encountered in a value (delimited\n    by '{'-'}', it will be replaced with the path.\u003c/dd\u003e\n\n\u003cdt\u003ecomment-prefix\u003c/dt\u003e\n\u003cdd\u003eSpecify another comment prefix vs the default.  The processor will only\n    look at the prefix to qualify a comment \u0026ndash; any other characters are ignored.\n\u003c/dl\u003e\n\u003c/dd\u003e\n\u003c/dl\u003e\n\n### Example Invocation\nHere is an example invocation from an app I'm working on:\n\n```\n  return processJSON(Settings.outputsConfigPath, \n                        'config.json', {  // 'options'\n                        paths: {\n                          output: Settings.outputsConfigPath,\n                          input: Settings.inputsConfigPath,\n                          contractors: Settings.contractorsFile\n                        }})\n```\n\nThis uses paths contained in a `Settings` object.  For another\nexample, see the test file.\n\n## Including Other File Types\n\nYou can include other file types by providing a list of objects,\neach specifying a file extension and a processor to return a js\nobject.  For instance, a text file with arbirtrarily formatted\ninformation could be processed by including it in a JSON file\nlike so:\n```\n  {\n    ...\n    \"other\": {\n      \"file\": \"somefile.txt\"\n    }\n    ...\n  }\n```\n\nAnd providing the extension and processor in the options:\n\n```\n  function parseText(text) {\n    let retObj = {}\n    // build up the object by parsing the text\n    return retObj \n  }\n\n  processJSON(path, fn, {\n    types: {\n      \".txt\": procText\n    }\n  })\n```\n\nYou can process YAML by providing a YAML parser.\n```\n  import yaml from 'js-yaml'\n\n  function procYaml(text) {\n    yaml.loadSafe(text) // provide additional args here, if needed\n  }\n\n  processJSON(path, fn, {\n    types: {\n      \".yml\": procYaml\n    }\n  })\n```\n\nSee the test for working examples\n\n## Notes\n - All paths are expected to be trailed with a path separator.\n\n - On the name: I initially wanted to call it 'json-pp' for 'json _pre-processor_', however _pre-processor_ isn't exactly right,\n    since the files are read and parsed before any processing.\n    \n    So, I'm just calling it _'json-processor'_\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faphorica%2Fjson-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faphorica%2Fjson-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faphorica%2Fjson-processor/lists"}