{"id":15657437,"url":"https://github.com/danielgtaylor/shorthand","last_synced_at":"2025-04-19T13:50:44.399Z","repository":{"id":43217571,"uuid":"418332377","full_name":"danielgtaylor/shorthand","owner":"danielgtaylor","description":"Structured data \u0026 CLI shorthand syntax for Go","archived":false,"fork":false,"pushed_at":"2024-01-30T17:28:47.000Z","size":345,"stargazers_count":33,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T08:23:50.339Z","etag":null,"topics":["cbor","cli","golang-library","hacktoberfest","json","shorthand","structured-data"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danielgtaylor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-10-18T03:37:10.000Z","updated_at":"2025-03-03T00:32:07.000Z","dependencies_parsed_at":"2024-02-06T04:00:56.396Z","dependency_job_id":null,"html_url":"https://github.com/danielgtaylor/shorthand","commit_stats":{"total_commits":106,"total_committers":4,"mean_commits":26.5,"dds":"0.028301886792452824","last_synced_commit":"94d3e969a467ba67d36a8cf708f47f35608a0c4f"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fshorthand","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fshorthand/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fshorthand/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielgtaylor%2Fshorthand/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danielgtaylor","download_url":"https://codeload.github.com/danielgtaylor/shorthand/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249707601,"owners_count":21313882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cbor","cli","golang-library","hacktoberfest","json","shorthand","structured-data"],"created_at":"2024-10-03T13:07:02.973Z","updated_at":"2025-04-19T13:50:44.378Z","avatar_url":"https://github.com/danielgtaylor.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Structured Data Shorthand Syntax\n\n[![Docs](https://godoc.org/github.com/danielgtaylor/shorthand?status.svg)](https://pkg.go.dev/github.com/danielgtaylor/shorthand?tab=doc) [![Go Report Card](https://goreportcard.com/badge/github.com/danielgtaylor/shorthand)](https://goreportcard.com/report/github.com/danielgtaylor/shorthand) [![CI](https://github.com/danielgtaylor/shorthand/workflows/CI/badge.svg?branch=main)](https://github.com/danielgtaylor/shorthand/actions?query=workflow%3ACI+branch%3Amain++) [![codecov](https://codecov.io/gh/danielgtaylor/shorthand/branch/main/graph/badge.svg)](https://codecov.io/gh/danielgtaylor/shorthand)\n\nShorthand is a superset and friendlier variant of JSON designed with several use-cases in mind:\n\n| Use Case             | Example                                          |\n| -------------------- | ------------------------------------------------ |\n| CLI arguments/input  | `my-cli post 'foo.bar[0]{baz: 1, hello: world}'` |\n| Patch operations     | `name: undefined, item.tags[]: appended`         |\n| Query language       | `items[created before 2022-01-01].{id, tags}`    |\n| Configuration format | `{json.save.autoFormat: true}`                   |\n\nThe shorthand syntax supports the following features, described in more detail with examples below:\n\n- Superset of JSON (valid JSON is valid shorthand)\n  - Optional commas, quotes, and sometimes colons\n  - Support for comments \u0026 trailing commas\n- Automatic type coercion\n  - Support for bytes, datetimes, and maps with non-string keys\n- Nested object \u0026 array creation\n- Loading values from files\n- Editing existing data\n  - Appending \u0026 inserting to arrays\n  - Unsetting properties\n  - Moving properties \u0026 items\n- Querying, array filtering, and field selection\n\nThe following are all completely valid shorthand and result in the same output:\n\n```\nfoo.bar[]{baz: 1, hello: world}\n```\n\n```\n{\n  // This is a comment\n  foo.bar[]{\n    baz: 1\n    hello: world\n  }\n}\n```\n\n```json\n{\n  \"foo\": {\n    \"bar\": [\n      {\n        \"baz\": 1,\n        \"hello\": \"world\"\n      }\n    ]\n  }\n}\n```\n\nThis library has excellent test coverage to ensure correctness and is additionally fuzz tested to prevent panics.\n\n## Alternatives \u0026 Inspiration\n\nThe CLI shorthand syntax is not the only one you can use to generate data for CLI commands. Here are some alternatives:\n\n- [jo](https://github.com/jpmens/jo)\n- [jarg](https://github.com/jdp/jarg)\n\nFor example, the shorthand example given above could be rewritten as:\n\n```sh\n$ jo -p foo=$(jo -p bar=$(jo -a $(jo -p baz=1 hello=world)))\n```\n\nThe shorthand syntax implementation described herein uses those and the following for inspiration:\n\n- [YAML](http://yaml.org/)\n- [W3C HTML JSON Forms](https://www.w3.org/TR/html-json-forms/)\n- [jq](https://stedolan.github.io/jq/)\n- [JMESPath](http://jmespath.org/)\n\nIt seems reasonable to ask, why create a new syntax?\n\n1. Built-in. No extra executables required. Your tool ships ready-to-go.\n2. No need to use sub-shells to build complex structured data.\n3. Syntax is closer to YAML \u0026 JSON and mimics how you do queries using tools like `jq` and `jmespath`.\n4. It's _optional_, so you can use your favorite tool/language instead, while at the same time it provides a minimum feature set everyone will have in common.\n\n## Features in Depth\n\nYou can use the included `j` executable to try out the shorthand format examples below. Examples are shown in JSON, but the shorthand parses into structured data that can be marshalled as other formats, like YAML or TOML if you prefer.\n\n```sh\ngo get -u github.com/danielgtaylor/shorthand/cmd/j\n```\n\nAlso feel free to use this tool to generate structured data for input to other commands.\n\nHere is a diagram overview of the language syntax, which is similar to [JSON's syntax](https://www.json.org/json-en.html) but adds a few things:\n\n\u003c!--\nhttps://tabatkins.github.io/railroad-diagrams/generator.html\n\nDiagram(\n    Choice(0,\n      Sequence('//', NonTerminal('comment')),\n      Sequence(\n        '{',\n        OneOrMore(\n          Sequence(\n            OneOrMore(Sequence(NonTerminal('string'), Optional(Sequence('[', Optional('^'), ZeroOrMore('0-9'), ']'))), '.'),\n            Choice(0,\n              Sequence(':', NonTerminal('value')),\n              Sequence('^', NonTerminal('query')),\n              NonTerminal('object'),\n            ),\n          ),\n          Choice(0, ',', '\\\\n'),\n        ),\n        '}'\n      ),\n      Sequence(\n        '[',\n        OneOrMore(NonTerminal('value'), Choice(0, ',', '\\\\n')),\n        ']'\n      ),\n      'undefined',\n      'null',\n      'true',\n      'false',\n      NonTerminal('integer'),\n      NonTerminal('float'),\n      Stack(\n        Sequence(\n          NonTerminal('YYYY'),\n          '-',\n          NonTerminal('MM'),\n          '-',\n          NonTerminal('DD'),\n        ),\n          Sequence(\n          'T',\n          NonTerminal('hh'),\n          ':',\n          NonTerminal('mm'),\n          ':',\n          NonTerminal('ss'),\n          NonTerminal('zone')\n        ),\n      ),\n      Sequence('%', NonTerminal('base64')),\n      Sequence('@', NonTerminal('filename')),\n      NonTerminal('string'),\n    ),\n)\n--\u003e\n\n![shorthand-syntax](https://user-images.githubusercontent.com/106826/198850895-a1a8481a-2c63-484c-9bf2-ce472effa8c3.svg)\n\nNote:\n\n- `string` can be quoted (with `\"`) or unquoted.\n- The `query` syntax in the diagram above is described below in the [Querying](#querying) section.\n\n### Keys \u0026 Values\n\nAt its most basic, a structure is built out of key \u0026 value pairs. They are separated by commas:\n\n```sh\n$ j hello: world, question: how are you?\n{\n  \"hello\": \"world\",\n  \"question\": \"how are you?\"\n}\n```\n\n### Types\n\nShorthand supports the standard JSON types, but adds some of its own as well to better support binary formats and its query features.\n\n| Type      | Description                                                      |\n| --------- | ---------------------------------------------------------------- |\n| `null`    | JSON `null`                                                      |\n| `boolean` | Either `true` or `false`                                         |\n| `number`  | JSON number, e.g. `1`, `2.5`, or `1.4e5`                         |\n| `string`  | Quoted or unquoted strings, e.g. `hello` or `\"hello\"`            |\n| `bytes`   | `%`-prefixed, unquoted, base64-encoded binary data, e.g. `%wg==` |\n| `time`    | Date/time in ISO8601, e.g. `2022-01-01T12:00:00Z`                |\n| `array`   | JSON array, e.g. `[1, 2, 3]`                                     |\n| `object`  | JSON object, e.g. `{\"hello\": \"world\"}`                           |\n\n### Type Coercion\n\nWell-known values like `null`, `true`, and `false` get converted to their respective types automatically. Numbers, bytes, and times also get converted. Similar to YAML, anything that doesn't fit one of those is treated as a string. This automatic coercion can be disabled by just wrapping your value in quotes.\n\n```sh\n# With coercion\n$ j empty: null, bool: true, num: 1.5, string: hello\n{\n  \"bool\": true,\n  \"empty\": null,\n  \"num\": 1.5,\n  \"string\": \"hello\"\n}\n\n# As strings\n$ j empty: \"null\", bool: \"true\", num: \"1.5\", string: \"hello\"\n{\n  \"bool\": \"true\",\n  \"empty\": \"null\",\n  \"num\": \"1.5\",\n  \"string\": \"hello\"\n}\n\n# Passing the empty string\n$ j blank1: , blank2: \"\"\n{\n  \"blank1\": \"\",\n  \"blank2\": \"\"\n}\n```\n\n### Objects\n\nNested objects use a `.` separator when specifying the key.\n\n```sh\n$ j foo.bar.baz: 1\n{\n  \"foo\": {\n    \"bar\": {\n      \"baz\": 1\n    }\n  }\n}\n```\n\nProperties of nested objects can be grouped by placing them inside `{` and `}`. The `:` becomes optional for nested objects, so `foo.bar: {...}` is equivalent to `foo.bar{...}`.\n\n```sh\n$ j foo.bar{id: 1, count.clicks: 5}\n{\n  \"foo\": {\n    \"bar\": {\n      \"count\": {\n        \"clicks\": 5\n      },\n      \"id\": 1\n    }\n  }\n}\n```\n\n### Arrays\n\nArrays are surrounded by square brackets like in JSON:\n\n```sh\n# Simple array\n$ j [1, 2, 3]\n[\n  1,\n  2,\n  3\n]\n```\n\nArray indexes use square brackets `[` and `]` to specify the zero-based index to set an item. If the index is out of bounds then `null` values are added as necessary to fill the array. Use an empty index `[]` to append to the an existing array. If the item is not an array, then a new one will be created.\n\n```sh\n# Nested arrays\n$ j [0][2][0]: 1\n[\n  [\n    null,\n    null,\n    [\n      1\n    ]\n  ]\n]\n\n# Appending arrays\n$ j a[]: 1, a[]: 2, a[]: 3\n{\n  \"a\": [\n    1,\n    2,\n    3\n  ]\n}\n```\n\n### Loading from Files\n\nSometimes a field makes more sense to load from a file than to be specified on the commandline. The `@` preprocessor lets you load structured data, text, and bytes depending on the file extension and whether all bytes are valid UTF-8:\n\n```sh\n# Load a file's value as a parameter\n$ j foo: @hello.txt\n{\n  \"foo\": \"hello, world\"\n}\n\n# Load structured data\n$ j foo: @hello.json\n{\n  \"foo\": {\n    \"hello\": \"world\"\n  }\n}\n```\n\nRemember, it's possible to disable this behavior with quotes:\n\n```sh\n$ j 'twitter: \"@user\"'\n{\n  \"twitter\": \"@user\"\n}\n```\n\n### Patch (Partial Update)\n\nPartial updates are supported on existing data, which can be used to implement HTTP `PATCH`, templating, and other similar features. The suggested content type for HTTP `PATCH` is `application/shorthand-patch`. This feature combines the best of both:\n\n- [JSON Merge Patch](https://datatracker.ietf.org/doc/html/rfc7386)\n- [JSON Patch](https://www.rfc-editor.org/rfc/rfc6902)\n\nPartial updates support:\n\n- Appending arrays via `[]`\n- Inserting before via `[^index]`\n- Removing fields or array items via `undefined`\n- Moving/swapping fields or array items via `^`\n  - The right hand side is a path to the value to swap. See Querying below for the path syntax.\n\nNote: When sending shorthand patches file loading via `@` should be disabled as the files will not exist on the server.\n\nSome examples:\n\n```sh\n# First, let's create some data we'll modify later\n$ j id: 1, tags: [a, b, c] \u003edata.json\n\n# Now let's append to the tags array\n$ j \u003cdata.json 'tags[]: d'\n{\n  \"id\": 1,\n  \"tags\": [\n    \"a\",\n    \"b\",\n    \"c\",\n    \"d\"\n  ]\n}\n\n# Array item insertion (prepend the array)\n$ j \u003cdata.json 'tags[^0]: z'\n{\n  \"id\": 1,\n  \"tags\": [\n    \"z\",\n    \"a\",\n    \"b\",\n    \"c\"\n  ]\n}\n\n# Remove stuff\n$ j \u003cdata.json 'id: undefined, tags[1]: undefined'\n{\n  \"tags\": [\n    \"a\",\n    \"c\"\n  ]\n}\n\n# Rename the ID property, and swap the first/last array items\n$ j \u003cdata.json 'id ^ name, tags[0] ^ tags[-1]'\n{\n  \"name\": 1,\n  \"tags\": [\n    \"c\",\n    \"b\",\n    \"a\"\n  ]\n}\n```\n\n### Querying\n\nA data query language is included, which allows you to query, filter, and select fields to return. This functionality is used by the patch move operations described above and is similar to tools like:\n\n- [jq](https://stedolan.github.io/jq/)\n- [JMESPath](http://jmespath.org/)\n- [JSON Path](https://www.ietf.org/archive/id/draft-ietf-jsonpath-base-06.html)\n\nThe query language supports:\n\n- Paths for objects \u0026 arrays `foo.items.name`\n- Wildcards for unknown props `foo.*.name`\n- Array indexing \u0026 slicing `foo.items[1:2].name`\n  - Including negative indexes `foo.items[-1].name`\n- Array filtering via [mexpr](https://github.com/danielgtaylor/mexpr) `foo.items[name.lower startsWith d]`\n- Object property selection `foo.{created, names: items.name}`\n- Recursive search `foo..name`\n- Stopping processing with a pipe `|`\n- Flattening nested arrays `[]`\n\nThe query syntax is recursive and looks like this:\n\n\u003c!--\nDiagram(\n  Stack(\n    OneOrMore(Sequence(\n      Choice(1,\n        Skip(),\n        NonTerminal('string'),\n        '*',\n      ),\n      Optional(\n        Sequence(\n          '[',\n          Choice(1,\n            Skip(),\n            NonTerminal('number'),\n            NonTerminal('slice'),\n            NonTerminal('filter')\n          ),\n          ']',\n        )\n      ),\n      Optional('|'),\n    ), Choice(0, '.', '..')),\n    Optional(\n      Sequence(\n        '.',\n        '{',\n        OneOrMore(\n          Sequence(\n            NonTerminal('string'),\n            Optional(\n              Sequence(':', NonTerminal('query')),\n              'skip'\n            ),\n          ),\n          ','\n        ),\n        '}',\n      ), 'skip',\n    ),\n  )\n)\n--\u003e\n\n![shorthand-query-syntax](https://user-images.githubusercontent.com/106826/198693468-fadf8d48-8223-4dd9-a2cb-a1651e342fc5.svg)\n\nThe `filter` syntax is described in the documentation for [mexpr](https://github.com/danielgtaylor/mexpr).\n\nExamples:\n\n```sh\n# First, let's make a complex file to query\n$ j 'users: [{id: 1, age: 5, friends: [a, b]}, {id: 2, age: 6, friends: [b, c]}, {id: 3, age: 5, friends: [c, d]}]' \u003edata.json\n\n# Query for each user's ID\n$ j \u003cdata.json -q 'users.id'\n[\n  1,\n  2,\n  3\n]\n\n# Get the users who are friends with `b`\n$ j \u003cdata.json -q 'users[friends contains b].id'\n[\n  1,\n  2\n]\n\n# Get the ID \u0026 age of users who are friends with `b`\n$ j \u003cdata.json -q 'users[friends contains b].{id, age}'\n[\n  {\n    \"age\": null,\n    \"id\": 1\n  },\n  {\n    \"age\": null,\n    \"id\": 2\n  }\n]\n```\n\n## Library Usage\n\nAside from `Marshal` and `Unmarshal` functions, the `GetInput` function provides an all-in-one quick and simple way to get input from both stdin and passed arguments for CLI applications:\n\n```go\npackage main\n\nimport (\n  \"fmt\"\n  \"github.com/danielgtaylor/shorthand/v2\"\n)\n\nfunc main() {\n  result, err := shorthand.GetInput(os.Args[1:])\n  if err != nil {\n    panic(err)\n  }\n\n  fmt.Println(result)\n}\n```\n\nIt's also possible to get the shorthand representation of an input, for example:\n\n```go\nexample := map[string]interface{}{\n  \"hello\": \"world\",\n  \"labels\": []interface{}{\n    \"one\",\n    \"two\",\n  },\n}\n\n// Prints \"hello: world, labels: [one, two]\"\nfmt.Println(shorthand.MarshalCLI(example))\n```\n\n## Benchmarks\n\nShorthand v2 has been completely rewritten from the ground up and is over 20 times faster than v1, putting it at a similar speed/efficiency as the standard library's `encoding/json` package and faster than the popular YAML package while supporting some compelling additional features:\n\n```sh\n# Comparing new (V2) vs. old (V1)\nBenchmarkShorthandV2-12     309817    2482 ns/op    1888 B/op    54 allocs/op\nBenchmarkShorthandV1-12      14670   83901 ns/op   36436 B/op   745 allocs/op\n\n# Comparing JSON \u0026 YAML to Shorthand\nBenchmarkMinJSON-10         825459    1446 ns/op    1808 B/op    31 allocs/op\nBenchmarkFormattedJSON-10   707174    1658 ns/op    1712 B/op    30 allocs/op\n\nBenchmarkYAML-10            107493   11053 ns/op   12100 B/op   140 allocs/op\n\nBenchmarkShorthand-10       477285    2389 ns/op    1888 B/op    54 allocs/op\nBenchmarkPretty-10          403887    2848 ns/op    1888 B/op    54 allocs/op\nBenchmarkParse-10          1277103     938 ns/op     160 B/op    12 allocs/op\nBenchmarkApply-10           811148    1421 ns/op    1733 B/op    42 allocs/op\n\n# Comparing Shorthand get path to JMESPath\nBenchmarkGetJMESPathSimple-10  414164   2790 ns/op   5799 B/op   74 allocs/op\nBenchmarkGetPathSimple-10     4332314    276 ns/op    224 B/op    5 allocs/op\n\nBenchmarkGetJMESPath-10        224778   5289 ns/op   9374 B/op   119 allocs/op\nBenchmarkGetPath-10            793628   1437 ns/op   1192 B/op    27 allocs/op\n\nBenchmarkGetJMESPathFlat-10   1743403  688.6 ns/op    632 B/op    14 allocs/op\nBenchmarkGetPathFlat-10       1964098  610.3 ns/op    560 B/op    12 allocs/op\n```\n\n## Design \u0026 Implementation\n\nThe shorthand syntax is implemented as a custom parser split into two pieces: `parse.go` to parse a shorthand input into a set of operations and `apply.go` to provide the mechanism for applying those operations on an existing input (or `nil`). Every operation will either set, delete, or swap some value or path. For example:\n\n```\n# Input\nfoo.bar{id: 1, tags: [{value: a}, {value: b}]}\n\n# Parsed\n[\n  [OpSet, \"foo.bar.id\", 1],\n  [OpSet, \"foo.bar.tags[0].value\", \"a\"],\n  [OpSet, \"foo.bar.tags[1].value\", \"b\"]\n]\n\n# Existing\n{\"foo\": {\"baz\": 2}}\n\n# Applied Output JSON\n{\n  \"foo\": {\n    \"bar\": {\n      \"id\": 1,\n      \"tags: [\n        {\"value\": \"a\"},\n        {\"value\": \"b\"}\n      ]\n    },\n    \"baz\": 2\n  }\n}\n```\n\nThis simplifies the code to apply changes, as it can process each operation independently.\n\nThe file `get.go` provides an implementation of query parsing. It also utilizes [danielgtaylor/mexpr](https://github.com/danielgtaylor/mexpr), a top-down operator precedence (Pratt) parser for simple filter expressions.\n\nNo special steps are necessary to test local changes to the grammar. You can just run the included `j` utility to test:\n\n```sh\n$ go run ./cmd/j your: new feature here\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielgtaylor%2Fshorthand","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanielgtaylor%2Fshorthand","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielgtaylor%2Fshorthand/lists"}