{"id":15659772,"url":"https://github.com/volks73/panser","last_synced_at":"2025-10-24T11:51:06.836Z","repository":{"id":19943416,"uuid":"88355836","full_name":"volks73/panser","owner":"volks73","description":"A command line utility for (de)serializing data","archived":false,"fork":false,"pushed_at":"2024-12-31T15:27:29.000Z","size":734,"stargazers_count":6,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-02T21:02:27.653Z","etag":null,"topics":["cli","deserialization","developer-tools","rust","serialization"],"latest_commit_sha":null,"homepage":"https://volks73.github.io/panser","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/volks73.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-15T15:26:10.000Z","updated_at":"2025-02-17T20:51:18.000Z","dependencies_parsed_at":"2024-10-23T09:32:38.393Z","dependency_job_id":null,"html_url":"https://github.com/volks73/panser","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volks73%2Fpanser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volks73%2Fpanser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volks73%2Fpanser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/volks73%2Fpanser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/volks73","download_url":"https://codeload.github.com/volks73/panser/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252566626,"owners_count":21769077,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","deserialization","developer-tools","rust","serialization"],"created_at":"2024-10-03T13:18:59.196Z","updated_at":"2025-10-24T11:51:01.816Z","avatar_url":"https://github.com/volks73.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Panser: A command line application for (de)serializing data\n\nThe Panser project is a Command-Line Interface (CLI) application for\n(de)serializing data formats in a pipe-friendly manner. The goal is to have a\nsingle application for reading data in one format on stdin and writing the same\ndata but in a different format to stdout.\n\n[![GitHub release](https://img.shields.io/github/release/volks73/panser.svg)](https://github.com/volks73/panser/releases)\n[![license](https://img.shields.io/github/license/volks73/panser.svg)](https://github.com/volks73/panser/blob/master/LICENSE)\n\n[Installation](#installation) | [Usage](#usage) | [Manual](https://volks73.github.io/panser/manpage.html) | [API](https://volks73.github.io/panser) | [Build](#build)\n\nThe project is primarily written in the [Rust](http://www.rust-lang.org)\nprogramming language. It can be installed on any [platform\nsupported](https://forge.rust-lang.org/platform-support.html) by the Rust\nprogramming language, including Linux, macOS, and Windows. It is possible to\nread data from a file and write to another file, but the application is focused\non creating streams of data that can be piped into a socket, such as a TCP\nstream. The primary motivator for the application is to read\n[JSON](http://www.json.org/) data and write output to the\n[MessagePack](http://msgpack.org/index.html) format which could be used with a\nTCP stream to develop low-level Application Programming Interfaces (APIs) for\nnetwork-enabled applications. The reverse is also a desired goal: reading in\nMessagePack data (binary, machine-readable) and transcoding it to JSON (text,\nhuman-readable).\n\nAfter accomplishing the primary goal of transcoding between JSON and MessagePack\n(Msgpack) formats, additional formats were gradually added using the\n[serde](https://github.com/serde-rs/serde) project and related libraries. Almost\nall of the formats listed in the [Data Formats](https://serde.rs/#data-formats)\nsection of the [Overview](https://serde.rs/) for the serde project are\nimplemented. The intention is to add more formats as more crates are developed\nusing the serde framework.\n\n## Installation\n\nPanser can be installed on any platform supported by the Rust programming\nlanguage, including Linux, macOS, and Windows. It is possible to run Panser on\nWindows using the native command prompt (cmd.exe) or a terminal emulator, like\n[Mintty](https://mintty.github.io/) via [Cygwin](https://www.cygwin.com/).\n\n### Windows\n\nAn installer (msi) with a pre-compiled binary is available with each\n[release](https://github.com/volks73/panser/releases). The installer will also\nadd the installation location to the PATH system environment variable so panser\ncan be executed from anywhere. Run the installer and follow the on-screen dialog\nto complete the installation.\n\nIt is also possible to install the application from source using Cargo. See the\ninstructions for [installation via Cargo](#source) and use a command prompt\n(cmd.exe) or terminal emulator to execute the commands.\n\n### macOS\n\nFollow the instructions for [installation from source](#source).\n\n### Linux\n\nFollow the instructions for [installation from source](#source).\n\n### Source\n\nDownload and install the following dependencies before installing the binary\nusing Cargo.\n\n- [Cargo](https://crates.io/), v0.17 or higher\n- [Pandoc](http://pandoc.org), v1.19 or higher, optional\n- [Rust](https://www.rust-lang.org/), v1.41 or higher\n\n#### Application\n\nRun the following commands from a terminal:\n\n    git clone https://github.com/volks73/panser.git\n    cd panser\n    cargo install\n\nOr obtain the source as an archive and run the following commands from a terminal:\n\n    tar xf panser-#.#.#.tar.gz\n    cd panser-#.#.#\n    cargo install\n\nwhere `#.#.#` is replaced with the version number of the source distribution,\nrespectively. It might be desirable to change the install location by using the\n`--root` option with the `cargo install` command. See the `cargo install --help`\nfor more information about installing a Rust binary crate using Cargo.\n\nIt might be desirable to change the install location by using the `--root`\noption with the `cargo install` command. See the `cargo install --help` for more\ninformation about installing a Rust binary crate using Cargo.\n\nNote, if the panser binary was installed using cargo, then it can be uninstalled\nusing `cargo uninstall panser`.\n\n#### Documentation (Optional)\n\nIf the [Pandoc](http://pandoc.org) application was installed prior to installing\nfrom source via Cargo, i.e. `cargo install`, then a manpage in the\n[grofff](https://www.gnu.org/software/groff/) format is automatically created\nfrom the [markdown](http://pandoc.org/MANUAL.html#pandocs-markdown) \"source\"\nfile in the `man` directory using pandoc as part of the build script\n(`build.rs`). Otherwise, the manpage can be built with the following command:\n\n    pandoc -s -t man -o man/panser.1 man/panser.1.md\n\nRegardless if the manpage (`panser.1`) was manually or automatically generated,\nit must be must be manually installed with the following command:\n\n    mkdir -p ~/.cargo/share/man/man1\n    cp man/panser.1 ~/.cargo/share/man/man1\n\nIf uninstalling panser using Cargo, i.e. `cargo uninstall panser`, then the\nmanpage must also be manually removed as follows:\n\n    rm ~/.cargo/share/man/man1/panser.1\n\n## Usage\n\nConvert [JSON](http://www.json.org) from stdin to\n[MessagePack](http://msgpack.org) (Msgpack) and write to stdout. Panser converts\nJSON to Msgpack by default. See the `-h,--help` text for more information and\noptions. Specifically, see the `-f,--from` and `-t,--to` help text for lists of\nsupported formats. The `-r,--radix` option is used to display the binary Msgpack\nformat in a human readable format.\n\n    ```bash\n    $ echo '{\"bool\":true}' | panser --radix hex\n    81 A4 62 6F 6F 6C C3\n    ```\n\nSimilarly, convert JSON from a file to Msgpack and write to stdout. If no file\nis specified, then input data is read continuously from stdin. The file\nextension is used to determine the input data format unless the `-f,--from`\noption is explicitly used. Here, the `-r` option with the single character value\nis used to demonstrate a more succinct command line.\n\n    ```bash\n    $ panser -r h file.json\n    81 A4 62 6F 6F 6C C3\n    ```\n\nRedirection can also be used.\n\n    ```bash\n    $ panser -r h \u003c file.json\n    81 A4 62 6F 6F 6C C3\n    ```\n\nWrite data to a file instead of stdout. The output file will contain the binary\nMessagePack data. The [xxd](http://linuxcommand.org/man_pages/xxd1.html) command\nis used to display the binary data as a [hex\ndump](https://en.wikipedia.org/wiki/Hex_dump). Using the `xxd` command is\nsimilar, but not identical, to using the `-r,--radix` option.\n\n    ```bash\n    $ echo '{\"bool\":true,\"number\":1.234}' | panser -o file.msgpack\n    $ cat file.msgpack | xxd -i\n    0x82, 0xa4, 0x62, 0x6f, 0x6f, 0x6c, 0xc3, 0xa6, 0x6e, 0x75, 0x6d, 0x62,\n    0x65, 0x72, 0xcb, 0x3f, 0xf3, 0xbe, 0x76, 0xc8, 0xb4, 0x39, 0x58\n    ```\n\nIf the `-r,--radix` option is used, then the contents of the output file would\n_not_ be Msgpack data, but the space-separated list of bytes as numeric strings.\n\n    ```bash\n    $ echo '{\"bool\":true,\"number\":1.234}' | panser -r h -o file.msgpack\n    $ cat file.msgpack\n    82 A4 62 6F 6F 6C C3 A6 6E 75 6D 62 65 72 CB 3F F3 BE 76 C8 B4 39 58\n    ```\n\nWrite data to a file instead of stdout, but use the file extension to determine\nthe format. The output file will contain the binary data in the\n[CBOR](http://cbor.io/) format.\n\n    ```bash\n    $ echo '{\"bool\":true,\"number\":1.234}' | panser -o file.cbor\n    $ cat file.cbor | xxd -i\n    0xa2, 0x64, 0x62, 0x6f, 0x6f, 0x6c, 0xf5, 0x66, 0x6e, 0x75, 0x6d, 0x62,\n    0x65, 0x72, 0xfb, 0x3f, 0xf3, 0xbe, 0x76, 0xc8, 0xb4, 0x39, 0x58\n    ```\n\nAdd size-based framing to the output.\n[Framing](\u003chttps://en.wikipedia.org/wiki/Frame_(networking)\u003e) is a process to\ncreate specific frames of data from a continuous stream. It aids in buffering,\nreducing memory usage, and simplifying network handling code in applications.\nSize-based framing is prepending the total serialized data length as an unsigned\n32-bit integer in Big Endian (Network Order) to the frame, or message.\n\n    ```bash\n    $ echo '{\"bool\":true,\"number\":1.234}' | panser -r h --sized-output\n    00 00 00 17 82 A4 62 6F 6F 6C C3 A6 6E 75 6D 62 65 72 CB 3F F3 BE 76 C8 B4 39 58\n    ```\n\nThe same can be done for input to remove the size-based framing. Note the use of\nthe `-f` option to indicate the input format is Msgpack and _not_ JSON. The\nfirst four bytes are removed. Framing can be added or removed from any supported\nformat, not just Msgpack.\n\n    ```bash\n    $ echo '{\"bool\":true,\"number\":1.234}' | panser --sized-output | panser -r h -f msgpack --sized-input\n    82 A4 62 6F 6F 6C C3 A6 6E 75 6D 62 65 72 CB 3F F3 BE 76 C8 B4 39 58\n    ```\n\nAn alternative form of framing uses a delimiter byte between frames, or\nmessages. Panser can handle delimiter-based framing in a similar manner to\nsize-based framing. The `--delimited-input` and `--delimited-output` options\ntake a value that is a numeric string representation of a single byte. A radix\nsuffix can be added to indicate the byte notation: (b) binary, (d) decimal, (h)\nhexadecimal, or (o) octal. If no radix suffix is used, then hexadecimal notation\nis assumed. Here, the ASCII newline character (`\\n`, 1010b, 10d, 0Ah, and 012o)\nis used to delimit the binary data. However, the delimiter is not included in\nthe output if the `-r,--radix` option is used.\n\n    ```bash\n    $ echo '{\"bool\":true,\"number\":1.234}' | panser --delimited-output 0Ah | panser -r h -f msgpack --delimited-input 0Ah\n    82 A4 62 6F 6F 6C C3 A6 6E 75 6D 62 65 72 CB 3F F3 BE 76 C8 B4 39 58\n    ```\n\nThe delimiter-based framing can be used to create an interactive console for\nPanser.\n\n    ```bash\n    $ panser -d 0Ah -t Hjson\n    {\"bool\":true}\n    {\n        \"bool\": true\n    }\n    {\"bool\":true,\"number\":1.234}\n    {\n        \"bool\": true,\n        \"number\": 1.234,\n    }\n    ```\n\nAn interactive console with binary output, such as Msgpack, can be created with\nthe delimiter-based framing and the `-r,--radix` option.\n\n    ```bash\n    $ panser -r h -d 0Ah\n    {\"bool\":true\"}\n    81 A4 62 6F 6F 6C C3\n    {\"bool\":true,\"number\":1.234}\n    82 A4 62 6F 6F 6C C3 A6 6E 75 6D 62 65 72 CB 3F F3 BE 76 C8 B4 39 58\n    ```\n\nData can be sent to a network device using the\n[nc](https://linux.die.net/man/1/nc) command. The JSON will be transcoded to\nsize-based framed Msgpack and streamed to the server at the IP address and TCP\nport used with the `nc` command. This was actually the primary motivation for\ncreating the Panser application.\n\n    ```bash\n    $ echo '{\"bool\":true,\"numeber\":1.234}' | panser --sized-output | nc 127.0.0.1 1234\n    ```\n\nInterestingly, Panser can be used in conjunction with the\n[wsta](https://github.com/esphen/wsta) application to send and receive data from\na web socket server.\n\n    ```bash\n    $ ehco '{\"bool\":true}' | panser | wsta 127.0.0.1:1234 | panser -f msgpack -t json\n    {\"bool\":true}\n    ```\n\n## Build\n\nDownload and install the same dependencies listed for [installing the\napplication from source](#source), this includes the latest versions of\n[Rust](https://www.rust-lang.org), [Cargo](https://crates.io), and optionally\n[Pandoc](http://pandoc.org).\n\n### Application\n\nRun the following commands from a terminal:\n\n    git clone https://github.com/volks73/panser.git\n    cd panser\n    cargo build\n\nOr obtain the source as an archive and run the following commands from a terminal:\n\n    tar xf panser-#.#.#.tar.gz\n    cd panser-#.#.#\n    cargo build\n\nwhere `#.#.#` is replaced with the version number of the source distribution,\nrespectively. The `--release` flag can be added to the cargo command to build a\nrelease application instead of a debug application.\n\n### Documentation\n\nDocumentation is available in two forms: (i) [API](#api) and (ii)\n[Manpage](#manpage). The API documentation is for the library/crate while the\nManpage documentation is helpful for the executable/binary.\n\n#### [API](https://volks73.github.io/panser)\n\nObtain the appropriate source and run the following commands from the root\ndirectory of the project in a terminal:\n\n    cargo doc --no-deps\n\nThe output will be available in the `target/doc` folder.\n\n#### [Manpage](https://volks73.github.io/panser/manpage.html)\n\nObtain the appropriate source and run the following commands from the root\ndirectory of the project in a terminal to build the manpage in the\n[groff](https://www.gnu.org/software/groff/) and html formats:\n\n    cargo build --release\n\nOr,\n\n    pandoc -s -t man -o man/panser.1 man/panser.1.md\n    pandoc -s -t html -o manpage.html man/panser.1.md\n\nWhen the `release` profile is used to build the binary, the manpage is\nautomatically generated if pandoc is installed.\n\n## License\n\nThe Panser project is licensed under the [GPL-3.0\nlicense](https://www.gnu.org/licenses/gpl-3.0.en.html). See the\n[LICENSE](https://github.com/volks73/panser/blob/master/LICENSE) file for more\ninformation about licensing and copyright.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvolks73%2Fpanser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvolks73%2Fpanser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvolks73%2Fpanser/lists"}