{"id":20333268,"url":"https://github.com/protocolbuffers/protoscope","last_synced_at":"2025-09-11T17:39:26.602Z","repository":{"id":41148364,"uuid":"508434677","full_name":"protocolbuffers/protoscope","owner":"protocolbuffers","description":"Protoscope is a simple, human-editable language for representing and emitting the Protobuf wire format. ","archived":false,"fork":false,"pushed_at":"2024-03-13T21:46:35.000Z","size":127,"stargazers_count":337,"open_issues_count":9,"forks_count":38,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-03-29T07:06:57.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/protocolbuffers.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-28T19:48:26.000Z","updated_at":"2025-03-28T11:20:33.000Z","dependencies_parsed_at":"2024-06-18T21:17:28.455Z","dependency_job_id":"e063565c-caad-4335-be30-67dae0cbac9f","html_url":"https://github.com/protocolbuffers/protoscope","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protocolbuffers%2Fprotoscope","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protocolbuffers%2Fprotoscope/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protocolbuffers%2Fprotoscope/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/protocolbuffers%2Fprotoscope/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/protocolbuffers","download_url":"https://codeload.github.com/protocolbuffers/protoscope/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247305933,"owners_count":20917208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T20:30:04.150Z","updated_at":"2025-04-05T08:05:28.582Z","avatar_url":"https://github.com/protocolbuffers.png","language":"Go","readme":"# Protoscope\n\n*Protobuf + Rotoscope*\n\nProtoscope is a simple, human-editable language for representing and emitting\nthe\n[Protobuf wire format](https://developers.google.com/protocol-buffers/docs/encoding).\nIt is inspired by, and is significantly based on,\n[DER ASCII](https://github.com/google/der-ascii), a similar tool for working\nwith DER and BER, wire formats of ASN.1.\n\nUnlike most Protobuf tools, it is normally ignorant of schemata specified in\n`.proto` files; it has just enough knowledge of the wire format to provide\nprimitives for constructing messages (such as field tags, varints, and length\nprefixes). A disassembler is included that uses heuristics to try convert\nencoded Protobuf into Protoscope, although the heuristics are necessarily\nimperfect.\n\nWe provide the Go package `github.com/protocolbuffers/protoscope`, as well as\nthe `protoscope` tool, which can be installed with the Go tool via\n\n```\ngo install github.com/protocolbuffers/protoscope/cmd/protoscope...@latest\n```\n\n`go install` will place the binary in the `GOBIN` directory, which is `~/go/bin`\nby default. See the\n[docs for `go install`](https://pkg.go.dev/cmd/go#hdr-Compile_and_install_packages_and_dependencies)\nfor more details.\n\nFor the language specification and basic examples, see\n[language.txt](/language.txt). Example disassembly can be found under\n[./testdata](/testdata).\n\n## Cookbook\n\nProtoscope can be used in a number of different ways to inspect or create binary\nProtobuf data. This isn't the full breadth of usecases, but they are the ones\nProtoscope (and its ancestor, DER ASCII) were designed for.\n\n### Exploring Binary Dumps\n\nSometimes, while working on a library that emits wire format, it may be\nnecessary to debug the precise output of a test failure. If your test prints out\na hex string, you can use the `xxd` command to turn it into raw binary data and\npipe it into `protoscope`.\n\nConsider the following example of a message with a `google.protobuf.Any` field:\n\n```sh\n$ cat hexdata.txt\n0a400a26747970652e676f6f676c65617069732e636f6d2f70726f746f332e546573744d65737361676512161005420e65787065637465645f76616c756500000000\n$ xxd -r -ps hexdata.txt | protoscope\n1: {\n  1: {\"type.googleapis.com/proto3.TestMessage\"}\n  2: {`1005420e65787065637465645f76616c756500000000`}\n}\n$ xxd -r -ps \u003c\u003c\u003c \"1005420e65787065637465645f76616c756500000000\" | protoscope\n2: 5\n8: {\"expected_value\"}\n`00000000`\n```\n\nThis reveals that four zero bytes sneaked into the output!\n\nIf your test failure output is made up of C-style escapes and text, the `printf`\ncommand can be used instead of `xxd`:\n\n```sh\n$ printf '\\x10\\x05B\\x0eexpected_value\\x00\\x00\\x00\\x00' | protoscope\n2: 5\n8: {\"expected_value\"}\n`00000000`\n```\n\nThe `protoscope` command has many flags for refining the heuristic used to\ndecode the binary.\n\nIf an encoded `FileDescriptorSet` proto is available that contains your\nmessage's type, you can use it to get schema-aware decoding:\n\n```sh\n$ cat hexdata.txt\n086510661867206828d20130d4013d6b000000416c000000000000004d6d000000516e000000000000005d0000de42610000000000005c40680172033131357a0331313683018801758401\n$ xxd -r -ps hexdata.txt | protoscope \\\n  -descriptor-set path/to/fds.pb -message-type unittest.TestAllTypes \\\n  -print-field-names\n1: 101        # optional_int32\n2: 102        # optional_int64\n3: 103        # optional_uint32\n4: 104        # optional_uint64\n5: 105z       # optional_sint32\n6: 106z       # optional_sint64\n7: 107i32     # optional_fixed32\n8: 108i64     # optional_fixed64\n9: 109i32     # optional_sfixed32\n10: 110i64    # optional_sfixed64\n11: 111.0i32  # optional_float, 0x42de0000i32\n12: 112.0     # optional_double, 0x405c000000000000i64\n13: true      # optional_bool\n14: {\"115\"}   # optional_string\n15: {\"116\"}   # optional_bytes\n16: !{        # optionalgroup\n  17: 117     # a\n}\n```\n\nYou can get an encoded `FileDescriptorSet` by invoking\n\n```sh\nprotoc -Ipath/to/imported/protos -o my_fds.pb my_proto.proto\n```\n\n### Modifying Existing Files\n\nSuppose that we have a proto file `foo.bin` of unknown schema:\n\n```sh\n$ protoscope foo.bin\n1: 42\n2: {\n  42: {\"my awesome proto\"}\n}\n```\n\nModifying the embedded string with a hex editor is very painful, because it's\npossible that the length prefix needs to be updated, which can lead to the\nlength prefix on outer messages needing to be changed as well. This is made\nworse by length prefixes being varints, which may grow or shrink and feed into\nfurther outer length prefix updates.\n\nBut `protoscope` makes this into a simple disassemble, edit, assembly loop:\n\n```sh\n$ xxd foo.bin\n00000000: 082a 1213 d202 106d 7920 6177 6573 6f6d  .*.....my awesom\n00000010: 6520 7072 6f74 6f                        e proto\n\n$ protoscope foo.bin \u003e foo.txt  # Disassemble.\n$ cat foo.txt\n1: 42\n2: {\n  42: {\"my awesome proto\"}\n}\n\n$ vim foo.txt  # Make some edits.\n$ cat foo.txt\n1: 43\n2: {\n  42: {\"my even more awesome awesome proto\"}\n}\n\n$ protoscope -s foo.txt \u003e foo.bin  # Reassemble.\n$ xxd foo.bin\n00000000: 082b 1225 d202 226d 7920 6576 656e 206d  .+.%..\"my even m\n00000010: 6f72 6520 6177 6573 6f6d 6520 6177 6573  ore awesome awes\n00000020: 6f6d 6520 7072 6f74 6f                   ome proto\n```\n\nThe `-message-type` option from above can be used when you know the schema to\nmake it easier to find specific fields.\n\n### Describing Invalid Binaries\n\nBecause Protoscope has a very weak understanding of Protobuf, it can be used to\ncreate invalid encodings to verify that some invariant is actually checked by a\nproduction parser.\n\nFor example, the following Protoscope text can be used to create a test that\nensures a too-long length prefix is rejected as invalid.\n\n```\n1: {\n  2:LEN 5   # Explicit length prefix.\n    \"oops\"  # One byte too short.\n}\n```\n\nThis is more conveinent than typing out bytes by hand, because Protoscope takes\ncare of tedious details like length prefixes, varint encoding, float encoding,\nand other things not relevant to the test. It also permits comments, which can\nbe used to specify why the Protoscope snippet produces a broken binary.\n\nProtoscope itself generates test data using Protoscope, which is then checked\nin. Other projects can either check in binary data directly, or use the build\nsystem to invoke `protoscope`, such as with a Bazel `genrule()`.\n\n## Backwards Compatibility\n\nThe Protoscope language itself may be extended over time, but the intention is\nfor extensions to be backwards-compatible. Specifically:\n\n*   The command-line interface to `protoscope` will remain compatible, though\n    new options may be added in the future.\n\n*   Previously valid Protoscope will remain valid and produce the same output.\n    In particular, checking in test data as Protoscope text should be\n    future-proof.\n\n*   Previously invalid Protoscope may become valid in the future if the language\n    is extended.\n\n*   Disassembly is necessarily a heuristic, so its output *may* change over\n    time, but it is guaranteed to produce Protoscope output that will reassemble\n    to the original byte string. `protoscope | protoscope -s` is always\n    equivalent to `cat`.\n\n## Disclaimer\n\nThis is not an official Google project.\n","funding_links":[],"categories":["Go"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprotocolbuffers%2Fprotoscope","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprotocolbuffers%2Fprotoscope","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprotocolbuffers%2Fprotoscope/lists"}