{"id":16188320,"url":"https://github.com/konsumer/rawproto","last_synced_at":"2025-04-24T01:12:36.692Z","repository":{"id":25228917,"uuid":"103500457","full_name":"konsumer/rawproto","owner":"konsumer","description":"Guess structure of protobuf binary from raw data","archived":false,"fork":false,"pushed_at":"2025-01-22T05:21:14.000Z","size":1297,"stargazers_count":47,"open_issues_count":4,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-24T01:12:29.809Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://konsumer.js.org/rawproto/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/konsumer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"konsumer"}},"created_at":"2017-09-14T07:28:00.000Z","updated_at":"2025-04-23T19:09:57.000Z","dependencies_parsed_at":"2024-03-19T20:28:07.074Z","dependency_job_id":"e2527cf9-2ef8-4938-9d9f-b0823697bece","html_url":"https://github.com/konsumer/rawproto","commit_stats":{"total_commits":68,"total_committers":4,"mean_commits":17.0,"dds":0.05882352941176472,"last_synced_commit":"5c531e5465ab6cd736f446205d01a2e5e1399a91"},"previous_names":[],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konsumer%2Frawproto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konsumer%2Frawproto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konsumer%2Frawproto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/konsumer%2Frawproto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/konsumer","download_url":"https://codeload.github.com/konsumer/rawproto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250540928,"owners_count":21447427,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T07:25:59.441Z","updated_at":"2025-04-24T01:12:36.686Z","avatar_url":"https://github.com/konsumer.png","language":"JavaScript","funding_links":["https://github.com/sponsors/konsumer"],"categories":[],"sub_categories":[],"readme":"# rawproto\n\n[![tests](https://github.com/konsumer/rawproto/actions/workflows/test.yml/badge.svg)](https://github.com/konsumer/rawproto/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/konsumer/rawproto/graph/badge.svg?token=PBL1G8S4WY)](https://codecov.io/gh/konsumer/rawproto) [![NPM Version](https://img.shields.io/npm/v/rawproto)](https://www.npmjs.com/package/rawproto)\n\nGuess structure of protobuf binary from raw data, query binary protobuf without the schema, and output guessed JSON or schema, some CLI utils, and a web tool for exploring raw protobuf.\n\nYou can explore your proto binary data [here](https://konsumer.js.org/rawproto/). Use it to view, generate proto/json files, or select how to parse fields.\n\nIf you are coming form an older version, or another library, check out [migration instructions](#migration).\n\n\n## installation\n\n`npm i rawproto` will add this to your project.\n\nYou can also use `npx rawproto` to run the CLI.\n\n## usage\n\n### CLI\n\nInstall it in your path with `npm i -g rawproto` or use it 1-off with `npx rawproto`. Get help with `rawproto --help`\n\n\n### code\n\nYou can use it, as a library, in code like this:\n\n```js\nimport { readFile } from 'fs/promises'\nimport RawProto from 'rawproto'\n\n// load proto\nconst proto = new RawProto(await readFile('data.pb'))\n\n// get a single field, without parsing the whole tree\nconsole.log(proto.query('1.2.4.10.5:string'))\n\n// you can also pull things like they are arrays/values\nconsole.log(proto['1'][0]['2'][0]['4'][0]['10'].map(r =\u003e r['5'][0].string ))\n\n// guess to decode as JS object\nconsole.log(proto.toJS())\n\n// guess to generate .proto file string\nconsole.log(proto.toProto())\n\n// walk over all fields recursively, calling your callback.\nconst mydata = proto.walk((field) =\u003e {\n  console.log(field)\n\n  // just do whatever it normally does to make JS-object\n  return walkerJS(field)\n})\n```\n\n## types\n\nProtobuf encodes several different possible types for every wire-type. In this lib, we guess the type based on some context-clues, but it will never be perfect, without hand-tuning. Here are the possible types we support:\n\n```\nVARINT - int, bool, string\nFIXED64 - uint, int, bytes, float, string\nLEN - string, bytes, packedIntVar, packedInt32, packedInt64, string\nFIXED32 - int, uint, bytes, float, string\n```\n\n- You can use any [protobuf scalar type-name](https://protobuf.dev/programming-guides/proto3/#scalar).\n- You can use `raw` for any type to get the raw field with bytes + meta.\n- Groups are treated as repeated `LEN` message-fields\n- `LEN` will try to be parsed as sub-tree, but you can override with other types in `query` or with a typemap (for example if it tries to make a sub-message with part of a string)\n\n## query-map\n\nMany things (ui, `toJS`, `toProto`, cli) use `queryMap` which is just a map of `name` to `path:type`. Here is one that works well with [hearthstone test data](test/hearthstone.bin):\n\n```json\n{\n  \"id\": \"1.2.4.1:string\",\n  \"title\": \"1.2.4.5:string\",\n  \"company\": \"1.2.4.6:string\",\n  \"description\": \"1.2.4.7:string\",\n\n  \"media\": \"1.2.4.10\",\n  \"media.dimensions\": \"1.2.4.10.2\",\n  \"media.dimensions.width\": \"1.2.4.10.2.3:uint\",\n  \"media.dimensions.height\": \"1.2.4.10.2.4:uint\",\n  \"media.url\": \"1.2.4.10.5:string\",\n  \"media.type\": \"1.2.4.10.1:uint\",\n  \"media.bg\": \"1.2.4.10.15:string\"\n}\n```\n\nYou can use any types, from above, and set the name to whatever you want. Since it's flat, and JSON doesn't allow multiple keys with same name, it helps to prefix it with the path (like `media`, above.) When JSON/proto is generated, or it is viewed in web UI, it will only use the last segment, anyway.\n\n### protoc\n\nIf you already have some of your types defined in a proto file, or just find that easier to make proto typemaps in (it generally is much nicer, and there is more tooling available) you can use [protoc-gen-typemap](https://github.com/konsumer/protoc-gen-typemap) to generate typemaps:\n\n```\nnpm i -g protoc-gen-typemap\ncat data.pb | protoc --typemap_out=MyMessage:generated mine.proto\n```\n\n\nYou can use partial definitions to just get the part you want, but you have to make sure to drill all the way into the data, like:\n\n```proto\n# this is entrypoint for stuff we want: 1.2.4\nmessage Payload {\n  Message1 field1 = 1;\n  message Message1 {\n    Message2 field2 = 2;\n    message Mesage2 {\n      App field4 = 4;\n    }\n  }\n}\n\n# this is the actual data we care about\nmessage App {\n  string id = 1;\n  string title = 5;\n  string company = 6;\n  string description = 7;\n  repeated Media media = 10;\n}\n\nmessage Media {\n  int type = 1;\n  Dimensions dimensions = 2;\n  string url = 5;\n  string bg = 15;\n}\n\nmessage Dimensions {\n  int width = 3;\n  int height = 4;\n}\n```\n\nThen you would use it like this to generate a typemap:\n\n```\ncat data.pb | protoc --typemap_out=Payload:generated app.proto\n```\n\nAnd you can also just decode, directly:\n\n```\ncat data.pb | protoc --decode=Payload app.proto\n```\n\n\n## migration\n\nI used to have the functionality of this lib split up into several other projects. Here is migration instructions, if you want to update to this one (recommended):\n\n- [protobuf-decoder](https://github.com/konsumer/protobuf-decoder) -  just use [site](https://konsumer.js.org/rawproto/). The code is [here](https://github.com/konsumer/rawproto/tree/master/ui)\n- [rawprotoparse](https://github.com/konsumer/rawprotoparse) - this originally would create JSON from protobuf binary. If you were using this as-is, it had a lot of options, which have been merged into either `toJS` (see [tests](https://github.com/konsumer/rawproto/blob/master/test/json.test.js) for examples.) It may require a little bit more custom-code, if you were not using it with defaults, but overall should work better, and merges shared code that was in both libs. Main thing is that regular `toJS`, without a custom-mapper, will make all values an array, since it's possible for any field ID to be found multiple times.\n- [newrawprotoparser](https://github.com/konsumer/newrawprotoparser) - this was some of the start of ideas for this. No one is probly using this. Essentially, it's the same stuff in [path](https://github.com/konsumer/rawproto/blob/master/test/path.test.js)\n- [protoquery](https://github.com/konsumer/protoquery) - this was some of the start of ideas for this. No one is probly using this. Essentially it's the same stuff in [query](https://github.com/konsumer/rawproto/blob/master/test/query.test.js)\n- rawproto - This lib used to be able to do JSON and generate proto, and provided a different CLI. You should be able to use the new APIs to accomplish all the same stuff, but it may require a bit of a change to your code. Have a look at the [unit-tests](https://github.com/konsumer/rawproto/tree/master/test), to get an idea of how it works.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonsumer%2Frawproto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkonsumer%2Frawproto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkonsumer%2Frawproto/lists"}