Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mxrch/ProtoDeep
🏄♂️ Decode and analyze protobuf efficiently.
https://github.com/mxrch/ProtoDeep
Last synced: 6 days ago
JSON representation
🏄♂️ Decode and analyze protobuf efficiently.
- Host: GitHub
- URL: https://github.com/mxrch/ProtoDeep
- Owner: mxrch
- License: mit
- Created: 2023-03-11T13:34:45.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-02T02:28:07.000Z (about 1 month ago)
- Last Synced: 2024-10-14T15:04:47.125Z (about 1 month ago)
- Language: Python
- Homepage:
- Size: 957 KB
- Stars: 107
- Watchers: 3
- Forks: 5
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
#### 🐍 Now Python 3.13 compatible !
# 🏄♂️ Description
ProtoDeep is an easy to use tool that allows you to decode and analyze protobuf data.
It is **heavily based** on the well-established [Blackbox Protobuf](https://github.com/nccgroup/blackboxprotobuf) package, and inspired by [Protobuf Inspector](https://github.com/mildsunrise/protobuf-inspector) for the custom definitions feature.\
This project was originally intended to be integrated into the [GHunt RDTK](https://github.com/mxrch/ghunt-rdtk), but the script grew a lot and ended up becoming a tool that can help many people.
Features :
- CLI usage
- Python library usage
- Make your own definitions
- Easily match / filter data
- Export and compile on the fly
Example of using ProtoDeep on the Google's Play Store `searchList` endpoint, with custom definitions :
# ✔️ Requirements
- Python >= 3.10
# ⚙️ Installation
```bash
$ pip3 install pipx
$ pipx ensurepath
$ pipx install protodeep
```
It will automatically use venvs to avoid dependency conflicts with other projects.
# 💃 Usage
## Help menu
```
Usage: main.py [-h] -t TYPE [-d DEFINITIONS] [-na] [-nk] [-s] [-b] [-hx] [-bi NUMBER] [-he] [-np] [-m MASK] [-mk MASK]
[-mv MASK] [-f MASK] [-fk MASK] [-fv MASK] [-epf [PROTOFILE_FILENAME]] [-epd [PROTODEEP_FILENAME]]
[-c [PYTHON_FILENAME]] [-n SCHEMA_NAME]
[proto_file]
Positional Arguments:
proto_file
Options:
-h, --help show this help message and exit
-t, --type TYPE Either protobuf (raw protobuf content), or protodeep (a ProtoDeep file).
-d, --definitions DEFINITIONS
The file containing the custom protobuf definitions.
-na, --no-autodetect Don't try to autodetect if it's a raw HTTP request.
-nk, --named-keychains
Show and extract only named keychains.
-s, --stdin Parse from stdin.
-b, --base64 If this is a base64 input, so it automatically decodes it.
-hx, --hex If this is a hex input, so it automatically decodes it.
-bi, --bruteforce-index NUMBER
The index up to which to try bruteforce to find Protobuf content. Default : 20
-he, --hide-empty Hide the empty values.
-np, --no-print Don't print the decoded protobuf.
-m, --match MASK Match anything with the given string. You can use '?' and '*' to wildcard match.Ex : "*token*"
-mk, --match-keychain MASK
Match keychains with the given string.
-mv, --match-value MASK
Match values with the given string.
-f, --filter MASK Filter anything with the given string. You can use '?' and '*' to wildcard match.
-fk, --filter-keychain MASK
Filter keychains with the given string.
-fv, --filter-value MASK
Filter values with the given string.
-epf, --export-protofile [PROTOFILE_FILENAME]
Export the proto file with the definitions.
-epd, --export-protodeep [PROTODEEP_FILENAME]
Export a protodeep file, to reuse in ProtoDeep.
-c, --compile [PYTHON_FILENAME]
Compile protobuf into a Python file.
-n, --name SCHEMA_NAME
Name of the schema when exporting into a proto file.
```
## Concepts
Here are the main concepts to know when using ProtoDeep :
### Output example
- Keychains : Since the protobuf is made of nested keys and values, keychains are a way to precisely identify a value in the decoded protobuf. It's the key sequence used to access the value.
- Pretty Keychains : This is the same as keychains, except that the keys are replaced by the names defined in the custom definitions.
- Type : Type of the value. Supported types are listed in this [blackboxprotobuf's file](https://github.com/nccgroup/blackboxprotobuf/blob/master/lib/blackboxprotobuf/lib/types/type_maps.py).
- Value : The value found in the protobuf data.
- Iterator : ProtoDeep will try to autodetect repeated messages, and will print elements of these arrays with the `i` key, so you can know the position of the element in the list.
### Custom definitions file
- Definitions : It is a JSON file, containg a dict with the keychains as keys, and names as values. You can specify the type of a value by adding `:` next to it. By doing so, ProtoDeep will detect it, and relaunch the decoding of the protobuf data with this new type. *Note that it will only work when using protobuf data, not a protodeep file, since data has already been decoded.*
**Have fun 🥰💞**
# 🧑💻 Developers
To use ProtoDeep as a lib, you can't use pipx because it uses a venv.\
So you should install ProtoDeep with pip :
```bash
$ pip3 install protodeep
```
And now, you should be able to `import protodeep` in your projects like this :
```python
from protodeep.lib import guess_schema
with open('protobuf_data.bin', 'rb') as f:
raw = f.read()
protodeep_schema = guess_schema(data=raw)
protodeep_schema.pretty_print(hide_empty=True, filter_any=["*term_to_filter*"])
protodeep_schema.export_protodeep("obj.pdeep")
```
## Testing
Thanks to [learn-more](https://github.com/learn-more), tests are now available, to test the CLI and lib usage !\
You can launch the tests by doing :
```bash
$ pip3 install -r requirements-dev.txt
$ pytest
```
*Tests are run automatically through [GitHub Actions](https://github.com/mxrch/ProtoDeep/actions).*
# 📕 Cheatsheet
*Some examples so you know how to use protodeep :*
Reading a protobuf file:
```bash
$ protodeep protobuf_data.bin -t protobuf
```
Read a protobuf file, provide a custom definitions file, hide the output, export to protofile & protodeep, and compile a Python file called "final.py" :
```bash
$ protodeep protobuf_data.bin -t protobuf -d search_ps_defs.json -np -epf -epd -c final.py
```
*Names for the arguments `--export-protofile` / `--export-protodeep` / `--compile` are optional. If they aren't set, a default name will be used.*
Read protobuf from stdin, provide a custom definitions file, match the keychain "11,1,1,2", hide the empty values, and filter lines where the word "access" and "denied" are present, and lines where the word "tiktok" is present:
```bash
$ curl -s | protodeep --stdin -t protobuf -mk "11,1,1,2" -he -f "*access*denied*" -f "*tiktok*"
```
*Matching / filtering arguments can be used as many times as you like.*
## Thanks
- The [NCC Group](https://github.com/nccgroup) for the super useful [blackboxprotobuf](https://github.com/nccgroup/blackboxprotobuf) project
- [mildsunrise](https://github.com/mildsunrise) for [protobuf-inspector](https://github.com/mildsunrise/protobuf-inspector)
- The HideAndSec team 💜 (blog : https://hideandsec.sh)
## Sponsors
Thanks to these awesome people for supporting me !
\
You like my work ?\
[Sponsor me](https://github.com/sponsors/mxrch) on GitHub ! 🤗