https://github.com/datanoisetv/shannon
Zero-instrumentation L7 observability for Linux via eBPF. See plaintext HTTP/HTTPS, gRPC, Postgres, MySQL, MongoDB, Redis, Kafka, Cassandra from any process — including through TLS — without keys, sidecars, or code changes. Pure Rust (aya), kernel 5.8+.
https://github.com/datanoisetv/shannon
apm aya bpf cassandra ebpf ebpf-tools grpc http kafka linux mongodb mysql networking observability postgresql redis rust service-mesh tls tracing
Last synced: about 2 months ago
JSON representation
Zero-instrumentation L7 observability for Linux via eBPF. See plaintext HTTP/HTTPS, gRPC, Postgres, MySQL, MongoDB, Redis, Kafka, Cassandra from any process — including through TLS — without keys, sidecars, or code changes. Pure Rust (aya), kernel 5.8+.
- Host: GitHub
- URL: https://github.com/datanoisetv/shannon
- Owner: DatanoiseTV
- License: apache-2.0
- Created: 2026-04-24T01:33:14.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-04-24T03:27:52.000Z (about 2 months ago)
- Last Synced: 2026-04-24T03:29:15.992Z (about 2 months ago)
- Topics: apm, aya, bpf, cassandra, ebpf, ebpf-tools, grpc, http, kafka, linux, mongodb, mysql, networking, observability, postgresql, redis, rust, service-mesh, tls, tracing
- Language: Rust
- Homepage:
- Size: 42.6 MB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE-APACHE
- Security: SECURITY.md
- Roadmap: docs/roadmap.md
Awesome Lists containing this project
README
# shannon
> Named after Claude Shannon. Turns a stream of bytes on a wire into meaning you can read.
`shannon` is a zero-instrumentation L7 observability tool for Linux. Drop a single
binary on a host (or a Kubernetes node) and see, in real time, every HTTP, gRPC,
Postgres, Redis, MySQL and DNS request the machine is making — including requests
over TLS, without configuring any keys.
It does this with eBPF: kernel probes on TCP, uprobes on `libssl` (and other TLS
runtimes), and protocol parsers in userspace. No sidecars. No application changes.
No private keys. No kernel modules.
## Status
Pre-alpha. Kernel ≥ 5.8 with BTF is required. Tested on Debian 13 / kernel 6.12.
API and event schema are **not** stable.
## Quick start
```bash
# interactive TUI (default)
sudo shannon
# just postgres queries from pid 1234
sudo shannon trace -p 1234 --protocol postgres
# top-like view, sorted by p99 (also: rps, p50, bytes, errors)
sudo shannon top --sort p99
# scope by process name (glob) or by Kubernetes pod
sudo shannon trace --comm 'nginx*'
sudo shannon record --pod billing-7d8c -o billing.jsonl.zst
# attach uprobes (libssl + libsqlite3) to a static binary. Covers Go apps
# bundling their own TLS, custom Rust builds, appliance firmware in a
# single ELF. Best-effort: missing symbols silently skip.
sudo shannon trace --attach-bin /opt/my-app/bin/server
# service-map: who talks to whom, by protocol
sudo shannon map # ANSI-redrawn table
sudo shannon map --format tui # ratatui: scroll, sort, quit
sudo shannon map --format dot | dot -Tsvg > map.svg
sudo shannon map --format json > edges.ndjson
# trace + dump every X.509 seen on the wire, flag self-signed / weak
# sig-alg / short RSA / expired / long-validity anomalies inline
sudo shannon trace --dump-certs /tmp/shannon-certs
# pin those certs: dump once, review, keep the .der files you trust,
# then re-run with --cert-pin — anything new trips "not in allowlist".
sudo shannon trace --cert-pin /tmp/shannon-certs
# pre-flight: is this box supported?
shannon doctor
# record everything to a file (zstd-compressed JSONL)
sudo shannon record -o capture.jsonl.zst --rotate 100M
# play it back later (pipe to jq, feed the TUI, or summarise)
sudo shannon trace --replay capture.jsonl.zst
shannon analyze capture.jsonl.zst
# build an API catalog from observed HTTP traffic and export OpenAPI 3.0
sudo shannon trace --catalog catalog.jsonl --openapi spec.yaml
# infer a .proto schema from raw protobuf messages a service is sending
shannon proto-infer --samples /tmp/grpc-bodies -o inferred.proto
# ask a local LLM (Ollama / LM Studio / vLLM) about an existing capture
shannon ask --catalog catalog.jsonl "what processes talked to s3?"
# shell completions
shannon completions zsh > ~/.zfunc/_shannon
```
## Install
### From a release `.deb` (Debian / Ubuntu, x86_64 + aarch64)
Each tagged release ships per-arch `.deb`s and SHA-256 sums:
```bash
ver=0.1.0
arch=$(dpkg --print-architecture) # amd64 or arm64
curl -LO https://github.com/DatanoiseTV/shannon/releases/download/v${ver}/shannon_${ver}-1_${arch}.deb
sudo dpkg -i shannon_${ver}-1_${arch}.deb
sudo shannon doctor # confirm BTF + kprobe targets
sudo systemctl enable --now shannon # optional: continuous background recorder
```
The package installs `/usr/bin/shannon`, a hardened systemd unit at
`/usr/lib/systemd/system/shannon.service` (disabled by default — opt
in if you want a recorder service), `/etc/default/shannon` for tunables,
and the docs under `/usr/share/doc/shannon/`.
### Kubernetes (DaemonSet)
```bash
kubectl apply -f https://raw.githubusercontent.com/DatanoiseTV/shannon/main/deploy/k8s/daemonset.yaml
```
Creates a `shannon` namespace, runs one recorder per node with
`hostPID + hostNetwork`, writes rotating captures under each node's
`/var/log/shannon/`, and exposes `:9750/metrics` for Prometheus.
See [`deploy/k8s/README.md`](deploy/k8s/README.md) for hardened
config (drop `privileged`, explicit caps) and a `PodMonitor`.
### Container image
```bash
docker run --rm --privileged --pid=host --network=host \
-v /sys/fs/bpf:/sys/fs/bpf \
-v /sys/kernel/btf:/sys/kernel/btf:ro \
ghcr.io/datanoisetv/shannon:latest doctor
```
Multi-arch (amd64 + arm64), tagged on every release plus `:latest`
on stable tags and `:main` on every push to main.
### From source
```bash
git clone https://github.com/DatanoiseTV/shannon
cd shannon
rustup toolchain install nightly --component rust-src
cargo install bpf-linker
cargo xtask build --release
sudo ./target/release/shannon doctor
```
### Cross-building from a non-Linux workstation
shannon is Linux-only (eBPF). If you develop on macOS, `scripts/push-build.sh`
pushes the tree to a Linux host and runs cargo there:
```bash
# defaults: rsync to syso@10.243.243.8:~/shannon, sources ~/.cargo/env
SHANNON_HOST=you@yourhost SHANNON_PATH=shannon scripts/push-build.sh build
scripts/push-build.sh test parsers # any cargo-test filter
scripts/push-build.sh cargo fmt --check # any cargo subcommand + args
scripts/push-build.sh smoke # build + run the UDP smoke test
scripts/push-build.sh shell # rsync + drop into an ssh session
```
See [docs/architecture.md](docs/architecture.md) for how it works,
[SECURITY.md](SECURITY.md) for what it can and cannot see, and
[DEMO.md](DEMO.md) for a scripted end-to-end run with verbatim output
from each parser (DNS, HTTP, TLS, Redis, MQTT, CoAP, SQLite uprobes).
## What it decodes today
**52 wire-format parsers** plus five overlay classifiers (gRPC,
GraphQL, NTLM, LLM-API, Socket.IO) span web, databases, messaging,
mail, directory, telephony, SMS / carrier, media-streaming,
remote-access, operational-technology, IoT / constrained devices,
file-sharing, VPN, AAA / network-management, and legacy chat.
### Web + APIs
| Protocol | Notes |
|---|---|
| HTTP/1.x | Requests, responses, headers, bodies, chunked + range reassembly |
| HTTP/2 | HPACK, per-stream framing |
| gRPC | On HTTP/2: service, method, status; body decode with `--proto` |
| GraphQL | On HTTP: operation name + root field (query / mutation / subscription) |
| LLM APIs | OpenAI / Anthropic (incl. `/v1/messages/count_tokens` Claude Code uses) / Gemini / Azure / Bedrock / Ollama / OpenAI-compat — provider, endpoint, model, streaming-mode, prompt-bytes |
| NTLM | Type 1 / 2 / 3 (Negotiate / Challenge / Authenticate); user + domain + workstation in cleartext; works inside HTTP `Authorization: NTLM/Negotiate`, SMB SessionSetup, LDAP SASL GSS-SPNEGO |
| WebSocket | RFC 6455 frames; follows HTTP/1.1 `101 Upgrade` handshake |
| Socket.IO / Engine.IO | Event name, namespace, JSON args, ack IDs |
| TLS 1.0-1.3 | ClientHello / ServerHello SNI + ALPN + cipher-suite inspection; ServerHello warnings for legacy versions, NULL / EXPORT / RC4 / 3DES / CBC-SHA1 / anon-DH ciphers |
### Databases
| Protocol | Notes |
|---|---|
| Postgres | Startup, Simple / Extended query, bind parameters |
| MySQL | COM_QUERY, COM_STMT_PREPARE/EXECUTE |
| MongoDB wire | `OP_MSG`, `OP_QUERY`, BSON decode |
| Redis | RESP2 + RESP3 |
| Cassandra CQL | Opcodes, frame headers, query strings |
| Memcached | ASCII + binary protocols |
| Oracle TNS | CONNECT descriptor, SERVICE_NAME / SID / PROGRAM / USER |
| MS SQL Server TDS | PreLogin / Login7 (user / server / app / db), batch / RPC |
### Messaging + streaming + IoT
| Protocol | Notes |
|---|---|
| Kafka wire | Produce / Fetch / Metadata / OffsetCommit, API 0-12 |
| AMQP 0.9.1 (RabbitMQ) | Full class/method table, basic.publish routing-key + exchange |
| MQTT 3.1.1 / 5 | CONNECT / PUBLISH / SUBSCRIBE with topic + QoS |
| CoAP | RFC 7252 on udp/5683; type, code, message-id, assembled Uri-Path / Uri-Query, Content-Format |
| NATS | Text protocol: PUB / SUB / MSG / HPUB / HMSG / INFO |
| STUN / TURN | WebRTC signalling, XOR-MAPPED-ADDRESS decode, SOFTWARE |
### Mail + directory
| Protocol | Notes |
|---|---|
| IMAP | Tagged command framing, LOGIN redacted |
| POP3 | USER + PASS (password redacted) |
| SMTP | HELO / EHLO / AUTH (credentials redacted) + MAIL FROM / RCPT TO |
| LDAP | BER: BindRequest (password redacted), SearchRequest with scopes |
| Kerberos v5 | AS-REQ / AS-REP / TGS-REQ / ... with realm + cname + sname |
### Remote access + proxy + telephony
| Protocol | Notes |
|---|---|
| SSH | Banner + software identification |
| RDP | X.224 ConnectionRequest, mstshash= username leak, TLS/CredSSP negotiation |
| Telnet | IAC option negotiation + cleartext text extraction |
| FTP | USER / PASS (redacted) / RETR / STOR / MLSD + reply codes |
| SOCKS4 / SOCKS5 | CONNECT / BIND / UDP-ASSOCIATE with DOMAIN / IPv4 / IPv6 |
| SIP | INVITE / REGISTER / ... with Call-ID + Via + From/To + User-Agent |
| RTSP | DESCRIBE / SETUP / PLAY / TEARDOWN on IP cameras + media servers |
| SMPP | SMS peer-to-peer: bind_* (system_id + redacted password), submit_sm |
| IRC | PASS (redacted) / NICK / USER / JOIN / PRIVMSG / numeric replies |
### Operational-technology (ICS / SCADA / building automation)
| Protocol | Notes |
|---|---|
| Modbus/TCP | Function codes on tcp/502 |
| Siemens S7comm | TPKT + COTP + S7; ROSCTR + function-code decode |
| EtherNet/IP + CIP | ODVA encapsulation (tcp/44818 + 2222); session + command decode |
| DNP3 | IEEE 1815 link-layer framing (tcp/20000) |
| IEC-104 | Telecontrol APDU: I / S / U frames + ASDU TypeID catalogue |
| OPC-UA | IEC 62541-6 §7.1.2 binary framing (tcp/4840) |
| BACnet/IP | BVLC + NPDU + APDU; readProperty / writeProperty / Who-Is / I-Am |
### Infrastructure + auth + management
| Protocol | Notes |
|---|---|
| DNS | Questions + answers over tcp/udp 53 |
| DHCP | Op / transaction / chaddr MAC / options (Host Name, Vendor Class) |
| TFTP | RRQ / WRQ / DATA / ACK / ERROR / OACK with options |
| NTP | Full 48-byte header; stratum / mode / ref_id (GPS, PPS, LOCL) |
| RADIUS | Access-Request / Accept / Reject; User-Name + Called/Calling-Station |
| TACACS+ | AUTHEN / AUTHOR / ACCT; flags incl. UNENCRYPTED warning |
| SNMP v1/v2c | Version + community string + PDU type + first OID |
| Syslog | RFC 3164 + RFC 5424 + RFC 6587 octet-counted framing |
| SSDP | UDP discovery (mDNS-adjacent) |
### File sharing + VPN
| Protocol | Notes |
|---|---|
| SMB2 / SMB3 | TreeConnect share path, Create filename (UCS-2LE), NT_STATUS names |
| NFS / ONC-RPC | Record-marker framing; NFSv3 / MOUNT / PORTMAP / NLM program + procedure decode |
| WireGuard | HandshakeInit / Response / CookieReply / TransportData; ephemeral-key preview |
### Library uprobes (local, not on the wire)
| Library | Hook |
|---|---|
| libsqlite3 | `sqlite3_prepare_v2` + `sqlite3_exec` — full SQL text + db handle |
### TLS runtimes lifted for plaintext
| Runtime | Hook | Status |
|---|---|---|
| OpenSSL / libssl | `SSL_{read,write,read_ex,write_ex}` uprobes | shipped |
| BoringSSL | same symbols as libssl | shipped |
| GnuTLS | `gnutls_record_{send,recv}` uprobes | shipped |
| QUIC v1 | client Initial decrypted with DCID-derived keys (RFC 9001); SNI + ALPN recovered | shipped (1-RTT deferred) |
| NSS | `PR_Read` / `PR_Write` + `ssl3_SendPlainText` uprobes | planned |
| Go `crypto/tls` | per-binary symbol scan → `crypto/tls.(*Conn).{R,W}` | planned |
| Rust `rustls` | per-binary symbol scan → `rustls::Connection` plaintext path | planned |
Each connection carries a protocol state machine that can *upgrade itself*:
HTTP/1.1 → WebSocket → Socket.IO (on event frames), HTTP/2 → gRPC
(on `application/grpc`), and any TCP → TLS (on a ClientHello record).
Nothing you configure.
Both transports captured in both directions — **TCP** via
`tcp_sendmsg` / `tcp_recvmsg` kprobes and **UDP** via `udp_sendmsg`
+ `udp_recvmsg` / `udp_recvmsg_ret` (IPv4 + IPv6 — dst / src address
+ port read off `struct sock` with a `msg->msg_name` fallback for
unconnected sockets).
Deferred to v0.2: NSS, Go `crypto/tls`, Rust `rustls`, Java JSSE,
QUIC 1-RTT payload (needs the TLS master secret shannon never sees).
### Process identity
Every event also carries the originating process — pid, comm, uid/gid,
and the resolved container or pod. Cgroup paths are walked at startup
and refreshed periodically; Docker, containerd, CRI-O, Podman and
plain systemd units all decode to a friendly name. On Kubernetes
nodes, `/var/log/pods/` is parsed so you see `payments/billing-7d8c`
instead of a UID hex string, and container entries inherit the pod
context as `payments/billing-7d8c/`.
## Privacy
`shannon` sees plaintext payloads. Redaction is **on by default** (`--redact auto`)
and strips `Authorization`, `Cookie`, `Set-Cookie`, and query-string params matching
`*token*|*password*|*secret*|api_key`. Run `shannon trace --redact strict` to strip
all headers and bodies. You opt into visibility with `--redact off`; never the
other way around. See [SECURITY.md](SECURITY.md).
## Use cases
- **Incident response**. "Why is the billing service slow?" → `sudo shannon
top --sort p99 --group-by endpoint` on the affected node. No redeploy,
no sidecar, no code change.
- **Debugging microservice calls**. See exact HTTP requests one service is
making to another, including headers and body, even over TLS, without
touching either service's code.
- **Tracing production bugs that don't reproduce locally**. Capture real
traffic with `shannon record -o live.jsonl.zst`, ship the file to a
dev box, replay with `shannon trace --replay`.
- **Security posture audits**. Which processes talk to which external
IPs, and what credentials flow? `shannon trace --peer 0.0.0.0/0
--redact off --protocol http` (use responsibly — see disclaimer).
- **Understanding a black-box binary**. Point shannon at a pid with
`shannon trace -p $(pidof mysterybin)` and watch it talk.
- **CI / integration tests**. Record a baseline of service calls, then
assert behavioural invariants in CI.
- **Capacity planning**. `shannon analyze capture.jsonl.zst` gives
per-endpoint RPS and latency distributions from a representative
recording.
## Disclaimer
This is **research-grade software**. It installs eBPF programs into the
Linux kernel and reads plaintext bytes from every TCP socket on the host
it runs on.
- **No warranty.** shannon is provided "as is". The authors and
contributors take no liability for any malfunction, system instability,
data loss, damage, or loss of business arising from its use.
- **No fitness for any particular purpose** — including production,
compliance, or legal use — is implied or guaranteed.
- **You are responsible for how you use it.** shannon is a tool; the
operator decides what to point it at. Using it to observe traffic
belonging to parties who have not authorised such observation may be
illegal in your jurisdiction (wiretap, privacy, data-protection laws).
The authors take no responsibility for misuse, abuse, or use in
violation of any law, contract, or policy.
- **Privilege boundary.** Running shannon is equivalent to running as
root — treat it accordingly. Do not deploy without understanding
[SECURITY.md](SECURITY.md).
Use only on systems you own or have explicit written permission to
observe. Respect the privacy of users whose traffic crosses those
systems.
## License
Licensed under either of [Apache 2.0](LICENSE-APACHE) or [MIT](LICENSE-MIT) at
your option. Unless you explicitly state otherwise, any contribution intentionally
submitted for inclusion in the work by you, as defined in the Apache-2.0 license,
shall be dual licensed as above, without any additional terms or conditions.