Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dreadl0ck/gopcap
Fast Golang PCAP Reader & Benchmark Comparison
https://github.com/dreadl0ck/gopcap
Last synced: 2 months ago
JSON representation
Fast Golang PCAP Reader & Benchmark Comparison
- Host: GitHub
- URL: https://github.com/dreadl0ck/gopcap
- Owner: dreadl0ck
- License: gpl-3.0
- Created: 2018-11-06T09:41:34.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2019-03-27T21:43:43.000Z (almost 6 years ago)
- Last Synced: 2024-08-03T17:11:17.220Z (5 months ago)
- Language: Go
- Homepage:
- Size: 27.3 KB
- Stars: 73
- Watchers: 2
- Forks: 7
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Authors: AUTHORS
Awesome Lists containing this project
- awesome-network-stuff - **42**星
README
# GOPCAP
[](https://goreportcard.com/report/github.com/dreadl0ck/gopcap)
[](https://raw.githubusercontent.com/dreadl0ck/gopcap/master/docs/LICENSE)
[](https://golang.org)
[](https://godoc.org/github.com/dreadl0ck/gopcap)
This package provides support for reading Packet Capture (PCAP) files efficiently in pure Go
and provides benchmarks against other packages offering the same functionality.
**Update**: I just discovered the go implementation from gopacket https://github.com/google/gopacket/tree/master/pcapgo,
which is comparatively fast and has support for both reading and writing PCAP + PCAP-NG and live capture in pure Go (awesome!).
I've added the gopacket/pcapgo implementation to the benchmarks as well.
I recommend to use the pcapgo library instead of this one, it offers far more functionality.
However, in my benchmarks **this implementation still is slightly faster**.
So, if performance is crucial and you only need to read PCAP, this lib might still be interesting.
**The problem:**
Existing implementations often use **binary.Read** for parsing the binary data into the expected structure.
However, the function makes use of reflection, which hurts performance.
This implementation does not use **binary.Read** to parse the file and packet headers,
and therefore is much faster.
The following API is exported by the package:
```golang
// Count all packet inside the PCAP file at path
func Count(path string) int64
```
```golang
// Reader for PCAP
type Reader
```
```golang
// Open file at path and return reader and error
func Open(filename string) (*Reader, error)
```
```golang
// Close reader
func (r *Reader) Close() error
```
```golang
// ReadNextPacket reads the next packet and returns header,data,error
func (r *Reader) ReadNextPacket() (PacketHeader, []byte, error)
```
```golang
// ReadNextPacketHeader reads the next packet header and returns header,data,error
func (r *Reader) ReadNextPacketHeader() (PacketHeader, []byte, error)
```
Data structures for the PCAP file header and packet headers are provided:
```golang
type FileHeader struct {
// magic number
MagicNumber uint32
// major version number
VersionMajor uint16
// minor version number
VersionMinor uint16
// GMT to local correction
Thiszone int32
// accuracy of timestamps
Sigfigs uint32
// max length of captured packets, in octets
Snaplen uint32
// data link type
Network uint32
}
type PacketHeader struct {
// timestamp seconds
TsSec int32
// timestamp microseconds
TsUsec int32
// number of octets of packet saved in file
CaptureLen int32
// actual length of packet
OriginalLen int32
}
```
## Usage
Reading a PCAP file from disk can be done by calling **gopcap.Open(path)** and looping on **r.ReadNextPacket()** of the returned reader.
Additionally there is a function called **gopcap.Count(path)** that returns the total count of packets in the file (useful for displaying progress).
```golang
// get total packet count
fmt.Println("total:", gopcap.Count(path))
// create reader
r, err := gopcap.Open(path)
if err != nil {
panic(err)
}
defer r.Close()
// loop over packets
for {
h, data, err := r.ReadNextPacket()
if err != nil {
if err == io.EOF {
println("EOF")
break
}
panic(err)
}
fmt.Println(h, len(data))
}
```
## Benchmarks
There are a few pure go implementations for parsing PCAP files available.
The following have not been evaluated in the benchmark, for the given reason(s):
- https://github.com/davecheney/pcap, Latest commit 10760a1 on Aug 19, 2012, fails to compile with various errors
- https://github.com/Lukasa/gopcap, limited in functionality, API only allows to parse a file completely, which is unpractical for big files.
These implementation are included in the benchmarks, in the order they are listed:
- https://github.com/dreadl0ck/gopcap (this package)
- https://github.com/github.com/0intro/pcap
- https://github.com/google/gopacket/pcap
- https://godoc.org/go.universe.tf/netboot
- https://github.com/github.com/miekg/pcap
- https://github.com/google/gopacket/tree/master/pcapgo
The benchmark code fetches a single packet in a loop, and discards all data that is not needed.
Make sure the PCAP file for the test is big enough, otherwise the tests wont produce meaningful results.
I didn't include the dump in the repo because it is around 1.0G in size.
The used PCAP file (**maccdc2012_00000.pcap**) is from the National CyberWatch Mid-Atlantic Collegiate Cyber Defense Competition (MACCDC),
which can be downloaded here: https://www.netresec.com/?page=MACCDC
Here are the results of a few runs on my development machine (MacBook Pro 2018, 32GB RAM, 2,9 GHz Intel Core i9):
$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12 10000000 142 ns/op 111 B/op 1 allocs/op
BenchmarkReadPcap0Intro-12 300000 4714 ns/op 203 B/op 8 allocs/op
BenchmarkReadPcapGoPacket-12 5000000 258 ns/op 0 B/op 0 allocs/op
BenchmarkReadPcapNetboot-12 3000000 510 ns/op 235 B/op 8 allocs/op
BenchmarkReadPcapMiekg-12 3000000 582 ns/op 283 B/op 6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12 10000000 149 ns/op 98 B/op 0 allocs/op
PASS
ok github.com/dreadl0ck/gopcap 10.555s
$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12 10000000 193 ns/op 111 B/op 1 allocs/op
BenchmarkReadPcap0Intro-12 200000 6181 ns/op 224 B/op 8 allocs/op
BenchmarkReadPcapGoPacket-12 5000000 356 ns/op 0 B/op 0 allocs/op
BenchmarkReadPcapNetboot-12 2000000 653 ns/op 223 B/op 8 allocs/op
BenchmarkReadPcapMiekg-12 2000000 737 ns/op 271 B/op 6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12 10000000 200 ns/op 98 B/op 0 allocs/op
PASS
ok github.com/dreadl0ck/gopcap 11.911s
$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12 10000000 145 ns/op 111 B/op 1 allocs/op
BenchmarkReadPcap0Intro-12 300000 4800 ns/op 203 B/op 8 allocs/op
BenchmarkReadPcapGoPacket-12 5000000 262 ns/op 0 B/op 0 allocs/op
BenchmarkReadPcapNetboot-12 3000000 664 ns/op 235 B/op 8 allocs/op
BenchmarkReadPcapMiekg-12 2000000 732 ns/op 271 B/op 6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12 10000000 197 ns/op 98 B/op 0 allocs/op
PASS
ok github.com/dreadl0ck/gopcap 11.557s
$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12 10000000 172 ns/op 125 B/op 2 allocs/op
BenchmarkReadPcap0Intro-12 300000 4175 ns/op 203 B/op 8 allocs/op
BenchmarkReadPcapGoPacket-12 5000000 275 ns/op 0 B/op 0 allocs/op
BenchmarkReadPcapNetboot-12 3000000 659 ns/op 235 B/op 8 allocs/op
BenchmarkReadPcapMiekg-12 2000000 560 ns/op 271 B/op 6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12 10000000 194 ns/op 98 B/op 0 allocs/op
PASS
ok github.com/dreadl0ck/gopcap 11.370s
$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/dreadl0ck/gopcap
BenchmarkReadPcap-12 10000000 196 ns/op 111 B/op 1 allocs/op
BenchmarkReadPcap0Intro-12 300000 4821 ns/op 203 B/op 8 allocs/op
BenchmarkReadPcapGoPacket-12 3000000 436 ns/op 0 B/op 0 allocs/op
BenchmarkReadPcapNetboot-12 2000000 766 ns/op 223 B/op 8 allocs/op
BenchmarkReadPcapMiekg-12 2000000 855 ns/op 271 B/op 6 allocs/op
BenchmarkReadPcapGopacketPcapGo-12 5000000 278 ns/op 104 B/op 1 allocs/op
PASS
ok github.com/dreadl0ck/gopcap 11.904s
Results vary, but this implementation appears to be **slightly faster** than gopacket/pcapgo.
**Can anyone explain why the amount of allocations varies in between different runs?
How can 0 allocs/op happen?**
The gopacket pcap library and fork from miekg both use C bindings.
For the benchmark of the gopacket pcap implementation, the **ZeroCopyReadPacketData()** function was used.