Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/assafmo/joincap
Merge multiple pcap files together, gracefully.
https://github.com/assafmo/joincap
command-line commandline concat forensics join merge network network-analysis packet packet-processing pcap pcap-files pcap-processor sysadmin sysadmin-tool tcpdump tcpdump-capture
Last synced: 1 day ago
JSON representation
Merge multiple pcap files together, gracefully.
- Host: GitHub
- URL: https://github.com/assafmo/joincap
- Owner: assafmo
- License: mit
- Created: 2018-05-31T16:57:22.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2024-11-14T14:51:17.000Z (about 1 month ago)
- Last Synced: 2024-12-13T13:05:51.164Z (9 days ago)
- Topics: command-line, commandline, concat, forensics, join, merge, network, network-analysis, packet, packet-processing, pcap, pcap-files, pcap-processor, sysadmin, sysadmin-tool, tcpdump, tcpdump-capture
- Language: Go
- Homepage:
- Size: 2.49 MB
- Stars: 212
- Watchers: 11
- Forks: 23
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
- awesome-go - joincap - Merge multiple pcap files together, gracefully. - ★ 106 (Software Packages)
- awesome-go-extra - joincap - 05-31T16:57:22Z|2021-03-15T16:44:16Z| (Go Tools / Other Software)
README
# joincap
Merge multiple pcap files together, gracefully.
[](https://circleci.com/gh/assafmo/joincap)
[](https://coveralls.io/github/assafmo/joincap?branch=master)
[](https://goreportcard.com/report/github.com/assafmo/joincap)
[](https://godoc.org/github.com/assafmo/joincap)
## Installation
- Download a precompiled binary from https://github.com/assafmo/joincap/releases
- Or... Use `go get`:
```bash
go get -u github.com/assafmo/joincap
```
- Or use Ubuntu PPA:
```bash
curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add -
sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list
sudo apt update
sudo apt install joincap
```
## Basic Usage
```bash
Usage:
joincap [OPTIONS] InFiles...
Application Options:
-v, --verbose Explain when skipping packets or entire input files
-V, --version Print the version and exit
-w= Sets the output filename. If the name is '-', stdout will be used (default: -)
-c= An integer argument for limiting the pcap size (default: 9223372036854775807)
-p=[micros|nanos] Sets timestamp precision (default: micros)
Help Options:
-h, --help Show this help message
```
## Why?
I believe skipping corrupt packets is better than failing the entire merge job.
When using `tcpslice` or `mergecap` sometimes `pcapfix` is needed to fix bad input pcap files.
1. One option is to try and run merge (`mergecap`/`tcpslice`), if we get errors then run `pcapfix` on the bad pcaps and then run merge again.
- Adds complexity (run -> check errors -> fix -> rerun)
- (If errors) Demands more resources (`pcapfix` processes)
- (If errors) Extends the total run time
2. Another option is to run `pcapfix` on the input pcap files and then merge.
- Extends the total run time by a lot (read and write each pcap twice instead of once)
- Demands more storage (for the fixed pcaps)
- Demands more resources (`pcapfix` processes)
3. We can use `pcapfix` "in memory" with process substitution: `mergecap -w out.pcap <(pcapfix -o /dev/stdout 1.pcap) <(pcapfix -o /dev/stdout 2.pcap)`.
- Adds complexity (build a complex command line)
- Demands more resources (`pcapfix` processes)
- Harder for us to use pathname expansion (e.g. `tcpslice -w out.pcap *.pcap`)
- We have to mind the command line character limit (in case of long pathnames)
- Doesn't work for `tcpslice` (seeks the last packets to calculate time ranges - cannot do this with pipes)
## Error handling: `joincap` vs `mergecap` vs `tcpslice`
### Results
| Use case | joincap | mergecap v2.4.5 | tcpslice v1.2a3 |
| ----------------------------------------------------------------------------------------------------- | ------------------ | ------------------ | ------------------ |
| Corrupt input global header | :heavy_check_mark: | :x: | :x: |
| Corrupt input packet header | :heavy_check_mark: | :x: | :x: |
| Unexpectd EOF
(last packet data is truncated) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Input pcap has no packets
(global header is ok, no first packet header) | :heavy_check_mark: | :heavy_check_mark: | :x: |
| Input file size is smaller than 24 bytes
(global header is truncated) | :heavy_check_mark: | :heavy_check_mark: | :x: |
| Input file size is between 24 and 40 bytes
(global header is ok, first packet header is truncated) | :heavy_check_mark: | :x: | :x: |
| Input file doesn't exists | :heavy_check_mark: | :x: | :x: |
| Input file is a directory | :heavy_check_mark: | :x: | :x: |
| Input file end is garbage | :heavy_check_mark: | :heavy_check_mark: | :x: |
| Input file is gzipped (.pcap.gz) | :heavy_check_mark: | :heavy_check_mark: | :x: |
### Error outputs
| Use case | Error outputs |
| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Corrupt input global header |
- `tcpslice: bad tcpdump file test_pcaps/bad_global.pcap: archaic pcap savefile format`
- `mergecap: The file "test_pcaps/bad_global.pcap" contains record data that mergecap doesn't support. (pcap: major version 0 unsupported)`
| Corrupt input packet header |
- tcpslice: Infinite loop?
- `mergecap: The file "test_pcaps/bad_first_header.pcap" appears to be damaged or corrupt. (pcap: File has 2368110654-byte packet, bigger than maximum of 262144)`
| Unexpectd EOF
(last packet data is truncated) | |
| Input pcap has no packets
(global header is ok, no first packet header) |
- tcpslice: Outputs empty pcap (Only global header)
| Input file size is smaller than 24 bytes
(global header is truncated) |
- `tcpslice: bad tcpdump file test_pcaps/empty: truncated dump file; tried to read 4 file header bytes, only got 0`
| Input file size is between 24 and 40 bytes
(global header is ok, first packet header is truncated) |
- `tcpslice: bad status reading first packet in test_pcaps/partial_first_header.pcap: truncated dump file; tried to read 16 header bytes, only got 11`
- `mergecap: The file "test_pcaps/partial_first_header.pcap" appears to have been cut short in the middle of a paket.`
| Input file doesn't exists |
- `tcpslice: bad tcpdump file ./not_here: ./not_here: No such file or directory`
- `mergecap: The file "./not_here" doesn't exist.`
| Input file is a directory |
- `tcpslice: bad tcpdump file examples: error reading dump file: Is a directory`
- `mergecap: "examples" is a directory (folder), not a file.`
| Input file end is garbage |
- `tcpslice: problems finding end packet of file test_pcaps/bad_end.pcap`
| Input file is gzipped (.pcap.gz) |
- `tcpslice: bad tcpdump file test_pcaps/ok.pcap.gz: unknown file format`
### How to reproduce
| Use case | How to reproduce |
| ----------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Corrupt input global header |
- `joincap -w out_joincap.pcap test_pcaps/bad_global.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/bad_global.pcap`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/bad_global.pcap`
| Corrupt input packet header |
- `joincap -w out_joincap.pcap test_pcaps/bad_first_header.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/bad_first_header.pcap`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/bad_first_header.pcap`
| Unexpectd EOF
(last packet data is truncated) |
- `joincap -w out_joincap.pcap test_pcaps/unexpected_eof_on_first_packet.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/unexpected_eof_on_first_packet.pcap`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/unexpected_eof_on_first_packet.pcap`
- `joincap -w out_joincap.pcap test_pcaps/unexpected_eof_on_second_packet.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/unexpected_eof_on_second_packet.pcap`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/unexpected_eof_on_second_packet.pcap`
| Input pcap has no packets
(global header is ok, no first packet header) |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap`
| Input file size is smaller than 24 bytes
(global header is truncated) |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/empty`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/empty`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/empty`
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap`
- `tcpslic -De -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap`
| Input file size is between 24 and 40 bytes
(global header is ok, first packet header is truncated) |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap`
- `tcpslic -De -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap`
| Input file doesn't exists |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap ./not_here`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap ./not_here`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap ./not_here`
| Input file is a directory |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/`
| Input file end is garbage |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap`
| Input file is gzipped (.pcap.gz) |
- `joincap -w out_joincap.pcap test_pcaps/ok.pcap.gz`
- `mergecap -w out_mergecap.pcap test_pcaps/ok.pcap.gz`
- `tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap.gz`
## Benchmarks
| | Version | Speed | Time |
| ------------ | ------- | -------- | -------- |
| **mergecap** | 3.2.2 | 590MiB/s | 0m5.632s |
| **tcpslice** | 1.2a3 | 838MiB/s | 0m3.666s |
| **joincap** | 0.10.2 | 562MiB/s | 0m5.462s |
- Merging 3 files with total size of 2.99994GiB.
- Running on Linux 5.4.0-21-generic, with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (with SSE4.2), with 31765 MB of physical memory, with locale C, with zlib 1.2.11.