https://github.com/erichutchins/geoipsed
Fast, inline geolocation decoration of IPv4 and IPv6 addresses written in Rust
https://github.com/erichutchins/geoipsed
cli dfir geoip
Last synced: 6 months ago
JSON representation
Fast, inline geolocation decoration of IPv4 and IPv6 addresses written in Rust
- Host: GitHub
- URL: https://github.com/erichutchins/geoipsed
- Owner: erichutchins
- License: unlicense
- Created: 2022-06-17T01:43:01.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-01-21T18:19:33.000Z (over 1 year ago)
- Last Synced: 2024-04-17T18:46:29.936Z (over 1 year ago)
- Topics: cli, dfir, geoip
- Language: Rust
- Homepage:
- Size: 44.9 KB
- Stars: 24
- Watchers: 4
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# geoipsed
*Fast, inline geolocation decoration of IPv4 and IPv6 addresses written in Rust*
## Concept
IP address metadata is essential to network defense and incident response. City and country-level geolocation can be clues to rule in or rule out IPs of interest. ASN metadata provides insights of network ownership as well as simply identifying which IPs are internet-routable vs which are internal, private, or bogons.
In command line log analysis, IP geolocation metadata is most useful in *addition* to the already available log metadata on each line. `geoipsed` enriches IP addresses *in place* leaving the existing context intact.
## Features
* IPv4 and IPv6 address support
* City, Country, ASN, time zone fields among the available metadata
* Flexible templating to customize how `geoipsed` decorates matching IPs
* Coloring to more readily spot the IPs in the logs
* Optional mode to just emit the matching IPs just like grep/ripgrep `-o` parameter
* Spaces are removed from decoration labels so as not to mess up column numbering in your logs
## Install
`geoipsed` uses Maxmind's GeoLite2-ASN.mmdb and GeoLite2-City.mmdb files. Follow the instructions to obtain these files here: https://dev.maxmind.com/geoip/updating-databases
Currently, `geoipsed` looks exclusively to `/usr/share/GeoIP` for the mmdb files, however you can override this with the `-I` command line option or the environment variable `MAXMIND_MMDB_DIR`.
Build and install with cargo:
```
cargo install geoipsed
```
## Usage
```
geoipsed 0.1.3
Inline decoration of IPv4 and IPv6 address geolocations
USAGE:
geoipsed [OPTIONS] [FILE]...
ARGS:
... Input file(s) to process. Leave empty or use "-" to read from stdin
OPTIONS:
-C, --color Use markers to highlight the matching strings [default: auto]
[possible values: always, never, auto]
-h, --help Print help information
-I Specify directory containing GeoLite2-ASN.mmdb and
GeoLite2-City.mmdb [env: MAXMIND_MMDB_DIR=]
-L, --list-templates Display a list of available template substitution parameters to use
in --template format string
-o, --only-matching Show only nonempty parts of lines that match
-t, --template Specify the format of the IP address decoration. Use the
--list-templates option to see which fields are available. Field
names are enclosed in {}, for example "{field1} any fixed string
{field2} & {field3}"
-V, --version Print version information
```
Available fields to use in customizing the `-t` parameter:
```
; geoipsed -L
Available template geoip field names are:
{ip}
{asnnum}
{asnorg}
{city}
{continent}
{country_iso}
{country_full}
{latitude}
{longitude}
{timezone}
```
## Benchmark
Comparing the Rust implementation to a basic Python version against 30,000 lines (~23MB decompressed) of Suricata json eve logs:
```
Benchmark 1: zstdcat ../30k.log.zst | target/release/geoipsed
Time (mean ± σ): 157.1 ms ± 16.7 ms [User: 167.8 ms, System: 28.9 ms]
Range (min … max): 137.1 ms … 209.9 ms 21 runs
Benchmark 2: zstdcat ../30k.log.zst | python python/geoipsed.py
Time (mean ± σ): 15.209 s ± 0.929 s [User: 15.213 s, System: 0.210 s]
Range (min … max): 14.312 s … 17.076 s 10 runs
Summary
'zstdcat ../30k.log.zst | target/release/geoipsed' ran
96.80 ± 11.84 times faster than 'zstdcat ../30k.log.zst | python python/geoipsed.py'
```
* *Note* that a significant factor of this speed difference is the regular expression matching, specifically the IPv6 pattern. If you match just on IPv4, it was only ~5x faster.
## Background & Gratitude
* The historical inspiration for geoipsed was when a sensei taught me long ago that perl's `s/find/replace/g` sed interface can interpret the replace pattern as a perl expression! For example, decoding hexascii, xor'ing, and printing the characters: `perl -pe 's/%([a-f0-9]{2})/chr(hex($1)^0x5e)/ieg`. If I can run perl chr and hex functions, can I import geoip libraries, and run those functions? It turns out, yes! Thus geoipsed was born.
* Writing this was an endeavor to learn more Rust, and I'm grateful for [burntsushi](https://github.com/burntsushi) and [sstadick](https://github.com/sstadick) whose awesome crates and well documented code are sources of inspiration and regular references.