https://github.com/martensson/f5elastic
A fast syslog server that receives high volumes of traffic logs from F5 Load Balancers and indexes them in Elasticsearch
https://github.com/martensson/f5elastic
elasticsearch f5
Last synced: over 1 year ago
JSON representation
A fast syslog server that receives high volumes of traffic logs from F5 Load Balancers and indexes them in Elasticsearch
- Host: GitHub
- URL: https://github.com/martensson/f5elastic
- Owner: martensson
- License: mit
- Created: 2015-10-16T10:51:14.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2021-02-23T16:36:13.000Z (over 5 years ago)
- Last Synced: 2025-04-11T23:52:54.128Z (over 1 year ago)
- Topics: elasticsearch, f5
- Language: Go
- Size: 37.1 KB
- Stars: 20
- Watchers: 4
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# f5elastic
A fast system tool for receiving high volumes of traffic logs sent from F5 LTM load balancers and indexes these requests inside ES.
It is today battle tested in a production environment handling loads of 500k req/min on a single host.
Features
* Works with Elasticsearch 7.x
* Support for clusters of ES nodes, for distribution of indexing load.
* Health checks for each ES node is enabled by default.
* Uses bulk indexing for better performance with large amount of requests.
* GeoIP lookup of each request, adds long/lat, city, and country for each client.
* Allows to obfuscate client ip if needed.
* ...
## Getting started
1. Create a Request Logging profile on your BigIP, and add the following Response Settings template (set HSL Protocol to TCP):
```
<34> $CLIENT_IP || $HTTP_METHOD || ${Host} || $HTTP_URI || $HTTP_STATCODE || $RESPONSE_SIZE || ${Referer} || ${User-Agent} || $SERVER_IP:$SERVER_PORT || $VIRTUAL_POOL_NAME || $VIRTUAL_NAME
```
2. Download compiled binary from `https://github.com/martensson/f5elastic/releases` or `go get github.com/martensson/f5elastic`
3. Download the latest Maxmind GeoLite2-City or GeoIP2-City db.
4. Edit `f5elastic.toml` (check example in repo)
5. Apply the ES template inside the repo. (optional but recommended). Create a corresponding ILM policy named `f5elastic` with the retention rules of your liking.
6. Run `f5elastic -f /path/to/f5elastic.toml`
7. Take a cup of coffee and make some nice dashboards inside Kibana :)