Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/adobe/blackhole
An HTTP sink (for testing) with optional recording and playback ability
https://github.com/adobe/blackhole
Last synced: 3 months ago
JSON representation
An HTTP sink (for testing) with optional recording and playback ability
- Host: GitHub
- URL: https://github.com/adobe/blackhole
- Owner: adobe
- License: apache-2.0
- Created: 2020-03-24T15:51:02.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-04-08T12:49:35.000Z (7 months ago)
- Last Synced: 2024-06-19T05:36:58.899Z (5 months ago)
- Language: Go
- Size: 183 KB
- Stars: 67
- Watchers: 5
- Forks: 7
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- awesome-repositories - adobe/blackhole - An HTTP sink (for testing) with optional recording and playback ability (Go)
README
# blackhole
### Introduction
* Backhole is an HTTP sink, to be used for testing & protoyping. Good for testing your outgoing http senders or collect sample request data for your real webservice.
- Pretends to be your real HTTP API/server
- Consumes everything thrown at it (any path, any "host")
- Reponds with a 200 and empty body
- No pre-canned reponse yet
- You can also collect & sample incoming requests to your real webservice by pairing this tool with something like [tcpcopy](https://github.com/session-replay-tools/tcpcopy)* Real server responds to your client
* tcpcopy ignores fake response from blackhole
* blackhole records requests.
* Record & Replay
- `blackhole` lets you record and `replay` will replay, aka send the traffic to yet another site.
See next section for instructions on `replay`
### Usage
`$ blackhole`Starts a service to receive, and acknowledge with 200-OK, all http requests to `:80` on any interface.
Default port, tsl (https) settings can be configured via a yaml config file.
It should be named `bhconfig.yaml`. Place in current working directory or under `$HOME/.blackhole````
$ cat bhconfig_sample.yaml
serve:
- "http://:80"
tls:
cert: /path/to/certs/www.foobar.com.pem
privkey: /path/to/certs/www.foobar.com.pem
```Data, payload of your request, is still ignored and dropped on the floor
When data is not saved, blackhole is nothing but a tiny wrapper around the excellent http library `fasthttp`
`$ blackhole -o /path/to/save/files/ -c`
Requests will be saved in a compressed format.
Many files will be created depending on the nunber of threads
This *recording* and subsequent *replay* is the main
additional value provided on top of fasthttp# replay
`$ replay -H host.domain.com:8080 -q /tmp/requests/requests_*.lz4`
Send all data to a different destination.
NOTE: without `-q`, all communication back and forth is printed to stdout.
This will be very verbose.blackhole - benchmarks
======Please note that **all benchmarks assume connection reuse**.
Without connection reuse performance will be horrible in any server solution.I use wrk to generate traffic as shown below
`wrk -t12 -c200 -s post-random.lua -d1m http://target.domain.com:8080/index.html`
* Test 1: **Server & Client**: Both running on a Macbook Pro
* 110,000+ request/sec accepted, read, and then discarded. Run with `--skip-stats`. Request *counting* has a slight overhead, but that is the only thing on top of vanilla fast-http hello-world at this point.
* 100,000 request/sec saved to disk (each with a 2k payload). Roughly 13 GB on disk for 6 million requests sent during a 1 minute test. Almost no overhead for disk i/o. `wrk` and other things running on the Mac is taking up some of the CPU, leaving only 4 cores for the app in either case.
* `post-random.lua` makes payload random to trigger the pathological case for compression. For truly random input, you will not get much compression. A previous version of this script incorrectly sent same data for all requests.
* 95,000 req/sec with 4:1 compression ratio (on compressible repeated content) with LZ4 compression is enabled. Ratio depends on payload. [LZ4](https://github.com/lz4/lz4) is truly awesome and gives us excellent compression without slowing us down.* Test 2: **Server**: One L8s vm on Azure-US-West, **Client:** One DS2 v3 in Azure-US-East as client
* 6,000 to 7,000 request/sec average with
* 400 concurrent connections
* A random payload of 2,000 characters.
* Server uses only 10% cpu resources. We could accomodate more clients.
* Test 3: **Server**: One L8s vm on Azure-US-West and **Client:** One L4s in Aure-US-West (same subnet)
* 138,000 to 140,000 requests/sec average with same payload example as above.
* Server uses 60% cpu resources.
* Test 4: **Server & Client**: One L8s vm on Azure-US-West (same host)
* 260,000 requests/sec average with non-randomized 2k payload. Roughly similar number if payload is not saved.
* 140,000 requests/sec with randomized payload (more to write to disk when compressed)
* Server uses 95% cpu resources.
### Go specific benchmarks
Second column is iteration
```
BenchmarkBHSave-8 521817 19763 ns/op 2752 B/op 36 allocs/op
BenchmarkBHNoSave-8 683246 17491 ns/op 2701 B/op 34 allocs/op
```INSTALLING
======* Steps
* Download and Install Go from https://golang.org/dl/
* Option 1 : run `go install ./...` from the cloned directory.
Please note the three ellipses `...`, not two, a directory reference.
By default, binaries will be installed in `${HOME}/go/bin`, unless you have
either `${GOPATH}` or `${GOBIN}` set in which case binaries
go to those directories.
* Option 2: run `make`. `make` will build to the current directory / project root.
If you don't usually develop applications in Go, this is probably the easiest method.
* To build for another machine (different architecture/os)
* Run `./build-all-archs.sh` (builds for Windows, Linux, as well as MacOS)Design
=======
![Design](design.png)Code Documentation (For contributors)
=======
This is a binary package, not a library, but some of the components are still written as reusable libraries and the
documentation is available here. Internal APIs may not stable and would change. Please import with caution.[github.com/adobe/blackhole/lib/archive](https://pkg.go.dev/github.com/adobe/blackhole/lib/archive)
[github.com/adobe/blackhole/lib/request](https://pkg.go.dev/github.com/adobe/blackhole/lib/request)
[github.com/adobe/blackhole/lib/fbr](https://pkg.go.dev/github.com/adobe/blackhole/lib/fbr)
[github.com/adobe/blackhole/lib/sender](https://pkg.go.dev/github.com/adobe/blackhole/lib/sender)
[github.com/adobe/blackhole/lib/slicehacks](https://pkg.go.dev/github.com/adobe/blackhole/lib/slicehacks)