Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/txn2/rxtx
Queue based data collector, transmitter with store-and-forward. Useful for online/offline data collection, back pressure buffering or general queuing. REST / HTTP post.
https://github.com/txn2/rxtx
data-collector docker-container docker-image golang iot iot-application iot-data json json-api message-queue mq rest-api rxtx transmitter
Last synced: 3 months ago
JSON representation
Queue based data collector, transmitter with store-and-forward. Useful for online/offline data collection, back pressure buffering or general queuing. REST / HTTP post.
- Host: GitHub
- URL: https://github.com/txn2/rxtx
- Owner: txn2
- License: mit
- Created: 2018-04-11T21:31:51.000Z (over 6 years ago)
- Default Branch: v2.x
- Last Pushed: 2024-04-23T02:04:05.000Z (8 months ago)
- Last Synced: 2024-09-27T02:04:40.353Z (3 months ago)
- Topics: data-collector, docker-container, docker-image, golang, iot, iot-application, iot-data, json, json-api, message-queue, mq, rest-api, rxtx, transmitter
- Language: Go
- Homepage: https://txn2.com
- Size: 3.3 MB
- Stars: 18
- Watchers: 5
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://github.com/txn2/rxtx/releases)
[](https://travis-ci.org/txn2/rxtx)
[](https://goreportcard.com/report/github.com/txn2/rxtx)
[](https://codeclimate.com/github/txn2/rxtx/maintainability)
[](https://godoc.org/github.com/txn2/rxtx/rtq)
[](https://app.fossa.io/projects/git%2Bgithub.com%2Ftxn2%2Frxtx?ref=badge_shield)
[](https://hub.docker.com/r/txn2/irsync/)
[](https://hub.docker.com/r/txn2/irsync/)
[](https://hub.docker.com/r/txn2/rxtx/)
# rxtx
**rxtx** is a queue based data collector > data transmitter. Useful for online/offline data collection, back pressure buffering or general queuing. **rxtx** uses [bbolt](https://github.com/coreos/bbolt) maintained by CoreOs, a single file database for storing messages before they can be sent.
[rtbeat](https://github.com/txn2/rtbeat) was developed to consume **rxtx** POST data and publish as events into [elasticsearch], [logstash], [kafka], [redis] or directly to log files.
## Test on MacOs
### Install with [brew]
```bash
brew tap txn2/homebrew-tap
brew install rxtx
```
### Help
```bash
rxtx -h
```
## Test [Docker] Container
### Help
```bash
docker run --rm -it txn2/rxtx -h
```
on arm 6/7 based device:
```bash
docker run --rm -it txn2/rxtx:arm32v6-1.2.0 -h
```
## Test Source
#### Help
```bash
go run ./rxtx.go -h
Usage of rxtx:
-batch int
Batch size. (default 5000)
-ingest string
Ingest server. (default "http://localhost:8081/in")
-interval int
Seconds between intervals. (default 30)
-maxq int
Max number of message in queue. (default 2000000)
-name string
Service name. (default "rxtx")
-path string
Directory to store database. (default "./")
-port string
Server port. (default "8080")
```
#### Start server on 8080
```bash
go run ./rxtx.go
```
#### Add message to queue
The **rxtx** services accepts HTTP **POST** data to an API endpoint in the following form /rx/**PRODUCER**/**KEY**/**LABEL/...**/. One label is required, however as many labels as necessary may be added, separated by a forward slash.
```bash
curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data
```
#### Add message to queue every second
```bash
while true; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; sleep 1; done
```
#### Add 1000 messages to the queue.
```bash
time for i in {1..1000}; do curl -w "\n" -d "{\"generic\": \"$RANDOM\"}" -X POST http://localhost:8080/rx/me/generic_data/generic/test/data; done
```
#### Example Batch
```json
{
"uuid":"d7642975-4241-4f43-b704-b67621b184b8",
"size":4,
"messages":[
{
"seq":"2019072100000000038",
"time":"2019-07-21T02:44:49.560087Z",
"uuid":"a93cfe42-79c5-4a32-9dc0-6b77b68d0926",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"6955"
}
},
{
"seq":"2019072100000000039",
"time":"2019-07-21T02:44:50.6214579Z",
"uuid":"66042da0-b0c1-447c-ab0b-66c887e8b56a",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"8923"
}
},
{
"seq":"2019072100000000040",
"time":"2019-07-21T02:44:51.622978Z",
"uuid":"0ec3f6d7-905f-42bc-896e-515bf8f2a06d",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"28324"
}
},
{
"seq":"2019072100000000046",
"time":"2019-07-21T02:44:57.8990519Z",
"uuid":"bfa2d06d-a8f4-4125-93bc-d08a070079d7",
"producer":"me",
"label":"/generic/test/data",
"key":"generic_data",
"payload":{
"generic":"31460"
}
}
]
}
```
### Profile
```bash
go build ./rxtx.go && time ./rxtx --path=./data/ --cpuprofile=rxtxcpu.prof --memprofile=rxtxmem.prof
```
Browser-based profile viewer:
```bash
go tool pprof -http=:8081 rxtxcpu.prof
```
### Building and Releasing
**rxtx** uses [GORELEASER] to build binaries and [Docker] containers.
#### Test Release Steps
Install [GORELEASER] with [brew] (MacOS):
```bash
brew install goreleaser/tap/goreleaser
```
Build without releasing:
```bash
goreleaser --skip-publish --rm-dist --skip-validate
```
#### Release Steps
- Commit latest changes
- [Tag] a version `git tag -a v2.0 -m "Version 2.0"`
- Push tag `git push origin v2.0`
- Run: `GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist`
## Resources
- [elasticsearch]
- [logstash]
- [kafka]
- [redis]
- [GORELEASER]
- [Docker]
- [homebrew]
## License
[](https://app.fossa.io/projects/git%2Bgithub.com%2Ftxn2%2Frxtx?ref=badge_large)
[homebrew]: https://brew.sh/
[brew]: https://brew.sh/
[GORELEASER]: https://goreleaser.com/
[Docker]: https://www.docker.com/
[Tag]: https://git-scm.com/book/en/v2/Git-Basics-Tagging
[elasticsearch]: https://www.elastic.co/
[logstash]: https://www.elastic.co/products/logstash
[kafka]: https://kafka.apache.org/
[redis]: https://redis.io/