https://github.com/redborder/synthetic-producer
Generates random JSON messages and sends them to Apache Kafka
https://github.com/redborder/synthetic-producer
Last synced: 2 months ago
JSON representation
Generates random JSON messages and sends them to Apache Kafka
- Host: GitHub
- URL: https://github.com/redborder/synthetic-producer
- Owner: redBorder
- License: agpl-3.0
- Created: 2015-08-26T07:10:38.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2024-10-24T15:09:49.000Z (over 1 year ago)
- Last Synced: 2024-10-25T06:12:56.829Z (over 1 year ago)
- Language: Java
- Homepage:
- Size: 173 KB
- Stars: 7
- Watchers: 10
- Forks: 8
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# synthetic-producer
[](https://www.codacy.com/app/redBorder/synthetic-producer)
Generates random JSON messages and sends them to Apache Kafka.
It uses Apache Zookeeper to discover the available brokers and partitions for a given topic.
Usage:
```bash
$ java -jar target/synthetic-producer-1.4.0-selfcontained.jar
usage: java -jar synthetic-producer.jar
-c,--config config file path
-h,--help show this help
-r,--rate messages rate per second
-t,--threads number of producer threads
-z,--zookeeper zookeeper connect string
```
## Config file
The config file specifies the Zookeeper connection string, the rate of
messages generated, the threads to use, the topic where you want to produce,
and the messages' schema.
For example, take a look at the following config:
```yaml
topic: testTopic
partitionKey: myPartitionKeyField
fields:
myConstant:
type: constant
value: 'Hello, World!'
randomNumber:
type: integer
min: 0
max: 100
```
Using a file like that will make the producer will connect to Zookeeper at "localhost:2181" in order create
two producing threads, and send messages to the topic "testTopic" at a constant rate of 100 msgs per
second, which would look like the following:
```json
{ "myConstant": "Hello, World!", "randomNumber": 4}
{ "myConstant": "Hello, World!", "randomNumber": 12}
{ "myConstant": "Hello, World!", "randomNumber": 94}
{ "myConstant": "Hello, World!", "randomNumber": 33}
```
You can find a full example of config file in [this file](https://github.com/redBorder/synthetic-producer/blob/master/configProducer.yml)
## Fields types
You can use a variety of different field types to generate your random message.
Some types accepts one or more parameters that modify the field randomness.
For example, the constant type always generates the same value for a field, but the integer
type generates a random number between 'min' and 'max' for each field.
The available types, with their parameters, are the following:
### constant
Generates a constant value
Parameters:
- value: The value to be generated
Example:
```yaml
myField:
type: constant
value: "This is a constant"
```
### integer
Generates a random integer
Parameters:
- min: The minimum integer to be generated (Default 0)
- max: The maximum integer to be generated (Default Integer.MAX_VALUE)
- negative: True if the generated number must be negative (It will be multiplied by -1)
Example:
```yaml
myField:
type: integer
min: 0
max: 50
```
### double
Generates a random double
Parameters:
- min: The minimum double to be generated (Default 0.00)
- max: The maximum double to be generated (Default Double.MAX_VALUE)
- negative: True if the generated number must be negative (It will be multiplied by -1)
- truncate: The decimals numbers (Default 10000, four decimals)
Example:
```yaml
myField:
type: double
min: 0.00
max: 50.00
```
### counter
Generates an integer counter between two values
Parameters:
- min: Minimum value that counter can return (Default 0)
- max: Maximum value that counter can return (Default Integer.MAX_VALUE)
- direction: True if the counter goes from minimum to maximum. False if it goes from maximum to minimum. (Default true)
Example:
```yaml
myField:
type: counter
min: 1
max: 10
```
### ip
Generates a random IP address
Parameters:
- min: The minimum ip to be generated (Default 0.0.0.0)
- max: The maximum ip to be generated (Default 255.255.255.255)
Example:
```yaml
myField:
type: ip
min: 192.168.0.1
max: 192.168.0.255
```
### mac
Generates a random MAC address
Parameters:
- min: The minimum ip to be generated (Default 00:00:00:00:00:00)
- max: The maximum ip to be generated (Default FF:FF:FF:FF:FF:FF)
- separator: The string using to split the mac on the output.
Example:
```yaml
myField:
type: mac
min: 00:00:00:00:00:11
max: 00:00:00:00:00:44
```
### timestamp
Generates a field with the current timestamp of the machine
Example:
```yaml
myField:
type: timestamp
```
### collection
Generates a random element from the given collection of elements
Parameters:
- values: An array of elements
Example:
```yaml
myField:
type: collection
values:
- Campus A
- Campus B
- Campus C
- Campus D
- Campus E
```
### composition
Generates a random field based on a composition of two or more random fields
Parameters:
- separator: The character that will be used as the separator between the two generated values
- components: An array of types definitions that will be generated before joining the result with the separator
Examples:
```yaml
myField:
type: composition
separator: ':'
components:
- type: integer
min: 0
max: 11
- type: integer
min: 0
max: 101
```
This will create values like the following: 2:34, 6:11, 1:75, 10:62...
Of course, you can use a composition type as a component for another composition type.
### set
Generate a fixed number of sets with multiple fixed field per set.
Parameters:
- numbers: The different numbers of sets
- components: The components that form a set
```yaml
set123:
type: set
numbers: 10
components:
client_mac:
type: mac
min: 00:00:00:00:00:11
max: 00:00:00:00:00:44
src:
type: ip
min: 80.82.34.12
max: 80.82.34.72
```
This will create 10 sets form by a **client_mac** of type *mac* and **src** of type *ip*. The sets are making on the init and them don't change after.
A message has 1 set that is choose randomly from 10 sets.
### random
Choose one component randomly among all the components inside it.
Parameters:
- components: List of the components that will be picked randomly.
You can find a simple example below:
```yaml
mySimpleRandom:
type: random
components:
-
myField:
type: integer
min: 101
max: 110
-
myField:
type: integer
min: 201
max: 210
-
myField:
type: integer
min: 301
max: 310
```
This will generate values from 101 to 110, from 201 to 210 and from 301 to 310. Also you can **nest** random types in order to get more complex combinations.
### json
Generate a new json inside other json.
Parameters:
- components: The components that form a set
```yaml
engine_id_name:
type: constant
value: IANA-L4
myNewJson:
type: json
components:
src:
type: ip
min: 192.168.0.1
max: 192.168.255.255
application_id:
type: composition
separator: ':'
components:
- type: integer
min: 0
max: 11
- type: integer
min: 0
max: 101
myNewJsonInside:
type: json
components:
dst:
type: ip
min: 192.168.0.1
max: 192.168.255.255
```
This will generate messages like this:
```json
{"myNewJson":{"myNewJsonInside":{"dst":"192.168.231.69"},"src":"192.168.161.220","application_id":"9:32"},"engine_id_name":"IANA-L4"}
```
### coordinates
Generate a random latitude and longitude inside a circle that is defined using radius.
- latPoint: The latitude center point.
- longPoint: The longitude center point.
- radius: The radius of the circle.
- latitudeDim: The field name of the latitude value.
- longitudeDim: The field name of the longitude value.
- truncate: The decimals numbers (Default 10000, four decimals)
Example:
```yaml
coordinates:
type: coordinate
latPoint: 30.000
longPoint: -40.000
radius: 1000
latitudeDim: latitude
longitudeDim: longitude
truncate: 1000
```
### array
Generate array with different components.
- components: The different types that build the array
Example:
```yaml
myArray:
type: array
components:
field1:
type: json
components:
dst:
type: ip
min: 192.168.0.1
max: 192.168.255.255
field2:
type: constant
value: 5
```
This example generate an array like:
```json
{"myArray":[5,{"dst":"192.168.142.39"}]}
{"myArray":[5,{"dst":"192.168.121.126"}]}
{"myArray":[5,{"dst":"192.168.189.54"}]}
```
### sequential
Generate one component each time. The sequential restart when reach the end.
- components: The different types that build the sequence
```yaml
sequentialField:
type: sequential
components:
-
seq1:
type: set
numbers: 1
components:
longitude:
type: constant
value: 1
latitude:
type: constant
value: -1
-
seq2:
type: set
numbers: 1
components:
longitude:
type: constant
value: 2
latitude:
type: constant
value: -2
-
seq3:
type: set
numbers: 1
components:
longitude:
type: constant
value: 3
latitude:
type: constant
value: -3
```
This example generate an array like:
```json
{"longitude":1,"latitude":-1}
{"longitude":2,"latitude":-2}
{"longitude":3,"latitude":-3}
{"longitude":1,"latitude":-1}
{"longitude":2,"latitude":-2}
```
## Contributing
1. [Fork it](https://github.com/redborder/synthetic-producer/fork)
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request
## License
[AGPL v3](http://www.gnu.org/licenses/agpl-3.0.html)