https://github.com/FINRAOS/DataGenerator
DataGenerator is a Java library for systematically producing large volumes of data. DataGenerator frames data production as a modeling problem, with a user providing a model of dependencies among variables and the library traversing the model to produce relevant data sets.
https://github.com/FINRAOS/DataGenerator
Last synced: about 2 months ago
JSON representation
DataGenerator is a Java library for systematically producing large volumes of data. DataGenerator frames data production as a modeling problem, with a user providing a model of dependencies among variables and the library traversing the model to produce relevant data sets.
- Host: GitHub
- URL: https://github.com/FINRAOS/DataGenerator
- Owner: FINRAOS
- License: apache-2.0
- Created: 2013-11-06T16:59:29.000Z (about 12 years ago)
- Default Branch: master
- Last Pushed: 2022-07-07T22:05:59.000Z (over 3 years ago)
- Last Synced: 2025-07-04T06:05:23.723Z (6 months ago)
- Language: Java
- Homepage: http://finraos.github.io/DataGenerator
- Size: 57.7 MB
- Stars: 163
- Watchers: 41
- Forks: 170
- Open Issues: 49
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-java - DataGenerator
README
[](https://travis-ci.org/FINRAOS/DataGenerator) [](https://www.versioneye.com/user/projects/577671b068ee07003cb5d56a) [](https://copilot.blackducksoftware.com/github/repos/FINRAOS/DataGenerator/branches/master) [](https://gitter.im/FINRAOS/DataGenerator?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
Quick Start Videos
------------------
https://www.youtube.com/playlist?list=PLB0Zha5q-7wJp3TLH782J7ZDQ2RwPS_hQ
Contributing
------------
We encourage contribution from the open source community to make DataGenerator better. Please refer to the [development](http://finraos.github.io/DataGenerator/index.html#get_involved) page for more information on how to contribute to this project.
Maven Dependency
-----------------
For the core
```sh
org.finra.datagenerator
dg-core
2.2
```
For the commons library
```sh
org.finra.datagenerator
dg-common
2.2
```
Building
------------
DataGenerator uses Maven for build. Please install Maven by downloading it from [here](http://maven.apache.org/download.cgi).
```sh
# Clone DataGenerator git repo
git clone git://github.com/FINRAOS/DataGenerator.git
cd DataGenerator
# Checkout master branch
git checkout master
# Run package to compile and create jar (also runs unit tests)
mvn package
# Compile and run unit tests only
mvn test
```
License
------------------------------------
The DataGenerator project is licensed under [Apache License Version 2.0](http://www.apache.org/licenses/LICENSE-2.0)
Overview
--------------------
Data Generator generates pattern using two pieces of user provided information:
1. An [SCXML](http://commons.apache.org/proper/commons-scxml/) state chart representing interactions between different states, and setting values to output variables
2. A user [Transformer](http://finraos.github.io/DataGenerator/apis/v2.0/org/finra/datagenerator/consumer/DataTransformer.html) that formats the variables and stores them.
The user can optionally provide their own [distributor](http://finraos.github.io/DataGenerator/apis/v2.0/org/finra/datagenerator/distributor/SearchDistributor.html) that distributes the search of bigger problems on systems like hadoop. By default, DataGenerator will use a multithreaded distributor.
Quick start
--------------------
For the full compilable code please see the [default example](https://github.com/FINRAOS/DataGenerator/blob/master/dg-example-default/)
First step, define an [SCXML](http://commons.apache.org/proper/commons-scxml/) model:
```xml
```
This model contains five variables controlled by two states. The transition between those states is unconditional. One of those variables is always constant ( var_out_V1_3 ). Three will acquire every value from a set ( var_out_V1_1, var_out_V1_2 and var_out_V2 ). var_out_V3 will be set to a holder value that will be replaced by the user at a later point.
The second step will be to write a [Transformer](http://finraos.github.io/DataGenerator/apis/v2.0/org/finra/datagenerator/consumer/DataTransformer.html). The code is [here](https://github.com/FINRAOS/DataGenerator/blob/master/dg-example-default/src/main/java/org/finra/datagenerator/samples/transformer/SampleMachineTransformer.java)
```java
public class SampleMachineTransformer implements DataTransformer {
private static final Logger log = Logger.getLogger(SampleMachineTransformer.class);
private final Random rand = new Random(System.currentTimeMillis());
/**
* The transform method for this DataTransformer
* @param cr a reference to DataPipe from which to read the current map
*/
public void transform(DataPipe cr) {
for (Map.Entry entry : cr.getDataMap().entrySet()) {
String value = entry.getValue();
if (value.equals("#{customplaceholder}")) {
// Generate a random number
int ran = rand.nextInt();
entry.setValue(String.valueOf(ran));
}
}
}
}
```
The above transformer will intercept every generated row, and convert the place holder "#customplaceholder" with a random number.
The last step will be writing a main function that ties both pieces together. Code is [here](https://github.com/FINRAOS/DataGenerator/blob/master/dg-example-default/src/main/java/org/finra/datagenerator/samples/CmdLine.java)
```java
public static void main(String[] args) {
Engine engine = new SCXMLEngine();
//will default to samplemachine, but you could specify a different file if you choose to
InputStream is = CmdLine.class.getResourceAsStream("/" + (args.length == 0 ? "samplemachine" : args[0]) + ".xml");
engine.setModelByInputFileStream(is);
// Usually, this should be more than the number of threads you intend to run
engine.setBootstrapMin(1);
//Prepare the consumer with the proper writer and transformer
DataConsumer consumer = new DataConsumer();
consumer.addDataTransformer(new SampleMachineTransformer());
consumer.addDataWriter(new DefaultWriter(System.out,
new String[]{"var_out_V1_1", "var_out_V1_2", "var_out_V1_3", "var_out_V2", "var_out_V3"}));
//Prepare the distributor
DefaultDistributor defaultDistributor = new DefaultDistributor();
defaultDistributor.setThreadCount(1);
defaultDistributor.setDataConsumer(consumer);
Logger.getLogger("org.apache").setLevel(Level.WARN);
engine.process(defaultDistributor);
}
```
The first few lines will open an input stream on the [SCXML](http://commons.apache.org/proper/commons-scxml/) file and pass the stream to the engine. Calling setBootStrapMin will attempt to split the graph generated from the state chart to at least the given number of splits. Here we passed 1 but in case you will execute the same code over hadoop or use a multithreaded version, you will need to increase that number to be at least the number of threads or mappers you wish to run. The rest of the code will set our transformer to the engine and create a writer based on the DefaultWriter. The function of the writer is to write the output to the user's desired destination.
The final piece sets the number of threads and called engine.process.