Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/feliixx/mgodatagen

Generate random data for MongoDB
https://github.com/feliixx/mgodatagen

bson database-population mongodb random scalability seeding test

Last synced: about 2 months ago
JSON representation

Generate random data for MongoDB

Host: GitHub
URL: https://github.com/feliixx/mgodatagen
Owner: feliixx
License: mit
Created: 2017-08-10T11:36:00.000Z (about 7 years ago)
Default Branch: master
Last Pushed: 2024-04-14T17:05:35.000Z (5 months ago)
Last Synced: 2024-05-01T14:41:45.690Z (5 months ago)
Topics: bson, database-population, mongodb, random, scalability, seeding, test
Language: Go
Homepage:
Size: 2.06 MB
Stars: 313
Watchers: 10
Forks: 63
Open Issues: 5
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

favorite-link - 为 MongoDB 生成随机数据。
awesome-mongodb - mgodatagen - Random data generator (Tools / Development)
awesome-mongodb - mgodatagen - Random data generator (Tools / Development)
awesome-ccamel - feliixx/mgodatagen - Generate random data for MongoDB (Go)

README

[![Go Report Card](https://goreportcard.com/badge/github.com/feliixx/mgodatagen)](https://goreportcard.com/report/github.com/feliixx/mgodatagen)
[![codecov](https://codecov.io/gh/feliixx/mgodatagen/branch/master/graph/badge.svg)](https://codecov.io/gh/feliixx/mgodatagen)
[![PkgGoDev](https://pkg.go.dev/badge/github.com/feliixx/mgodatagen/datagen)](https://pkg.go.dev/github.com/feliixx/mgodatagen/datagen)

# mgodatagen

A small CLI tool to quickly generate millions of pseudo-random BSON documents and insert them into a MongoDB instance. Quickly test new data structure or how your application responds when your database grows!

Try it online: [**mongoplayground.net**](https://mongoplayground.net/)

## Features

- Support all bson types listed in [MongoDB bson types](https://docs.mongodb.com/manual/reference/bson-types/)
- Generate _real_ data using [gofakeit](https://github.com/brianvoe/gofakeit)
- Create referenced fields across collections
- Aggregate data across collections
- Create sharded collection
- Create collections in multiple databases
- Output result as JSON

![Demo](demo.gif)

## installation

Download the binary from the [release page](https://github.com/feliixx/mgodatagen/releases)

Build from source:

First, make sure that go is installed on your machine (see [install go](https://golang.org/doc/install) for details ). Then clone the repo and build it:

```
git clone https://github.com/feliixx/mgodatagen.git
cd mgodatagen
go build
```

If you are using go 1.17 or higher, you can simply use `go install`:

```
go install github.com/feliixx/mgodatagen@latest
```

## Options

Several options are available (use `mgodatagen --help` to print this):

```
Usage:
mgodatagen -f config_file.json

template:
--new= create an empty configuration file

configuration:
-f, --file= JSON config file. This field is required

-a, --append if present, append documents to the collection without
removing older documents or deleting the collection
-i, --indexonly if present, mgodatagen will just try to rebuild index
-x, --indexfirst if present, mgodatagen will create index before
inserting documents
-n, --numWorker= number of concurrent workers inserting documents
in database. Default is number of CPU
-b, --batchsize= bulk insert batch size (default: 1000)
-s, --seed= specific seed to use. Passing the same seed garentees
the same output for evey run with the same config.
Has to be in [1, 18446744073709551615]
-o, --output= where documents should be written. Options are:
- mongodb (default)
- stdout
- filename
--prettyprint if present, indent the output. Only for stdout or file
output

connection infos:
--uri= connection string URI. If present, takes precedence over all
other options. For detail on URI format, see
https://docs.mongodb.com/manual/reference/connection-string/
-h, --host= mongodb host to connect to (default: 127.0.0.1)
--port= server port (default: 27017)
-u, --username= username for authentification
-p, --password= password for authentification
--authenticationMechanism= authentication mechanism
for now only PLAIN and MONGODB-X509 are supported
--tlsCertificateKeyFile= PEM certificate/key file for TLS
--tlsCAFile= Certificate Authority file for TLS

general:
--help show this help message
-v, --version print the tool version and exit
-q, --quiet quieter output
```

Only the configuration file is required ( **-f | --file flag**). A basic usage of mgodatagen would be

```
./mgodatagen -f config.json
```

If no host/port is specified, mgodatagen tries to connect to **`mongodb://127.0.0.1:27017`**.

# Configuration file

The config file is an array of JSON documents, where each documents holds the configuration
for a collection to create

See **MongoDB documentation** for details on parameters:

- shardConfig: [**shardCollection**](https://docs.mongodb.com/manual/reference/command/shardCollection/)
- indexes: [**indexes**](https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/)
- collation: [**collation**](https://docs.mongodb.com/manual/reference/bson-type-comparison-order/#collation)

```scala
[
// first collection to create
{
// REQUIRED FIELDS
//
"database": , // required, database name
"collection": , // required, collection name
"count": , // required, number of document to insert in the collection
"content": { // required, the actual schema to generate documents
"fieldName1": , // optional, see Generator below
"fieldName2": ,
...
},
// OPTIONAL FIELDS
//
// compression level (for WiredTiger engine only)
// possible values:
// - none
// - snappy
// - zlib
"compressionLevel": , // optional, default: snappy

// configuration for sharded collection
"shardConfig": { // optional
"shardCollection": ., // required. .
"key": , // required, shard key, eg: {"_id": "hashed"}
"unique": , // optional, default: false
"numInitialChunks": , // optional

"collation": { // optional
"locale": , // required
"caseLevel": , // optional
"caseFirst": , // optional
"strength": , // optional
"numericOrdering": , // optional
"alternate": , // optional
"maxVariable": , // optional
"backwards": , // optional
"normalization": // optional
}
},

// list of index to build
"indexes": [ // optional
{
"name": , // required, index name
"key": , // required, index key, eg: {"name": 1}
"sparse": , // optional, default: false
"unique": , // optional, default: false
"bits": , // optional, for 2d indexes only, default: 26
"min": , // optional, for 2d indexes only, default: -180.0
"max": , // optional, for 2d index only, default: 180.0
"bucketSize": , // optional, for geoHaystack indexes only
"expireAfterSeconds": , // optional, for TTL indexes only
"weights": , // optional, for text indexes only
"defaultLanguage": , // optional, for text index only
"languageOverride": , // optional, for text index only
"textIndexVersion": , // optional, for text index only
"partialFilterExpression": , // optional
"hidden": , // optional
"storageEngine": , // optional
"wildcardProjection": , // optional, for wildcard index only
"2dsphereIndexVersion": , // optional, for 2dsphere index only

### Example

A set of sample config files can be found in **[datagen/generators/testdata/](https://github.com/feliixx/mgodatagen/tree/master/datagen/generators/testdata)**. To use it,
make sure that you have a mongodb instance running (on 127.0.0.1:27017 for example)
and run

```
./mgodatagen -f datagen/generators/testdata/ref.json
```

This will insert 1000 random documents in collections `test` and `link` of database
`mgodatagen_test` with the structure defined in the config file.

# Generator types

Generators have a common structure:

```scala
"fieldName": { // required, field name in generated document
"type": , // required, type of the field
"typeParam": ..., // specific parameters for this type

"maxDistinctValue": , // optional, maximum number of distinct values for this field
"nullPercentage": // optional, int between 0 and 100. Percentage of documents
// that will have this field
}
```

List of `` types:

- [string](#string)
- [stringFromParts](#stringFromParts)
- [int](#int)
- [long](#long)
- [double](#double)
- [decimal](#decimal)
- [autoincrement](#autoincrement)
- [boolean](#boolean)
- [objectId](#objectid)
- [UUID](#uuid)
- [binary](#binary)
- [date](#date)
- [coordinates (formerly position)](#coordinates)
- [constant](#constant)
- [enum (formerly fromArray)](#enum)
- [reference](#reference)
- [faker](#faker)
- [array](#array)
- [object](#object)
- [countAggregator](#countAggregator)
- [valueAggregator](#valueAggregator)
- [boundAggregator](#boundAggregator)

### String

Generates a random string of a certain length. String is composed of char within this list:
`abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_`

```scala
"fieldName": {
"type": "string", // required
"minLength": , // optional, must be >= 0
"maxLength": , // optional, must be >= minLength
"unique": , // optional, see details below
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

#### Unique String

If `unique` is set to true, the field will only contains unique strings. Unique strings
have a **fixed length**, `minLength` is taken as length for the string.
There is `64^x` possible unique string for strings of length `x`. This number has to
be inferior or equal to the number of documents you want to generate.
For example, if you want unique strings of length 3, there is `64 * 64 * 64 = 262144` possible
strings

They will look like

```
"aaa",
"aab",
"aac",
"aad",
...
```

### StringFromParts

Generates a random string from several generators. `parts` generators can't have `unique` or `maxDistinctValue` attributes set.

```scala
"fieldName": {
"type": "stringFromParts", // required
"parts": [ // required. Can't be empty. An array
, // of generators of any basic type

...
],
"nullPercentage": // optional
}
```

**Example:**

To generate phone number like `'(555) 565-2431'`, you can combine several generators
like this:

```scala
"phone": {
"type": "stringFromParts",
"parts": [
{
"type": "constant",
"constVal": "(555) "
},
{
"type": "int",
"min": 100,
"max": 999
},
{
"type": "constant",
"constVal": "-"
},
{
"type": "int",
"min": 1000,
"max": 9999
},
]
}
```

### Int

Generates a random `int` within bounds.

```scala
"fieldName": {
"type": "int", // required
"min": , // optional
"max": , // optional, must be >= min
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Long

Generates a random `long` within bounds.

```scala
"fieldName": {
"type": "long", // required
"min": , // optional
"max": , // optional, must be >= min
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Double

Generates a random `double` within bounds.

```scala
"fieldName": {
"type": "double", // required
"min": , // optional
"max": , // optional, must be >= min
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Decimal

Generates a random `decimal128`.

```scala
"fieldName": {
"type": "decimal", // required
"nullPercentage": , // optional
"maxDistinctValue": , // optional
}
```

### Autoincrement

Generates an autoincremented value (type `` or ``).

```scala
"fieldName": {
"type": "autoincrement", // required
"autoType": , // required, either "int" or "long"
"start": , // optional, start value
"nullPercentage": // optional
}
```

### Boolean

Generates a random `boolean`.

```scala
"fieldName": {
"type": "boolean", // required
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### ObjectId

Generates a random `objectId`.

```scala
"fieldName": {
"type": "objectId", // required
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### UUID

Generates a random UUID

```scala
"fieldName": {
"type": "uuid", // required
"format": // optional, either "string" or "binary".
// default is "string"
"nullPercentage": // optional
}
```

If `format` is `"string"`, the field will be a simple string like `"f1b9b567-9b34-45af-9d9c-35f565d57716"`.

If `format` is `"binary"`, the field will be stored as a [bson UUID](https://docs.mongodb.com/manual/reference/method/UUID/) like `UUID("f1b9b567-9b34-45af-9d9c-35f565d57716")`

### Binary

Generates random binary data of length within bounds.

```scala
"fieldName": {
"type": "binary", // required
"minLength": , // optional, must be >= 0
"maxLength": , // optional, must be >= minLength
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Date

Generates a random date (stored as [`ISODate`](https://docs.mongodb.com/manual/reference/method/Date/) ).

`startDate` and `endDate` are string representation of a Date following RFC3339:

**format**: "yyyy-MM-ddThh:mm:ss+00:00"

```scala
"fieldName": {
"type": "date", // required
"startDate": , // required
"endDate": , // required, must be >= startDate
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Coordinates

Generates random [GeoJSON](https://docs.mongodb.com/manual/geospatial-queries/#std-label-geospatial-geojson) coordinates (a GPS position in WGS84 Decimal Degrees with folowing format: `[ longitude, latitude ]` )

eg : [40.741895, -73.989308]

```scala
"fieldName": {
"type": "coordinates", // required
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Constant

Adds the same value to each document.

```scala
"fieldName": {
"type": "constant", // required
"constVal": , // required, can be of any type including object and array
// eg: {"k": 1, "v": "val"}
// to set constant ObjectId eg: { "$oid": "5a934e000102030405000001" }
"nullPercentage": // optional
}
```

### Enum

Picks an object from an array as value for the field. Currently, objects in the
array have to be of the same type. By default, items are picked from the array
in the order where they appear.

```scala
"fieldName": {
"type": "enum", // required
"values": [ // required. Can't be empty. An array of object of
, // any type, including object and array.

...
],
"randomOrder": , // optional. If set to true, objects will be picked
// from the array in random order.
"nullPercentage": // optional

}
```

### Reference

Use the same list of values for fields in different collection.

```scala
"fieldName":{
"type": "reference", // required
"id": , // required, generator id used to link
// field between collections
"refContent": , // required, generator to use to create the
// list of values
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

generator in other collections:

```scala
"fieldName": {
"type": "reference", // required
"id": , // required, same id as previous generator
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

It can also be used to duplicate a field in a single collection ( see [reference_same_collection.json](https://github.com/feliixx/mgodatagen/tree/master/datagen/testdata/reference_same_collection.json) )

### Array

Generates a random array of bson object.

```scala
"fieldName": {
"type": "array", // required
"arrayContent": , // required, generator use to create element
// to fill the array. Can be of any type
"minLength": , // optional, must be >= 0
"maxLength": , // optional, must be >= minLength
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### Object

Generates random nested object.

```scala
"fieldName": {
"type": "object", // required
"objectContent": { // required, list of generator used to
"nestedFieldName1": , // generate the nested document
"nestedFieldName2": ,
...
},
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

### CountAggregator

Count documents from `.` matching a specific query. To use a
variable of the document in the query, prefix it with "$$".

The query can't be empty or null.

```scala
"fieldName": {
"type": "countAggregator", // required
"database": , // required, db to use to perform aggregation
"collection": , // required, collection to use to perform aggregation
"query": // required, query that selects which documents to count in
// the collection
}
```

**Example:**

Assuming that the collection `first` contains:

```scala
{"_id": 1, "field1": 1, "field2": "a" }
{"_id": 2, "field1": 1, "field2": "b" }
{"_id": 3, "field1": 2, "field2": "c" }
```

and that the generator for collection `second` is:

```scala
{
"database": "test",
"collection": "second",
"count": 2,
"content": {
"_id": {
"type": "autoincrement",
"autoType": "int"
"start": 0
},
"count": {
"type": "countAggregator",
"database": "test",
"collection": "first",
"query": {
"field1": "$$_id"
}
}
}
}
```

The collection `second` will contain:

```scala
{"_id": 1, "count": 2}
{"_id": 2, "count": 1}
```

### ValueAggregator

Get distinct values for a specific field for documents from
`.` matching a specific query. To use a variable of
the document in the query, prefix it with "$$".

The query can't be empty or null.

```scala
"fieldName": {
"type": "valueAggregator", // required
"database": , // required, db to use to perform aggregation
"collection": , // required, collection to use to perform aggregation
"key": , // required, the field for which to return distinct values.
"query": // required, query that specifies the documents from which
// to retrieve the distinct values
}
```

**Example**:

Assuming that the collection `first` contains:

```scala
{"_id": 1, "field1": 1, "field2": "a" }
{"_id": 2, "field1": 1, "field2": "b" }
{"_id": 3, "field1": 2, "field2": "c" }
```

and that the generator for collection `second` is:

```scala
{
"database": "test",
"collection": "second",
"count": 2,
"content": {
"_id": {
"type": "autoincrement",
"autoType": "int"
"start": 0
},
"count": {
"type": "valueAggregator",
"database": "test",
"collection": "first",
"key": "field2",
"values": {
"field1": "$$_id"
}
}
}
}
```

The collection `second` will contain:

```scala
{"_id": 1, "values": ["a", "b"]}
{"_id": 2, "values": ["c"]}
```

### BoundAggregator

Get the lowest and highest value for a specific field of documents in
`.` matching a specific query. To use a variable of
the document in the query, prefix it with "$$"

The query can't be empty or null

```scala
"fieldName": {
"type": "boundAggregator", // required
"database": , // required, db to use to perform aggregation
"collection": , // required, collection to use to perform aggregation
"key": , // required, the field for which to return distinct values.
"query": // required, query that specifies the documents from which
// to retrieve lower/higher value
}
```

**Example**:

Assuming that the collection `first` contains:

```scala
{"_id": 1, "field1": 1, "field2": "0" }
{"_id": 2, "field1": 1, "field2": "10" }
{"_id": 3, "field1": 2, "field2": "20" }
{"_id": 4, "field1": 2, "field2": "30" }
{"_id": 5, "field1": 2, "field2": "15" }
{"_id": 6, "field1": 2, "field2": "200" }
```

and that the generator for collection `second` is:

The collection `second` will contain:

```scala
{"_id": 1, "values": {"m": 0, "M": 10}}
{"_id": 2, "values": {"m": 15, "M": 200}}
```

where `m` is the min value, and `M` the max value.

### Faker

Generate 'real' data using [gofakeit library](https://github.com/brianvoe/gofakeit).

```scala
"fieldName": {
"type": "faker", // required
"method": , // required, faker method to use, for example: Name
"nullPercentage": , // optional
"maxDistinctValue": // optional
}
```

List of faker methods:

```
"FirstName"
"LastName"
"Name"
"NamePrefix"
"NameSuffix"
"Gender"
"Phone"
"PhoneFormatted"
"Username"
"Email"

"BS"
"BuzzWord"
"Company"
"CompanySuffix"
"JobDescriptor"
"JobLevel"
"JobTitle"
"Language"
"LanguageAbbreviation"

"CreditCardCvv"
"CreditCardExp"
"CreditCardType"
"CurrencyLong"
"CurrencyShort"

"DomainName"
"DomainSuffix"
"HTTPMethod"
"IPv4Address"
"IPv6Address"
"MacAddress"
"FileMimeType"
"SSN"
"URL"
"UserAgent"
"SafariUserAgent"
"OperaUserAgent"
"ChromeUserAgent"
"FileExtension"
"FirefoxUserAgent"

"TimeZone"
"TimeZoneAbv"
"TimeZoneFull"
"Month"
"WeekDay"

"Word"
"Question"
"Quote"
"Letter"
"ProgrammingLanguage"
"ProgrammingLanguageBest"
"HexColor"
"Color"
"HipsterWord"
"SafeColor"

"Street"
"StreetName"
"StreetNumber"
"StreetPrefix"
"StreetSuffix"
"City"
"State"
"StateAbr"
"Zip"
"Country"
"CountryAbr"

"Emoji"
"EmojiAlias"
"EmojiCategory"
"EmojiDescription"
"EmojiTag"

"HackerAbbreviation"
"HackerAdjective"
"HackeringVerb"
"HackerNoun"
"HackerPhrase"
"HackerVerb"

"CarMaker"
"CarModel"
"CarTransmissionType"
"CarFuelType"
"CarType"

"Animal"
"AnimalType"
"Cat"
"Dog"
"FarmAnimal"
"PetName"

"BeerAlcohol"
"BeerBlg"
"BeerHop"
"BeerIbu"
"BeerMalt"
"BeerName"
"BeerStyle"
"BeerYeast"
```

If you're building large datasets (10M+ documents) you should avoid faker generators
and use main or custom generators instead, as faker may be way slower.