https://github.com/redis-field-engineering/redis-connect-crud-loader
A generic CRUD loader for redis-connect
https://github.com/redis-field-engineering/redis-connect-crud-loader
crud-generator crud-loader loader redis-connect redisconnect
Last synced: 2 months ago
JSON representation
A generic CRUD loader for redis-connect
- Host: GitHub
- URL: https://github.com/redis-field-engineering/redis-connect-crud-loader
- Owner: redis-field-engineering
- Created: 2021-04-01T00:17:54.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-04-18T20:27:10.000Z (about 1 year ago)
- Last Synced: 2025-01-25T12:09:05.103Z (4 months ago)
- Topics: crud-generator, crud-loader, loader, redis-connect, redisconnect
- Language: TSQL
- Homepage:
- Size: 4.91 MB
- Stars: 1
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# redis-connect-crud-loader
## Prerequisites
• Redis Connect is set up and running with the same source database, schema and table(s) as redis-connect-crud-loader.
• Please have Java Runtime Environment ([OpenJRE](https://openjdk.java.net/install/) or OracleJRE) installed prior to running redis-connect-crud-loader.
## Download
Download the [latest release](https://github.com/redis-field-engineering/redis-connect-crud-loader/releases/download/v1.0/redis-connect-crud-loader-1.0-SNAPSHOT.tar.gz) and un-tar (tar -xvf redis-connect-crud-loader-1.0-SNAPSHOT.tar.gz) the redis-connect-crud-loader-1.0-SNAPSHOT.tar.gz archive.
All the contents would be extracted under redis-connect-crud-loader directory
Contents of redis-connect-crud-loader
• bin – contains script files
• lib – contains java libraries
• config – contains sample config and data files for crud loader
## Configuration
redis-connect-crud-loader includes drivers for the most common RDBMSs:
• Oracle
jdbc:oracle:thin:@myhost:1521:orcl
• IBM DB2
jdbc:db2://host:port/database
• MS SQL Server
jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]
• MySQL
jdbc:mysql://[host]:[port][/database][?properties]
• PostgreSQL
jdbc:postgresql://host:port/database
Sample config.yml
#### Sample config.yml under redis-connect-crud-loader/config folder
```yml
connections:
source:
username: sa #DB user
password: Redis@123 #DB password
type: mssqlserver #this value can not be changed for mssqlserver
jdbcUrl: "jdbc:sqlserver://127.0.0.1:1433;database=RedisLabsCDC"
maximumPoolSize: 10 #This property controls the maximum size that the pool is allowed to reach, including both idle and in-use connections.
minimumIdle: 2 #This property controls the maximum amount of time that a connection is allowed to sit idle in the pool. This setting only applies when minimumIdle is defined to be less than maximumPoolSize.
tableName: dbo.emp #table name in schema.table format
batchSize: 100 #batch size
truncateBeforeLoad: true # default is true and the table will be truncated before initial load. Applies to both crudloader and loadsql options.
loadQueryFile: insert.sql #insert query for loadsql option
csvFile: emp.csv #csv data with header to load (crudloader option)
select: select.sql #select query for the continuous crud (crudloader option). To skip select, comment it out i.e. "select: #select.sql"
updatedSelect: updatedSelect.sql #updated select query for the continuous crud. To skip updatedSelect, comment it out i.e. "select: #updatedSelect.sql"
update: update.sql #update query for the continuous crud (crudloader option). To skip update, comment it out i.e. "select: #update.sql"
delete: delete.sql #delete query for the continuous crud (crudloader option). To skip delete, comment it out i.e. "select: #delete.sql"
#loadQuery: "select * from dbo.emp" #This can be used instead of loadQueryFile property
iteration: 100 #number of iterations to run (crudloader option)
```
## Generate Data
Sample [emp table schema](https://www.mockaroo.com/f1faabd0) as provided in the package.
Generate data using cURL with the following command:
`curl "https://api.mockaroo.com/api/f1faabd0?count=1000&key=a4b4ee70" > "emp.csv"`
## Launch
[*nix OS](https://en.wikipedia.org/wiki/Unix-like):
`redis-connect-crud-loader/bin$ ./start.sh`
Windows OS:
`redis-connect-crud-loader\bin> start.bat`
```bash
Usage: redis-connect-crud-loader [OPTIONS] [COMMAND]
CRUD loader for redis-connect with random Insert, Update and Delete events.
-h, --help Show this help message and exit.
Commands:
crudloader Load CSV data to source database and execute random Insert, Update and Delete events.
genloader Generate and load fake data into the source table
loadsql Load data into source table using sql insert statements.
```