https://github.com/nhl/link-move
A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.
https://github.com/nhl/link-move
cayenne etl etl-framework
Last synced: 2 days ago
JSON representation
A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.
- Host: GitHub
- URL: https://github.com/nhl/link-move
- Owner: nhl
- License: apache-2.0
- Created: 2014-08-12T12:27:10.000Z (over 11 years ago)
- Default Branch: main
- Last Pushed: 2025-04-19T13:28:50.000Z (9 months ago)
- Last Synced: 2025-07-23T22:05:10.435Z (6 months ago)
- Topics: cayenne, etl, etl-framework
- Language: Java
- Homepage:
- Size: 1.78 MB
- Stars: 36
- Watchers: 13
- Forks: 16
- Open Issues: 24
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-java - LinkMove
README
[](https://github.com/nhl/link-move/actions)
[](https://maven-badges.herokuapp.com/maven-central/com.nhl.link.move/link-move/)
## LinkMove
LinkMove is a model-driven dynamically-configurable framework to acquire data from external sources and save it in your
database. Its primary motivation is to facilitate
[domain-driven design](https://en.wikipedia.org/wiki/Domain-driven_design) architectures. In DDD terms LinkMove is a
tool to synchronize data between related models from different ["bounded contexts"](http://martinfowler.com/bliki/BoundedContext.html).
It can also be used as a general purpose ETL framework.
LinkMove connects data models in a flexible way that anticipates independent changes between sources and targets. It
will reuse your existing ORM mapping for the _target_ database, reducing configuration to just describing the _source_.
It supports JDBC, XML, JSON, CSV sources out of the box.
## Support
There are two options:
* Post your question on the [GitHub Discussions](https://github.com/nhl/link-move/discussions)
* If you think you encountered a bug, open a [GitHub issue](https://github.com/nhl/link-move/issues)
## Getting Started
Add LinkMove dependency:
```XML
com.nhl.link.move
link-move
3.0.0-RC3
```
The core module above supports relational and XML sources. The following optional modules may be added if you need to work with other formats:
```XML
com.nhl.link.move
link-move-json
3.0.0-RC3
```
```XML
com.nhl.link.move
link-move-csv
3.0.0-RC3
```
Use it:
```Java
// bootstrap shared runtime that will run tasks
DataSource srcDS = // define how you'd connect to data source
ServerRuntime targetRuntime = // Cayenne setup for data target .. targets are mapped in Cayenne
File rootDir = .. // this is a parent dir of XML descriptors
LmRuntime lm = LmRuntime.builder()
.connector(JdbcConnector.class, "myconnector", new DataSourceConnector(srcDS))
.targetRuntime(targetRuntime)
.extractorModelsRoot(rootDir)
.build();
// create a reusable task for a given transformation
LmTask task = lm.getTaskService()
.createOrUpdate(MyTargetEntity.class)
.sourceExtractor("my-etl.xml")
.matchBy(MyTargetEntity.NAME).task();
// run task, e.g. in a scheduled job
Execution e = task.run();
```
## Extractor XML Format
Extractor XML format is described by a formal schema: https://nhl.github.io/link-move/xsd/extractor_config_3.xsd
An example using JDBC connector for the source data:
```XML
jdbc
myconnector
java.lang.Integer
AGE
db:age
java.lang.String
DESCRIPTION
db:description
java.lang.String
NAME
db:name
SELECT age, description, name FROM etl1
```
## Logging Configuration
LinkMove uses Slf4J abstraction for logging, that will work with most common logging frameworks (Log4J2, Logback, etc.).
With any framework you use, you will need to configure the following log levels depending on the desired verbosity of
your ETL tasks.
### Logging ETL Progress
You need to configure the `com.nhl.link.move.log` logger to log the progress of the ETL tasks. The following table
shows what is logged at each log level:
|Log Level| What is Logged |
|---------|-------------------------------------------------------------------------------|
| WARN | Nothing |
| INFO | Task start/stop with stats |
| DEBUG | Same as INFO, but also includes start/stop of each segment with segment stats |
|TRACE|Same as DEBUG, but also includes IDs of all affected target objects (deleted, created, updated)|
### Logging SQL
ETL-related SQL generated by Cayenne is extremely verbose and barely human-readable. You need to configure the
`org.apache.cayenne.log` logger to turn it on and off:
|Log Level| What is Logged |
|---------|-------------------------------------------------------------------------|
| WARN | Nothing |
| INFO | Cayenne-generated SQL queries and updates|