https://github.com/bengreenier/node-data-map-reduce
n-tiered map reduce for databases.
https://github.com/bengreenier/node-data-map-reduce
data-processing database example
Last synced: 3 months ago
JSON representation
n-tiered map reduce for databases.
- Host: GitHub
- URL: https://github.com/bengreenier/node-data-map-reduce
- Owner: bengreenier
- License: mit
- Created: 2019-10-27T15:25:05.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2023-01-04T23:43:03.000Z (about 3 years ago)
- Last Synced: 2025-01-23T08:20:04.529Z (about 1 year ago)
- Topics: data-processing, database, example
- Language: TypeScript
- Homepage:
- Size: 289 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# node-data-map-reduce
n-tiered map reduce for databases. `ndmr` for short.
This project provides one possible way to select data from multiple downstream databases and run a query against the result of all those downstream queries on-demand. For instance:
```
SELECT * FROM users @ https://my-downstream-sql-provider.com
SELECT * FROM pets @ https://your-sql-animal-db.com
SELECT pet_type FROM pets INNER JOIN users ON pets.owner_id=users.id WHERE users.name='ben'
```
In the above example, we'd collect all user records from one source, and all pet records from another. We'd then correlate the two sources, using the pet records `owner_id` and the user records `id`. Finally, we'd select the `pet_type` field from the pet record, where the `name` field from the users record matches `ben`.
This type of scenario is likely not very widely applicable, and is probably only interesting to you if:
- You have many downstream data sources that cannot be moved/centralized for business reasons
- These downstream sources have different schemas
- These downstream sources **are relational data stores**
- You wish to query these data sources as one source
## Usage
```
ndmr [command]
Commands:
ndmr execute [query] execute a query
Options:
--help Show help [boolean]
--version Show version number [boolean]
--config Path to the configuration file[required] [default: "./config.json"]
```
### Execute
```
ndmr execute [query]
execute a query
Positionals:
query the path to a query (JSON file) to execute
[string] [required] [default: "./query.json"]
Options:
--help Show help [boolean]
--version Show version number [boolean]
--config Path to the configuration file[required] [default: "./config.json"]
```
## Examples
For example queries and configurations, see the [examples](./examples) directory. Each subdirectory is an example, complete with it's own README.
## Binary Releases
To simplify cases when one might simply want to run the examples without needing to build the source,binary releases are provided for Windows, Mac, and Linux on the [Releases Page](https://github.com/bengreenier/node-data-map-reduce/releases/latest).
## Contributing
Open a PR against this repository, or create an Issue.