https://github.com/farovictor/mongodbloader
This project is intended to be used as a data loader to support ELT pipelines or any kind of process that requires a heavy data load into a MongoDb database.
https://github.com/farovictor/mongodbloader
data go mongodb pipeline
Last synced: about 1 month ago
JSON representation
This project is intended to be used as a data loader to support ELT pipelines or any kind of process that requires a heavy data load into a MongoDb database.
- Host: GitHub
- URL: https://github.com/farovictor/mongodbloader
- Owner: farovictor
- License: mit
- Created: 2023-10-02T19:30:07.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-06T00:37:55.000Z (over 2 years ago)
- Last Synced: 2025-09-12T08:32:42.561Z (10 months ago)
- Topics: data, go, mongodb, pipeline
- Language: Go
- Homepage:
- Size: 14.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Usage
This tool can be used as a package or cli tool.
It serves as a data loader to support ELT pipelines or any kind of process that requires a heavy data load.
## CLI
This package allows the user to load data from one or multiple json files into a mongodb database.
### Ping database
The `ping` command does a ping in database and returns a connection check.
```bash
mongoloader ping --conn-uri "$MONGO_CONN_URI"
```
### Check if a collection exists
The `collxst` command does a ping in database and returns a connection check.
```bash
mongoloader collxst \
--conn-uri "$MONGO_CONN_URI" \
--db-name "$MONGO_DBNAME" \
--collection "$MONGO_COLLECTION" \
--app-name "$APPNAME"
```
### Loads in batches - Read and streams file content into a pool of workers (async)
The `load-batch` command iterates over a directory and read/marshal files into documents and insert them into a mongodb database.
```bash
mongoloader load-batch \
--conn-uri "$MONGO_CONN_URI" \
--db-name "$MONGO_DBNAME" \
--collection "$MONGO_COLLECTION" \
--app-name "$APPNAME" \
--search-path "./data" \
--file-prefix "some_prefix" \
--num-concurrent-files 10
```
### Extract data
The `load` command load a file or set of files into an slice of documents and inserts it into mongodb database.
```bash
mongoloader load \
--conn-uri "$MONGO_CONN_URI" \
--db-name "$MONGO_DBNAME" \
--collection "$MONGO_COLLECTION" \
--app-name "$APPNAME" \
```
## Releases
All releases are documented in [CHANGELOG](CHANGELOG.md), please check there for more details.
### Docker
All releases are containerized and available [here](https://hub.docker.com/r/victorfaro/mongoloader)
To pull the latest version, execute the following command:
```bash
docker pull victorfaro/mongoloader:latest
```
All images uses the cli as entrypoint, check the above examples to see how to use it.