Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/quix-labs/thunder
Fast, efficient, and optimized synchronization between SQL databases and indexers.
https://github.com/quix-labs/thunder
elasticsearch event-driven go postgresql realtime synchronization
Last synced: 5 days ago
JSON representation
Fast, efficient, and optimized synchronization between SQL databases and indexers.
- Host: GitHub
- URL: https://github.com/quix-labs/thunder
- Owner: quix-labs
- License: mit
- Created: 2024-09-07T15:55:54.000Z (26 days ago)
- Default Branch: main
- Last Pushed: 2024-09-21T15:25:49.000Z (12 days ago)
- Last Synced: 2024-09-24T06:02:11.594Z (9 days ago)
- Topics: elasticsearch, event-driven, go, postgresql, realtime, synchronization
- Language: Go
- Homepage:
- Size: 7.7 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE.md
Awesome Lists containing this project
README
# Thunder
**Thunder** is a lightweight tool designed to effortlessly synchronize your structured SQL databases (Postgres, MySQL,
etc.) with search engine indexes (ElasticSearch, OpenSearch, etc.) without impacting your database's performance.
It’s built for real-time updates and optimized to ensure your data is always indexed and searchable.
## Notes
**This library is currently under active development.**
Features and APIs may change.
Contributions and feedback are welcome!
## Features
- **Non-Intrusive Synchronization**: Sync your SQL data without any performance hit to your database.
- **Real-Time Indexing**: Automatically update indexes in near real-time whenever your data changes.
- **Scalable**: Designed to handle databases of all sizes, ensuring quick and efficient syncs.
## Table of Contents
* [Notes](#notes)
* [Features](#features)
* [Installation](#installation)
* [Using go (build from source)](#using-go-build-from-source)
* [How to configure](#how-to-configure)
* [Sources](#sources)
* [Processors](#processors)
* [List processors](#list-processors)
* [Edit processor](#edit-processor)
* [Targets](#targets)
* [Contributing](#contributing)
* [Credits](#credits)
* [License](#license)
## Installation
### Using Docker
- Pull the Docker image from the GitHub Container Registry:
```bash
docker pull ghcr.io/quix-labs/thunder:latest
```
- Start container
```bash
docker run -p 3000:3000 -v "./config.json:/config.json" --name thunder ghcr.io/quix-labs/thunder:latest
```
### Using prebuilt assets
You can also install the tool using release assets such as `.deb`, `.apk`, or others.
Download the appropriate package from the [Releases page](https://github.com/quix-labs/thunder/releases), and then
follow the instructions provided for your specific platform.
### Using go (build from source as library)
- Create boilerplate
```bash
cd /path/your/want
go mod init your_app_name
go get -u github.com/quix-labs/thunder/app@main
touch main.go # Create base app file
```
- Edit `main.go` and put this:
```go
package main
import (
"github.com/quix-labs/thunder"
// Remove unnecessary modules/drivers
_ "github.com/quix-labs/thunder/modules/api"
_ "github.com/quix-labs/thunder/modules/frontend"
_ "github.com/quix-labs/thunder/modules/http_server"
_ "github.com/quix-labs/thunder/source-drivers/postgresql_flash"
_ "github.com/quix-labs/thunder/target-drivers/elastic"
)
func main() {
err := thunder.Start()
if err != nil {
panic(err)
}
}
```
- Compile your app:
```bash
go mod tidy #Download add modules/drivers from main.go
# CGO_ENABLED=0 is optional, but better for cross-platform compilation
# Ignore it if your are building directly on the target
CGO_ENABLED=0 go build -ldflags="-s -w" -o your_app_name
```
- Start your compiled app:
```bash
./your_app_name
# Actually, the config.json file will be placed in the current directory
# You can now access localhost:3000 and configure
```
## How to configure
### Sources
Sources define your database connections
Go to `/sources` to configure them
### Processors
Processor describe how your structured data will be destructured
Go to `/processors` to configure them
#### List processors
#### Edit processor
* Using the frontend, you can have autocompletion of table and columns
### Targets
Targets define your indexer connections
Go to `/targets` to configure them
## Realtime compatibility Table
| Target | Insert | Insert rel_col | Update | Update rel_col | Delete | Delete rel col | Truncate |
|:----------------:|:------:|:--------------:|:------:|:--------------:|:------:|:--------------:|:--------:|
| Base table | ❌ ◀ | ❌ ◀ | ✅ | ❌ ◀ | ✅ | ❌ | ✅ |
| one_to_one | N/A | ❌ ◀ | ✅ | ❌ ◀ | ✅ | ❌ | ✅ |
| has-many | ❌ ⏳ | ❌ ◀ | ✅ | ❌ ◀ | ✅ | ❌ | ✅ |
| has-many (pivot) | ❌ ◀ | ❌ ◀ | ❌ ◀ | ❌ ◀ | ❌ | ❌ | ❌ |
- ◀: `Need to send select request (impact DB)`
## Contributing
1. Fork the repository.
2. Create a new branch for your feature or bugfix.
3. Commit your changes.
4. Push your branch.
5. Create a pull request.
## Credits
- [COLANT Alan](https://github.com/alancolant)
- [All Contributors](../../contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE.md) for more information.