https://github.com/saifulaiub123/elasticsync.net
Real-time high-performance synchronization engine for syncing relational database changes to Elasticsearch index
https://github.com/saifulaiub123/elasticsync.net
background-service data database dotnet dotnetcore elastic elastic-search elasticsearch fulltext-search fulltextsearch npgsql postgresql sqlserver sync
Last synced: 4 months ago
JSON representation
Real-time high-performance synchronization engine for syncing relational database changes to Elasticsearch index
- Host: GitHub
- URL: https://github.com/saifulaiub123/elasticsync.net
- Owner: saifulaiub123
- Created: 2025-08-09T06:02:09.000Z (6 months ago)
- Default Branch: master
- Last Pushed: 2025-08-22T05:18:38.000Z (5 months ago)
- Last Synced: 2025-09-27T05:58:32.274Z (4 months ago)
- Topics: background-service, data, database, dotnet, dotnetcore, elastic, elastic-search, elasticsearch, fulltext-search, fulltextsearch, npgsql, postgresql, sqlserver, sync
- Language: C#
- Homepage:
- Size: 167 KB
- Stars: 5
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# ElasticSync.Net
[](https://www.nuget.org/packages/ElasticSync.Net)
[](https://www.nuget.org/packages/ElasticSync.Net)
**ElasticSync.Net** is a **high-performance**, **real-time** data synchronization engine for syncing relational database changes to Elasticsearch. It supports **PostgreSQL** today, with a **Modular Core** that allows future extensions for **SQL Server**, **MySQL**, and more.
It supports both **trigger-based** and **interval-based** change tracking, with **parallel processing**, **dead-letter queuing**, and **bulk indexing** support.
It’s designed for the teams who want a lighweight system for **Instant Search Indexing** without building and maintaining complex change-tracking pipelines or without installing any additional tools/server like Debzium, Kafka etc.
---
## Features
- **Supported database so far** PostgreSQL
- **Real-time sync** from Relational Database to Elasticsearch
- **Interval-based** syncing with configurable intervals
- **Parallel work** Configurable Parallel workers for high-throughput scenarios like (1000/2000+ rows/sec).
- **Impact on DB** Minimal impact on source database
- **Multiple entity type support** (e.g., Products, Orders, Customers)
- **Automatic trigger & change tracking** in PostgreSQL
- **Retry logic** Retry logic with exponential backoff
- **Dead-letter** queue for failed operations
- **Bulk indexing** for performance optimization
- **Dashboard and monitoring** Pending...
- **Modular design** (core + PostgreSQL-specific provider)
-** Metrics**-friendly architecture
---
## Performance
Tested with 1000 inserts/sec:
- Throughput: ~12,000 records/min per node (scalable with workers)
- Latency: <2 seconds (average) with trigger-based realtime mode
- Durable retry with exponential backoff
Partitioning **esnet.elastic_sync_change_log** table can improves batch processing performance significantly.
## Sync with PostgreSql Database:
---
## Install the NuGet Package
```sh
dotnet add package ElasticSync.NET.PostgreSql
```
## Requirements
.NET 8, .NET 9
PostgreSQL 13+ (with trigger support)
Elasticsearch 8.x (or compatible OpenSearch version)
## Use Real-Time Sync
```sh
builder.Services.AddElasticSyncEngine(options =>
{
options.ElasticsearchUrl = builder.Configuration["Elasticsearch:Uri"];
options.RealTimeSync(batchSize: 500);
options.MaxRetries = 5;
options.RetryDelayInSeconds = 20;
options.Entities = new List
{
new TrackedEntity { Table = "Customers", EntityType = typeof(Customer), PrimaryKey = "Id", IndexName = "customers"},
new TrackedEntity { Table = "Products", EntityType = typeof(Product), PrimaryKey = "Id", IndexName = "products"},
};
}, (options, services) =>
{
options.AddElasticSyncPostgreSqlServices(services, connectionString);
});
```
## Use Real-Time Sync With Multiple Workers for High Volume Data Changes
```sh
builder.Services.AddElasticSyncEngine(options =>
{
options.ElasticsearchUrl = builder.Configuration["Elasticsearch:Uri"];
options.RealTimeSync()
.EnableMultipleWorkers(new WorkerOptions
{
BatchSizePerWorker = 300,
NumberOfWorkers = 4
});
options.MaxRetries = 5;
options.RetryDelayInSeconds = 20;
options.Entities = new List
{
new TrackedEntity { Table = "Customers", EntityType = typeof(Customer), PrimaryKey = "Id", IndexName = "customers"},
new TrackedEntity { Table = "Products", EntityType = typeof(Product), PrimaryKey = "Id", IndexName = "products"},
};
}, (options, services) =>
{
options.AddElasticSyncPostgreSqlServices(services, connectionString);
});
```
## Use Interval Sync
```sh
builder.Services.AddElasticSyncEngine(options =>
{
options.ElasticsearchUrl = builder.Configuration["Elasticsearch:Uri"];
options.IntervalSync(intervalInSeconds: 20, batchSize: 500);
options.MaxRetries = 5;
options.RetryDelayInSeconds = 20;
options.Entities = new List
{
new TrackedEntity { Table = "Customers", EntityType = typeof(Customer), PrimaryKey = "Id", IndexName = "customers"},
new TrackedEntity { Table = "Products", EntityType = typeof(Product), PrimaryKey = "Id", IndexName = "products"},
};
}, (options, services) =>
{
options.AddElasticSyncPostgreSqlServices(services, connectionString);
});
```
## Use Interval Sync With Multiple Workers for High Volume Data Changes
```sh
builder.Services.AddElasticSyncEngine(options =>
{
options.ElasticsearchUrl = builder.Configuration["Elasticsearch:Uri"];
options.IntervalSync(intervalInSeconds: 20, batchSize: 500)
.EnableMultipleWorkers(new WorkerOptions
{
BatchSizePerWorker = 300,
NumberOfWorkers = 4 //number of parallel worker
});
options.MaxRetries = 5;
options.RetryDelayInSeconds = 20;
options.Entities = new List
{
new TrackedEntity { Table = "Customers", EntityType = typeof(Customer), PrimaryKey = "Id", IndexName = "customers"},
new TrackedEntity { Table = "Products", EntityType = typeof(Product), PrimaryKey = "Id", IndexName = "products"},
};
}, (options, services) =>
{
options.AddElasticSyncPostgreSqlServices(services, connectionString);
});
```
## Real Time Syncing Background Process with Diagram
## Project Structure
- **ElasticSync.Net** – core interfaces, abstraction, and engine
- **ElasticSync.Net.PostgreSql** – PostgreSQL-specific implementation
- **ElasticSync.Net.SqlServer** – Coming soon
## Uninstall/Clean Up DB Object
If you want to remove the package to clean up all the database object you need to run a script which you will find under **UninstallScript** folder. **UninstallScript** is applicable from version **1.0.1**
## Future Plans
- Extend the project for **Sql Server** and **Mysql database**
- Expose API to get the Statistics like **Average processing lag**, **Unprocessed logs count per table**, **Success/failure rate per sync** etc
- Auth middleware for Elastic endpoints
- Dashboard and Real Time Monitoring
- Built-in metrics export
## Contributing
We welcome contributions!
Fork the repository
Create a new branch (git checkout -b feature/my-feature)
Commit your changes (git commit -m 'Add my feature')
Push to your fork (git push origin feature/my-feature)
Open a Pull Request
## License
This project is licensed under the MIT License
MIT © 2025 **ElasticSync.Net**
## 📅 Maintainer
**ElasticSync.NET** is maintained by **Md. Saiful Islam**
For support, bug reports, or feature requests, please open an issue on GitHub.