https://github.com/pierrekieffer/mongostream
MongoDB streaming service to launch async streams in parallel
https://github.com/pierrekieffer/mongostream
async asynchronous-programming change-streams mongodb mongodb-streaming-service streaming
Last synced: 2 months ago
JSON representation
MongoDB streaming service to launch async streams in parallel
- Host: GitHub
- URL: https://github.com/pierrekieffer/mongostream
- Owner: PierreKieffer
- License: bsd-2-clause
- Created: 2020-11-18T10:52:28.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2021-04-02T09:00:59.000Z (about 5 years ago)
- Last Synced: 2024-06-21T15:50:52.735Z (about 2 years ago)
- Topics: async, asynchronous-programming, change-streams, mongodb, mongodb-streaming-service, streaming
- Language: Go
- Homepage:
- Size: 17.6 KB
- Stars: 0
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# mongoStream
mongoStream is a mongodb streaming service based on mongodb change streams feature.
The service allows to launch asynchronous streams in parallel.
* [Requirements](#requirements)
* [Installation](#installation)
* [Usage](#usage)
* [Example](#example)
## Requirements
* System requirements:
- mongodb 4.2 or higher
- mongodb replica set available
* Dependencies :
- mongodb driver `go get go.mongodb.org/mongo-driver/mongo`
## Installation
`go get github.com/PierreKieffer/mongoStream`
## Usage
To get started, import `mongoStream` package
```go
import (
"github.com/PierreKieffer/mongoStream/services"
)
```
Set up the mongo connection string
```go
mongoUri := "mongodb://localhost:27017"
```
Initialize messages buffer :
The buffer is a channel and allows to accumulate messages generated by the streaming service. The initialization takes the size of the buffer as a parameter (ie the number of messages that the buffer can contain).
```go
var buffer = services.InitBuffer(10)
```
Start one or more workers in parallel :
Parameters :
- mongodb connection string
- database name
- collection name
- buffer
- stream options [optional]
```go
go services.ListenerWorker(mongoUri, "database", "collection1", buffer)
go services.ListenerWorker(mongoUri, "database", "collection2", buffer)
```
Structure of messages generated by workers and received by the buffer :
```json
{
"DocumentKey": {
"DocumentId": "string type"
},
"Namespace": {
"Database": "string type",
"Collection": "string type"
},
"OperationType": "string type",
"UpdateDescription": {
"updatedFields": "map[string]interface{} type"
}
}
```
To resume a stream from a timestamp :
```go
cso := services.SetOptions(true, 1586360547)
go services.ListenerWorker(mongoUri, "database", "collection1", buffer,cso)
```
## Example
```go
package main
import (
"github.com/PierreKieffer/mongoStream/dataModel"
"github.com/PierreKieffer/mongoStream/services"
"log"
)
var exit = make(chan bool)
func main() {
mongoUri := "mongodb://localhost:27017"
// Init oplog buffer channel
var buffer = services.InitBuffer(10)
// Start consumer
go Consumer(buffer)
// Start producers
go services.ListenerWorker(mongoUri, "database", "collection1", buffer)
go services.ListenerWorker(mongoUri, "database", "collection2", buffer)
<-exit
}
func Consumer(logBuffer chan dataModel.Oplog) {
for {
log.Println("reveived data : ", <-logBuffer)
}
}
```