https://github.com/chandanpasunoori/event-sync
Event Sync is for syncing events from multiple sources to multiple destinations, targetted for adhoc events, where sources support acknowledgement functionality.
https://github.com/chandanpasunoori/event-sync
bigquery golang-tools google-cloud-platform pubsub
Last synced: 3 months ago
JSON representation
Event Sync is for syncing events from multiple sources to multiple destinations, targetted for adhoc events, where sources support acknowledgement functionality.
- Host: GitHub
- URL: https://github.com/chandanpasunoori/event-sync
- Owner: chandanpasunoori
- License: mit
- Created: 2021-09-13T22:02:24.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-03-18T12:24:53.000Z (over 1 year ago)
- Last Synced: 2025-04-04T05:48:24.101Z (3 months ago)
- Topics: bigquery, golang-tools, google-cloud-platform, pubsub
- Language: Go
- Homepage:
- Size: 207 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# event-sync
Event Sync is for syncing events from multiple sources to multiple destinations, targetted for adhoc events, where sources support acknowledgement functionality.
## suported
- input sources:
- Google Pub/Sub
- destination:
- Google BigQuery
- Streaming
- Load Jobs (uses Google Cloud Storage)
- Google Cloud Storage
### Synopsis
Built to ease process of syncing data between different storage systems
```
event-sync [flags]
```
### Options
```
-c, --config string job configuration file path (default "app.json")
-h, --help help for event-sync
-v, --verbose verbose mode
```
## Config Example
```json
{
"jobs": [
{
"name": "app-events",
"suspend": false,
"source": {
"type": "google-pubsub",
"pubsubConfig": {
"projectId": "gcp-project",
"subscriptionId": "pubsub-sub",
"maxOutstandingMessages": 5000,
"attributeKeyName": "event_type"
}
},
"filters": [
{
"type": "attribute",
"name": "event_name",
"action": "ingest",
"target": {
"table": "bq-table"
},
"schema": [
{
"name": "ts",
"type": "TIMESTAMP"
},
{
"name": "field1",
"type": "STRING"
},
{
"name": "field1",
"type": "STRING"
},
{
"name": "field3",
"type": "STRING"
}
]
}
],
"destination": {
"type": "google-storage-load",
"timestampColumnName": "ts",
"timestampFormat": "2006-01-02T15:04:05.999999999Z",
"timePartitioningType": "DAY",
"expiration": "2160h",
"clusterBy": [
"field1",
"field2",
"field3"
],
"bigqueryConfig": {
"projectId": "gcp-project",
"dataset": "bq-dataset"
},
"googleStorageConfig": {
"projectId": "gcp-project",
"bucket": "gcs-bucket",
"blobPrefix": "blob-prefix"
},
"batchSize": 1000
}
}
]
}
```