{"id":17239900,"url":"https://github.com/chandanpasunoori/event-sync","last_synced_at":"2025-08-21T22:11:56.406Z","repository":{"id":73915580,"uuid":"406142128","full_name":"chandanpasunoori/event-sync","owner":"chandanpasunoori","description":"Event Sync is for syncing events from multiple sources to multiple destinations, targetted for adhoc events, where sources support acknowledgement functionality.","archived":false,"fork":false,"pushed_at":"2024-03-18T12:24:53.000Z","size":212,"stargazers_count":1,"open_issues_count":8,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-08-13T15:59:00.617Z","etag":null,"topics":["bigquery","golang-tools","google-cloud-platform","pubsub"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chandanpasunoori.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-13T22:02:24.000Z","updated_at":"2023-12-23T21:25:46.000Z","dependencies_parsed_at":"2023-12-14T12:26:35.185Z","dependency_job_id":"a02d1d14-898b-4037-9288-f843db07697e","html_url":"https://github.com/chandanpasunoori/event-sync","commit_stats":null,"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/chandanpasunoori/event-sync","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandanpasunoori%2Fevent-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandanpasunoori%2Fevent-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandanpasunoori%2Fevent-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandanpasunoori%2Fevent-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chandanpasunoori","download_url":"https://codeload.github.com/chandanpasunoori/event-sync/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chandanpasunoori%2Fevent-sync/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271552783,"owners_count":24779817,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-21T02:00:08.990Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigquery","golang-tools","google-cloud-platform","pubsub"],"created_at":"2024-10-15T05:50:01.725Z","updated_at":"2025-08-21T22:11:56.383Z","avatar_url":"https://github.com/chandanpasunoori.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# event-sync\r\n\r\nEvent Sync is for syncing events from multiple sources to multiple destinations, targetted for adhoc events, where sources support acknowledgement functionality.\r\n\r\n## suported\r\n\r\n- input sources:\r\n - Google Pub/Sub\r\n- destination:\r\n - Google BigQuery\r\n - Streaming\r\n - Load Jobs (uses Google Cloud Storage)\r\n - Google Cloud Storage\r\n\r\n\r\n### Synopsis\r\n\r\nBuilt to ease process of syncing data between different storage systems\r\n\r\n```\r\nevent-sync [flags]\r\n```\r\n\r\n### Options\r\n\r\n```\r\n -c, --config string job configuration file path (default \"app.json\")\r\n -h, --help help for event-sync\r\n -v, --verbose verbose mode\r\n```\r\n\r\n## Config Example\r\n\r\n```json\r\n{\r\n \"jobs\": [\r\n {\r\n \"name\": \"app-events\",\r\n \"suspend\": false,\r\n \"source\": {\r\n \"type\": \"google-pubsub\",\r\n \"pubsubConfig\": {\r\n \"projectId\": \"gcp-project\",\r\n \"subscriptionId\": \"pubsub-sub\",\r\n \"maxOutstandingMessages\": 5000,\r\n \"attributeKeyName\": \"event_type\"\r\n }\r\n },\r\n \"filters\": [\r\n {\r\n \"type\": \"attribute\",\r\n \"name\": \"event_name\",\r\n \"action\": \"ingest\",\r\n \"target\": {\r\n \"table\": \"bq-table\"\r\n },\r\n \"schema\": [\r\n {\r\n \"name\": \"ts\",\r\n \"type\": \"TIMESTAMP\"\r\n },\r\n {\r\n \"name\": \"field1\",\r\n \"type\": \"STRING\"\r\n },\r\n {\r\n \"name\": \"field1\",\r\n \"type\": \"STRING\"\r\n },\r\n {\r\n \"name\": \"field3\",\r\n \"type\": \"STRING\"\r\n }\r\n ]\r\n }\r\n ],\r\n \"destination\": {\r\n \"type\": \"google-storage-load\",\r\n \"timestampColumnName\": \"ts\",\r\n \"timestampFormat\": \"2006-01-02T15:04:05.999999999Z\",\r\n \"timePartitioningType\": \"DAY\",\r\n \"expiration\": \"2160h\",\r\n \"clusterBy\": [\r\n \"field1\",\r\n \"field2\",\r\n \"field3\"\r\n ],\r\n \"bigqueryConfig\": {\r\n \"projectId\": \"gcp-project\",\r\n \"dataset\": \"bq-dataset\"\r\n },\r\n \"googleStorageConfig\": {\r\n \"projectId\": \"gcp-project\",\r\n \"bucket\": \"gcs-bucket\",\r\n \"blobPrefix\": \"blob-prefix\"\r\n },\r\n \"batchSize\": 1000\r\n }\r\n }\r\n ]\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchandanpasunoori%2Fevent-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchandanpasunoori%2Fevent-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchandanpasunoori%2Fevent-sync/lists"}