{"id":30461639,"url":"https://github.com/ydb-platform/ydb-cdc-processor","last_synced_at":"2026-01-20T17:38:50.717Z","repository":{"id":279444246,"uuid":"938832536","full_name":"ydb-platform/ydb-cdc-processor","owner":"ydb-platform","description":"Change Data Capture application example","archived":false,"fork":false,"pushed_at":"2025-08-01T15:13:28.000Z","size":94,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-08-01T17:35:23.431Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ydb-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-25T15:17:04.000Z","updated_at":"2025-08-01T15:13:32.000Z","dependencies_parsed_at":"2025-02-25T16:29:29.398Z","dependency_job_id":"6b34df94-e8ef-4654-82ee-2086c8eaaa88","html_url":"https://github.com/ydb-platform/ydb-cdc-processor","commit_stats":null,"previous_names":["ydb-platform/ydb-cdc-processor"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ydb-platform/ydb-cdc-processor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-cdc-processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-cdc-processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-cdc-processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-cdc-processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ydb-platform","download_url":"https://codeload.github.com/ydb-platform/ydb-cdc-processor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-cdc-processor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271768993,"owners_count":24817598,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-23T21:44:42.073Z","updated_at":"2026-01-20T17:38:50.677Z","avatar_url":"https://github.com/ydb-platform.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"## YDB Async replication app\n\n### Build with Maven\n#### Requirements\n\n* Java 17 or newer\n* Maven 3.0.0 or newer\n\n#### Build one-in-all JAR\n\n```\nmvn clean package spring-boot:repackage\n```\n\n### Usage\n\nCreate table\n\n```sql\nCREATE TABLE table_source (\n    event_time Text NOT NULL,\n    event_type Text NOT NULL,\n    product_id Uint32 NOT NULL,\n    category_id Uint64 NOT NULL,\n    category_code Text,\n    brand Text,\n    price Double NOT NULL,\n    user_id Uint32 NOT NULL,\n    user_session Text NOT NULL,\n    PRIMARY KEY (product_id, category_id, user_id, user_session)\n);\n```\n\nCreate dependended tables\n\n```sql\nCREATE TABLE mat_view1 (\n    event_time Timestamp NOT NULL, -- event_time with other type\n    event_type Text NOT NULL,\n    product_id Uint32 NOT NULL,\n    user_id Uint32 NOT NULL,\n    category_id Uint64 NOT NULL,\n    user_session Text NOT NULL,\n    PRIMARY KEY (product_id, category_id, user_id, user_session)\n);\n\nCREATE TABLE mat_view2 (\n    user_session Text NOT NULL,\n    user_id Uint32 NOT NULL,\n    event_type Text NOT NULL,\n    category_id Uint64 NOT NULL,\n    category_code Text,\n    brand Text,\n    price Double NOT NULL,\n    PRIMARY KEY (user_session, user_id, event_type, category_id) -- other primary key\n);\n\n```\n\nAdd changefeed to source table and consumers for every dependent table\n```sql\nALTER TABLE table_source ADD CHANGEFEED cdc_topic WITH (\n    FORMAT = 'JSON',\n    MODE = 'NEW_IMAGE',\n    INITIAL_SCAN = true,\n    VIRTUAL_TIMESTAMPS = true\n);\n\nALTER TOPIC `table_source/cdc_topic`\n    ADD CONSUMER v1_consumer,\n    ADD CONSUMER v2_consumer;\n```\n\n\nCreate config file with content\n```xml\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cconfig\u003e\n    \u003ccdc changefeed=\"table_source/cdc_topic\" consumer=\"v1_consumer\" batchSize=\"10\" \u003e\n\u003c![CDATA[\nDECLARE $rows AS List\u003cStruct\u003c\n    event_time: Text,\n    event_type: Text,\n    product_id: Uint32,\n    category_id: Uint64,\n    category_code: Text?,\n    user_id: Uint32,\n    user_session: Text\n\u003e\u003e;\n\n$parse=DateTime::Parse('%Y-%m-%d %H:%M:%S %Z');\n\nUPSERT INTO mat_view1 SELECT\n    Unwrap(DateTime::MakeTimestamp($parse(event_time))) AS event_time,\n    event_type,\n    product_id,\n    user_id,\n    category_id,\n    user_session\nFROM AS_TABLE($rows);\n]]\u003e\n    \u003c/cdc\u003e\n    \u003ccdc changefeed=\"table_source/cdc_topic\" consumer=\"v2_consumer\"\u003e\n\u003c![CDATA[\nDECLARE $rows AS List\u003cStruct\u003c\n    event_type: Text,\n    category_id: Uint64,\n    category_code: Text?,\n    brand: Text?,\n    price: Double,\n    user_id: Uint32,\n    user_session: Text\n\u003e\u003e;\nUPSERT INTO mat_view2 SELECT * FROM AS_TABLE($rows);\n]]\u003e\n    \u003c/cdc\u003e\n\u003c/config\u003e\n```\n\nAnd run application\n```\njava -jar ydb-cdc-view-0.9.0-SNAPSHOT.jar --ydb.connection.url=\u003cconnection-url\u003e  \u003cpath-to-config.xml\u003e\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fydb-platform%2Fydb-cdc-processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fydb-platform%2Fydb-cdc-processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fydb-platform%2Fydb-cdc-processor/lists"}