{"id":18359009,"url":"https://github.com/ev2900/mongodb_streams_glue_iceberg","last_synced_at":"2025-10-15T02:02:47.135Z","repository":{"id":169405613,"uuid":"644900405","full_name":"ev2900/MongoDB_Streams_Glue_Iceberg","owner":"ev2900","description":"Process DynamoDB change streams via. AWS Glue w Iceberg to keep a copy of a collection in S3 upto date","archived":false,"fork":false,"pushed_at":"2025-03-30T01:10:07.000Z","size":28638,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-10T03:13:44.061Z","etag":null,"topics":["apache-iceberg","aws-glue","glue","mondodb","mongodb-change-streams","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ev2900.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-24T13:40:54.000Z","updated_at":"2025-03-30T01:10:08.000Z","dependencies_parsed_at":"2024-01-17T00:37:25.704Z","dependency_job_id":"640a6417-5af8-4d09-a065-44a1d501a0fe","html_url":"https://github.com/ev2900/MongoDB_Streams_Glue_Iceberg","commit_stats":null,"previous_names":["ev2900/mongodb_streams_glue_iceberg"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FMongoDB_Streams_Glue_Iceberg","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FMongoDB_Streams_Glue_Iceberg/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FMongoDB_Streams_Glue_Iceberg/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ev2900%2FMongoDB_Streams_Glue_Iceberg/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ev2900","download_url":"https://codeload.github.com/ev2900/MongoDB_Streams_Glue_Iceberg/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248148242,"owners_count":21055548,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-iceberg","aws-glue","glue","mondodb","mongodb-change-streams","python"],"created_at":"2024-11-05T22:20:22.628Z","updated_at":"2025-10-15T02:02:47.130Z","avatar_url":"https://github.com/ev2900.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MongoDB Streams AWS Glue\n\n\u003cimg width=\"85\" alt=\"map-user\" src=\"https://img.shields.io/badge/views-1662-green\"\u003e \u003cimg width=\"125\" alt=\"map-user\" src=\"https://img.shields.io/badge/unique visits-253-green\"\u003e\n\nThis repository has a solution for a 1 time copy of data from a MongoDB collection into an Apache Iceberg table in S3 and a solution to use the MongoDB change stream to keep the iceberg copy of the data up to date.\n\nThe architecture below depicts the solution\n\n\u003cbr\u003e\n\n\u003cimg width=\"1000\" alt=\"map-user\" src=\"https://github.com/ev2900/MongoDB_Streams_Glue_Iceberg/blob/main/0_Architecture/mongoDB_glue_iceberg_architecture.png\"\u003e\n\nThe repository is broken down into several sections. Each section has its own read me that will explains its components\n\n* [1_Sample_MongoDB_Data](https://github.com/ev2900/MongoDB_Streams_Glue_Iceberg/tree/main/1_Sample_MongoDB_Data) steps 1 and 2 in architecture diagram.\n\n* [2_Glue_Iceberg_Initial_Load](https://github.com/ev2900/MongoDB_Streams_Glue_Iceberg/tree/main/2_Glue_Iceberg_Initial_Load) step 3 in the architecture diagram\n\n* [3_Sample_MongoDB_Change_Stream_Data](https://github.com/ev2900/MongoDB_Streams_Glue_Iceberg/tree/main/2_Glue_Iceberg_Initial_Load) steps 4 and 5 in the architecture diagram\n\n* [4_Glue_Iceberg_Change_Stream](https://github.com/ev2900/MongoDB_Streams_Glue_Iceberg/tree/main/2_Glue_Iceberg_Initial_Load) step 6 in the architecture diagram\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fev2900%2Fmongodb_streams_glue_iceberg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fev2900%2Fmongodb_streams_glue_iceberg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fev2900%2Fmongodb_streams_glue_iceberg/lists"}