{"id":22152138,"url":"https://github.com/findinpath/parquet-table-generator","last_synced_at":"2025-03-24T13:25:17.415Z","repository":{"id":112790989,"uuid":"449821991","full_name":"findinpath/parquet-table-generator","owner":"findinpath","description":"Proof of concept project for generating a wide Parquet partitioned table containing a lot of columns.","archived":false,"fork":false,"pushed_at":"2022-01-21T04:59:14.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T18:32:14.274Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-19T19:09:10.000Z","updated_at":"2022-01-21T04:58:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"ae5fb734-3fc5-42eb-bcca-3f128ef3a1ab","html_url":"https://github.com/findinpath/parquet-table-generator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fparquet-table-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fparquet-table-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fparquet-table-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fparquet-table-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/parquet-table-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245277472,"owners_count":20589154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T00:47:24.609Z","updated_at":"2025-03-24T13:25:17.409Z","avatar_url":"https://github.com/findinpath.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"Parquet Table Generator\n============================\n\nProof of concept project for generating:\n- wide Parquet partitioned table containing a lot of columns.\n- long Parquet partitioned table containing a lot of partitions.\n\nThe purpose of generating the wide table is to upload it AWS Glue and hit into\nthe use case of dealing with _UnprocessedKeys_ when trying  to obtain the partitions.\n\nhttps://docs.aws.amazon.com/cli/latest/reference/glue/batch-get-partition.html\n\n\nIt is initially not that obvious why AWS Glue API response for `batch-get-partition`\ncontains the field _UnprocessedKeys_ :\n\n\u003e A list of the partition values in the request for which partitions were not returned.\n\n\nWhen actually calling for `1000` partitions which each have information about `3000` columns,\nthe payload to be delivered is really big. This is probably why the AWS Glue API developers\ndid opt for introducing the _UnprocessedKeys_ field. It is somehow a soft way of saying that\nthe request involves delivering a response payload too big. By using this rather unusual method\nthe response can be batched by forcing the client to do multiple smaller batch partition calls.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fparquet-table-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Fparquet-table-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fparquet-table-generator/lists"}