{"id":16293905,"url":"https://github.com/sambaiz/athena-admin","last_synced_at":"2025-04-05T16:33:05.528Z","repository":{"id":57186063,"uuid":"115265799","full_name":"sambaiz/athena-admin","owner":"sambaiz","description":" Migrate the table schema, replace objects so that it has partition key=value prefix and add partitions.","archived":false,"fork":false,"pushed_at":"2018-01-12T00:30:49.000Z","size":122,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-21T07:11:11.549Z","etag":null,"topics":["athena","aws"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sambaiz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-24T13:49:20.000Z","updated_at":"2018-03-05T07:13:03.000Z","dependencies_parsed_at":"2022-09-14T09:50:21.271Z","dependency_job_id":null,"html_url":"https://github.com/sambaiz/athena-admin","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sambaiz%2Fathena-admin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sambaiz%2Fathena-admin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sambaiz%2Fathena-admin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sambaiz%2Fathena-admin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sambaiz","download_url":"https://codeload.github.com/sambaiz/athena-admin/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247367174,"owners_count":20927679,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["athena","aws"],"created_at":"2024-10-10T20:12:58.257Z","updated_at":"2025-04-05T16:33:05.222Z","avatar_url":"https://github.com/sambaiz.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# athena-admin\n\nMigrate the table schema, replace objects so that it has partition key=value prefix and add partitions.\n\n![overview](https://github.com/sambaiz/athena-admin/blob/master/doc/overview.png?raw=true)\n\n```\n$ npm install athena-admin\n```\n\n```\nconst AthenaAdmin = require('athena-admin').AthenaAdmin;\nconst dbDef = require('./sampledatabase.json');\nconst admin = new AthenaAdmin(dbDef);\nawait admin.replaceObjects();\nawait admin.migrate();\nawait admin.partition();\n```\n\n## Database definition\n\nDescribe the database definition in the following format.\n\n```\n{\n  \"general\": {\n    \"athenaRegion\": \"ap-northeast-1\",\n    \"databaseName\": \"aaaa\",\n    \"saveDefinitionLocation\": \"s3://saveDefinitionBucket/aaaa.json\"\n  },\n  \"tables\": {\n    \"sample_data\": {\n      \"columns\": {\n        \"user_id\": \"int\",\n        \"some_value\": { /* = \"struct\u003cscore:int,category:string\u003e\" */\n          \"score\": \"int\",\n          \"category\": \"string\"\n        },\n        \"some_array1\": [\"string\"], /* = array\u003cstring\u003e */\n        \"some_array2\": [{ /* = array\u003cstruct\u003caaa:int,bbb:string\u003e\u003e */\n          \"aaa\": \"int\",\n          \"bbb\": \"string\"\n        }]\n      },\n      \"srcLocation\": \"s3://src/location/\",\n      \"partition\": {\n        \"prePartitionLocation\": \"s3://pre/partition/\", /* optional */\n        \"regexp\": \"(\\\\d{4})/(\\\\d{2})/(\\\\d{2})/\", /* optional */\n        \"keys\": [\n          {\n            \"name\": \"dt\",\n            \"type\": \"string\",\n            \"format\": \"{1}-{2}-{3}\", /* optional */\n          }\n        ]\n      }\n    }\n  }\n}\n```\n\n### general\n\n| Field  | Description |\n|:-----------|:------------|\n| athenaRegion | Region for Athena |\n| databaseName | Athena database name |\n| saveDefinitionLocation | Location to save the previous definition |\n\n### tables\n\n- Root field name (sample_data) is a table name.\n\n| Field  | Description |\n|:-----------|:------------|\n| columns | Column name and type pairs. struct\u003c\u003e and array\u003c\u003e can also be described as a json object so you can describe these by converting the actual data values to the type. |\n| srcLocation | Location to be refferenced by Athena |\n| partition | Partition detectable by key=value prefix.\u003cbr\u003eIf objects' location don't have partition's key=value prefix, you can replace from prePartitionLocation to srcLocation by `replaceObjects()`. This is for `partition()` automatically detecting and adding partitions with keys.key as its key and keys.format as its value of keys.type as its type.\u003cbr\u003ekeys.format's {n} corresponds to the group of regexp. (e.g. `s3://pre/partition/2017/12/01/00/aaa.png` =\u003e `[2017/12/01, 2017, 12, 01]`) |\n\n### API\n\n### replaceObjects(deletePreObject=true, matchedHandler=(matched, objKey, table)=\u003ematched)\n\nReplaces object located in prePartitionLocation to srcLocation with partition key=value prefix.\n(e.g. `s3://pre/partition/2017/12/01/00/aaa.png` =\u003e `s3://src/location/dt=2017-12-01/00/aaa.png`)\n\nIf you need to change the key before this operation, use matchedHandler.\nThe following example is changing the UTC string to that of TimeZone.\n(e.g. `2017/12/01/19` =\u003e `2017/12/02/04`)\nThere are full codes in /sample.\n\n```\nconst utcToTZ = (matched, objKey, table) =\u003e {\n  let existsDt = false;\n  table.partition.keys.forEach((key) =\u003e {\n    if (key.name === 'dt') {\n      existsDt = true;\n    }\n  });\n  if (!existsDt) {\n    return matched;\n  }\n\n  let tz = moment(`${matched[0]} +00:00`, 'YYYY/MM/DD/HH ZZ');\n  matched[1] = tz.format('YYYY');\n  matched[2] = tz.format('MM');\n  matched[3] = tz.format('DD');\n  matched[4] = tz.format('HH');\n  return matched;\n};\n\nawait admin.replaceObjects(false, utcToTZ);\n```\n\n### migrate()\n\nIf there are differences from the previous saved definition in S3, create/drop the table or update the schema.\n\n### partition()\n\nJust run `MSCK REPAIR TABLE`. Partition is automatically detected and added by objects' key=value prefix.\n\n## Article\n\n[Athenaのmigrationやpartitionするathena-managerを作った - sambaiz-net](https://www.sambaiz.net/article/145/)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsambaiz%2Fathena-admin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsambaiz%2Fathena-admin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsambaiz%2Fathena-admin/lists"}