{"id":19705535,"url":"https://github.com/treeverse/lakefs-metastore-clone","last_synced_at":"2026-05-15T19:32:49.890Z","repository":{"id":48439466,"uuid":"355456810","full_name":"treeverse/lakefs-metastore-clone","owner":"treeverse","description":"Create a Hive Metastore clone pointing to a requested branch.","archived":false,"fork":false,"pushed_at":"2021-07-26T11:58:44.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-11-20T20:26:22.911Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/treeverse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-07T07:46:15.000Z","updated_at":"2021-07-26T11:58:46.000Z","dependencies_parsed_at":"2022-08-24T06:30:23.989Z","dependency_job_id":null,"html_url":"https://github.com/treeverse/lakefs-metastore-clone","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/treeverse/lakefs-metastore-clone","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treeverse%2Flakefs-metastore-clone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treeverse%2Flakefs-metastore-clone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treeverse%2Flakefs-metastore-clone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treeverse%2Flakefs-metastore-clone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/treeverse","download_url":"https://codeload.github.com/treeverse/lakefs-metastore-clone/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/treeverse%2Flakefs-metastore-clone/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33076193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-15T11:35:32.926Z","status":"ssl_error","status_checked_at":"2026-05-15T11:35:31.362Z","response_time":103,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T21:28:48.842Z","updated_at":"2026-05-15T19:32:49.872Z","avatar_url":"https://github.com/treeverse.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Hive metastore clone\n\n### About\n\nThis repository provides a way to get a containerized clone of your Glue Metastore, \npointing to a lakeFS branch.  \n\nThis repository contains \n- docker-compose for running Hive Metastore\n- the clone script that uses docker compose \n- the import script that uses docker compose\n\n### Prerequisite\n\nexport your lakeFS endpoint and lakeFS credentials \n\n```sh\n$ export S3A_ENDPOINT=s3.my.lakefs.com\n$ export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n``` \n\nEdit the following fields in the `lakectl.yaml` file\n\u003c/br\u003e`catalog_id` - aws numerical account-id\u003c/br\u003e`region`\u003c/br\u003e`access_secret_key`\u003c/br\u003e`access_key_id`\n\n### Running the clone script\n\nIn order to create a new Hive Metastore with tables from our source Hive Metastore\nwe should run the script `clone.sh`.\n\n`clone.sh` should receive the following arguments\n- port to export for new Hive Metastore\n- destination branch name\n- filter for schemas to be copied [OPTIONAL]\n- filter for tables to be copied [OPTIONAL]\n\nFor example, in order to get a copy of all schemas and tables (directed to example_branch) from glue to new hive metastore on port 9084 run:\n\n```sh\n$ ./clone.sh  9084 example_branch \n```\n\nFor filtering schemas starting with \"s\" and tables starting with \"a\" run:\n\n```sh\n$ ./clone.sh 9084 example_branch \"s.*\" \"a.*\"\n```\n\nThe Metastore for the requested branch is available on port `9084` for all interfaces.\n\n\n### Running the import script\n\nIn case our tables in the source Metastore do not exist in lakeFS\nwe should use the import script `import.sh`.\nThe import script transforms the table location from an external location to a lakeFS location.\ne.g table with location s3://my-bucket/path/to/table will be imported to the destination metastore with location s3://my-repo/my-branch/path/to/table\n\n`import.sh` should receive the following arguments\n- port to export for new Hive Metastore\n- destination repository name\n- destination branch name\n- filter for schemas to be copied [OPTIONAL]\n- filter for tables to be copied [OPTIONAL]\n\nFor example, in order to get a copy of all schemas and tables (directed to example_repo/example_branch) from glue to new hive metastore on port 9084 run:\n\n```sh\n$ ./import.sh 9084 example_repo example_branch \n```\n\nFor example, filtering schemas starting with \"s\" and tables starting with \"a\" run:\n\n```sh\n$ ./import.sh 9084 example_repo example_branch \"s.*\" \"a.*\"\n```\n\nThe Metastore for the requested branch is available on port `9084` for all interfaces.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreeverse%2Flakefs-metastore-clone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftreeverse%2Flakefs-metastore-clone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftreeverse%2Flakefs-metastore-clone/lists"}