{"id":34721663,"url":"https://github.com/equancy/datalake-catalog","last_synced_at":"2026-05-27T19:31:26.897Z","repository":{"id":62566627,"uuid":"488094069","full_name":"equancy/datalake-catalog","owner":"equancy","description":"REST API for datalake catalog","archived":false,"fork":false,"pushed_at":"2022-05-16T13:59:41.000Z","size":107,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-03-25T15:52:24.822Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/equancy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-03T05:58:13.000Z","updated_at":"2022-05-04T06:37:35.000Z","dependencies_parsed_at":"2022-11-03T17:47:48.231Z","dependency_job_id":null,"html_url":"https://github.com/equancy/datalake-catalog","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/equancy/datalake-catalog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equancy%2Fdatalake-catalog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equancy%2Fdatalake-catalog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equancy%2Fdatalake-catalog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equancy%2Fdatalake-catalog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/equancy","download_url":"https://codeload.github.com/equancy/datalake-catalog/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equancy%2Fdatalake-catalog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33581549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-27T02:00:06.184Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T01:48:27.577Z","updated_at":"2026-05-27T19:31:26.892Z","avatar_url":"https://github.com/equancy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Setup and start the API \n\nConfigure the parameters with a [python file](https://flask.palletsprojects.com/en/2.0.x/config/#configuring-from-python-files) \n\nFor example, `catalog.conf.py`\n\n```python\nSECRET_KEY = b\"changemenow\"\nDB_STRING = \"sqlite://localhost/catalog.sqlite\"\n```\n\nStart the catalog \n\n```shell\ndocker run -d \\\n    -p '8080:8080' \\\n    -v 'catalog.conf.py:/etc/datacatalog/catalog.conf.py' \\\n    -e 'CATALOG_SETTINGS=/etc/datacatalog/catalog.conf.py' \\\n    public.ecr.aws/equancy-tech/datalake-catalog\n```\n\n## Generate an API token\n\nThe `catalog create-api-key` generated tokens that can be used with restricted endpoint.\n\n- **admin** role can access all restricted endpoints\n- **author** role can only access the restricted endpoints in `/catalog`\n\n```shell\nUsage: catalog create-api-key [OPTIONS] NAME\n\nOptions:\n  -e, --expires INTEGER      number of days before token expires\n  -r, --role [author|admin]  the role associated with the name\n  --help                     Show this message and exit.\n```\n\n## Update the storages configuration\n\nStorage configures aliases to help resolve actual buckets (S3, Azure, local fs, etc.).\nIt consists of a bucket name and an optional prefix.\n\nFor example:\n\n```json\n{\n    \"landing\": {\n        \"bucket\": \"my-raw-bucket\",\n        \"prefix\": \"landing\"\n    },\n    \"archive\": {\n        \"bucket\": \"my-raw-bucket\",\n        \"prefix\": \"archives\"\n    },\n    \"bronze\": {\n        \"bucket\": \"my-bucket-bronze\"\n    },\n    \"silver\": {\n        \"bucket\": \"my-bucket-silver\"\n    },\n    \"gold\": {\n        \"bucket\": \"my-bucket-gold\",\n    }\n}\n```\n\n```shell\ncurl -XPUT http://localhost:8080/storage \\\n    -H \"Content-Type: application/json\" \\\n    -H \"Authorization: Bearer ${CATALOG_ADMIN_TOKEN}\" \\\n    --data \"@${STORAGE_CONFIG_FILE}\" \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequancy%2Fdatalake-catalog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fequancy%2Fdatalake-catalog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequancy%2Fdatalake-catalog/lists"}