{"id":13671093,"url":"https://github.com/gabledata/recap","last_synced_at":"2026-03-11T12:12:19.410Z","repository":{"id":65304018,"uuid":"575524661","full_name":"gabledata/recap","owner":"gabledata","description":"Work with your web service, database, and streaming schemas in a single format.","archived":false,"fork":false,"pushed_at":"2025-12-30T22:10:01.000Z","size":1737,"stargazers_count":348,"open_issues_count":20,"forks_count":27,"subscribers_count":10,"default_branch":"main","last_synced_at":"2026-01-03T20:20:44.350Z","etag":null,"topics":["data-catalog","data-discovery","data-engineering","data-integration","data-pipelines","etl","metadata","recap"],"latest_commit_sha":null,"homepage":"https://recap.build","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gabledata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-12-07T17:50:44.000Z","updated_at":"2025-12-30T22:10:05.000Z","dependencies_parsed_at":"2025-12-31T08:01:08.468Z","dependency_job_id":null,"html_url":"https://github.com/gabledata/recap","commit_stats":{"total_commits":325,"total_committers":13,"mean_commits":25.0,"dds":"0.13230769230769235","last_synced_commit":"a35571c20baabb0aa06018b1cfc3426063418fc5"},"previous_names":["recap-cloud/recap","gabledata/recap","recap-build/recap"],"tags_count":40,"template":false,"template_full_name":null,"purl":"pkg:github/gabledata/recap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabledata%2Frecap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabledata%2Frecap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabledata%2Frecap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabledata%2Frecap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gabledata","download_url":"https://codeload.github.com/gabledata/recap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gabledata%2Frecap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30380935,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-11T06:09:32.197Z","status":"ssl_error","status_checked_at":"2026-03-11T06:09:17.086Z","response_time":84,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-catalog","data-discovery","data-engineering","data-integration","data-pipelines","etl","metadata","recap"],"created_at":"2024-08-02T09:00:58.813Z","updated_at":"2026-03-11T12:12:19.389Z","avatar_url":"https://github.com/gabledata.png","language":"Python","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/recap-build/recap/blob/main/static/recap-logo.png?raw=true\" alt=\"recap\"\u003e\n\u003c/div\u003e\n\n## What is Recap?\n\nRecap reads and writes schemas from web services, databases, and schema registries in a standard format.\n\n⭐️ _If you like this project, please give it a star! It helps the project get more visibility._\n\n## Table of Contents\n\n* [What is Recap?](#what-is-recap)\n* [Supported Formats](#supported-formats)\n* [Install](#install)\n* [Usage](#usage)\n   * [CLI](#cli)\n   * [Gateway](#gateway)\n   * [Registry](#registry)\n   * [API](#api)\n   * [Docker](#docker)\n* [Schema](#schema)\n* [Documentation](#documentation)\n\n## Supported Formats\n\n| Format      | Read | Write |\n| :---------- | :-: | :-: |\n| [Avro](https://recap.build/docs/integrations/avro/) | ✅ | ✅ |\n| [BigQuery](https://recap.build/docs/integrations/bigquery/) | ✅ |  |\n| [Confluent Schema Registry](https://recap.build/docs/integrations/confluent-schema-registry/) | ✅ |  |\n| [Hive Metastore](https://recap.build/docs/integrations/hive-metastore/) | ✅ |  |\n| [JSON Schema](https://recap.build/docs/integrations/json-schema/) | ✅ | ✅ |\n| [MySQL](https://recap.build/docs/integrations/mysql/) | ✅ |  |\n| [PostgreSQL](https://recap.build/docs/integrations/postgresql/) | ✅ |  |\n| [Protobuf](https://recap.build/docs/integrations/protobuf/) | ✅ | ✅ |\n| [Snowflake](https://recap.build/docs/integrations/snowflake/) | ✅ |  |\n| [SQLite](https://recap.build/docs/integrations/sqlite/) | ✅ |  |\n\n## Install\n\nInstall Recap and all of its optional dependencies:\n\n```bash\npip install 'recap-core[all]'\n```\n\nYou can also select specific dependencies:\n\n```bash\npip install 'recap-core[avro,kafka]'\n```\n\nSee `pyproject.toml` for a list of optional dependencies.\n\n## Usage\n\n### CLI\n\nRecap comes with a command line interface that can list and read schemas from external systems.\n\nList the children of a URL:\n\n```bash\nrecap ls postgresql://user:pass@host:port/testdb\n```\n\n```json\n[\n  \"pg_toast\",\n  \"pg_catalog\",\n  \"public\",\n  \"information_schema\"\n]\n```\n\nKeep drilling down:\n\n```bash\nrecap ls postgresql://user:pass@host:port/testdb/public\n```\n\n```json\n[\n  \"test_types\"\n]\n```\n\nRead the schema for the `test_types` table as a Recap struct:\n\n```bash\nrecap schema postgresql://user:pass@host:port/testdb/public/test_types\n```\n\n```json\n{\n  \"type\": \"struct\",\n  \"fields\": [\n    {\n      \"type\": \"int64\",\n      \"name\": \"test_bigint\",\n      \"optional\": true\n    }\n  ]\n}\n```\n\n### Gateway\n\nRecap comes with a stateless HTTP/JSON gateway that can list and read schemas from data catalogs and databases.\n\nStart the server at [http://localhost:8000](http://localhost:8000):\n\n```bash\nrecap serve\n```\n\nList the schemas in a PostgreSQL database:\n\n```bash\ncurl http://localhost:8000/gateway/ls/postgresql://user:pass@host:port/testdb\n```\n\n```json\n[\"pg_toast\",\"pg_catalog\",\"public\",\"information_schema\"]\n```\n\nAnd read a schema:\n\n```bash\ncurl http://localhost:8000/gateway/schema/postgresql://user:pass@host:port/testdb/public/test_types\n```\n\n```json\n{\"type\":\"struct\",\"fields\":[{\"type\":\"int64\",\"name\":\"test_bigint\",\"optional\":true}]}\n```\n\nThe gateway fetches schemas from external systems in realtime and returns them as Recap schemas.\n\nAn OpenAPI schema is available at [http://localhost:8000/docs](http://localhost:8000/docs).\n\n### Registry\n\nYou can store schemas in Recap's schema registry.\n\nStart the server at [http://localhost:8000](http://localhost:8000):\n\n```bash\nrecap serve\n```\n\nPut a schema in the registry:\n\n```bash\ncurl -X POST \\\n    -H \"Content-Type: application/x-recap+json\" \\\n    -d '{\"type\":\"struct\",\"fields\":[{\"type\":\"int64\",\"name\":\"test_bigint\",\"optional\":true}]}' \\\n    http://localhost:8000/registry/some_schema\n```\n\nGet the schema (and version) from the registry:\n\n```bash\ncurl http://localhost:8000/registry/some_schema\n```\n\n```json\n[{\"type\":\"struct\",\"fields\":[{\"type\":\"int64\",\"name\":\"test_bigint\",\"optional\":true}]},1]\n```\n\nPut a new version of the schema in the registry:\n\n```bash\ncurl -X POST \\\n    -H \"Content-Type: application/x-recap+json\" \\\n    -d '{\"type\":\"struct\",\"fields\":[{\"type\":\"int32\",\"name\":\"test_int\",\"optional\":true}]}' \\\n    http://localhost:8000/registry/some_schema\n```\n\nList schema versions:\n\n```bash\ncurl http://localhost:8000/registry/some_schema/versions\n```\n\n```json\n[1,2]\n```\n\nGet a specific version of the schema:\n\n```bash\ncurl http://localhost:8000/registry/some_schema/versions/1\n```\n\n```json\n[{\"type\":\"struct\",\"fields\":[{\"type\":\"int64\",\"name\":\"test_bigint\",\"optional\":true}]},1]\n```\n\nThe registry uses [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) to store schemas in a variety of filesystems like S3, GCS, ABS, and the local filesystem. See the [registry](https://recap.build/docs/registry/) docs for more details.\n\nAn OpenAPI schema is available at [http://localhost:8000/docs](http://localhost:8000/docs).\n\n### API\n\nRecap has `recap.converters` and `recap.clients` packages.\n\n- Converters convert schemas to and from Recap schemas.\n- Clients read schemas from external systems (databases, schema registries, and so on) and use converters to return Recap schemas.\n\nRead a schema from PostgreSQL:\n\n```python\nfrom recap.clients import create_client\n\nwith create_client(\"postgresql://user:pass@host:port/testdb\") as c:\n    c.schema(\"testdb\", \"public\", \"test_types\")\n```\n\nConvert the schema to Avro, Protobuf, and JSON schemas:\n\n```python\nfrom recap.converters.avro import AvroConverter\nfrom recap.converters.protobuf import ProtobufConverter\nfrom recap.converters.json_schema import JSONSchemaConverter\n\navro_schema = AvroConverter().from_recap(struct)\nprotobuf_schema = ProtobufConverter().from_recap(struct)\njson_schema = JSONSchemaConverter().from_recap(struct)\n```\n\nTranspile schemas from one format to another:\n\n```python\nfrom recap.converters.json_schema import JSONSchemaConverter\nfrom recap.converters.avro import AvroConverter\n\njson_schema = \"\"\"\n{\n    \"type\": \"object\",\n    \"$id\": \"https://recap.build/person.schema.json\",\n    \"properties\": {\n        \"name\": {\"type\": \"string\"}\n    }\n}\n\"\"\"\n\n# Use Recap as an intermediate format to convert JSON schema to Avro\nstruct = JSONSchemaConverter().to_recap(json_schema)\navro_schema = AvroConverter().from_recap(struct)\n```\n\nStore schemas in Recap's schema registry:\n\n```python\nfrom recap.storage.registry import RegistryStorage\nfrom recap.types import StructType, IntType\n\nstorage = RegistryStorage(\"file:///tmp/recap-registry-storage\")\nversion = storage.put(\n    \"postgresql://localhost:5432/testdb/public/test_table\",\n    StructType(fields=[IntType(32)])\n)\nstorage.get(\"postgresql://localhost:5432/testdb/public/test_table\")\n\n# Get all versions of a schema\nversions = storage.versions(\"postgresql://localhost:5432/testdb/public/test_table\")\n\n# List all schemas in the registry\nschemas = storage.ls()\n```\n\n### Docker\n\nRecap's gateway and registry are also available as a Docker image:\n\n```bash\ndocker run \\\n    -p 8000:8000 \\\n    -e RECAP_URLS=[\"postgresql://user:pass@localhost:5432/testdb\"]' \\\n    ghcr.io/recap-build/recap:latest\n```\n\nSee [Recap's Docker documentation](https://recap.build/docs/gateway/docker) for more details.\n\n## Schema\n\nSee [Recap's type spec](https://recap.build/specs/type) for details on Recap's type system.\n\n## Documentation\n\nRecap's documentation is available at [recap.build](https://recap.build).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabledata%2Frecap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgabledata%2Frecap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabledata%2Frecap/lists"}