{"id":27384050,"url":"https://github.com/charlescol/schema-manager","last_synced_at":"2025-08-08T16:08:10.974Z","repository":{"id":257821155,"uuid":"867901649","full_name":"charlescol/schema-manager","owner":"charlescol","description":"Centralize and automate schema file management in a repository, and register schemas automatically with a schema registry.","archived":false,"fork":false,"pushed_at":"2025-05-09T12:54:28.000Z","size":356,"stargazers_count":24,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-20T14:42:01.366Z","etag":null,"topics":["avro","avro-schema","confluent-schema-registry","devops","grpc","json","json-schema","kafka","microservices","protobuf","protobuf-schema","schema-registry"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/charlescol.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-05T00:55:28.000Z","updated_at":"2025-06-03T10:23:05.000Z","dependencies_parsed_at":"2024-10-11T02:15:30.744Z","dependency_job_id":"b7aa1278-5be2-4017-83db-1db71ed4abf3","html_url":"https://github.com/charlescol/schema-manager","commit_stats":null,"previous_names":["charlescol/schema-manager"],"tags_count":15,"template":false,"template_full_name":null,"purl":"pkg:github/charlescol/schema-manager","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charlescol%2Fschema-manager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charlescol%2Fschema-manager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charlescol%2Fschema-manager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charlescol%2Fschema-manager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/charlescol","download_url":"https://codeload.github.com/charlescol/schema-manager/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/charlescol%2Fschema-manager/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269448828,"owners_count":24418961,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-schema","confluent-schema-registry","devops","grpc","json","json-schema","kafka","microservices","protobuf","protobuf-schema","schema-registry"],"created_at":"2025-04-13T16:17:09.687Z","updated_at":"2025-08-08T16:08:10.944Z","avatar_url":"https://github.com/charlescol.png","language":"TypeScript","readme":"## Centralize Your Schema Files in One Git Repository\n\n![npm version](https://img.shields.io/npm/v/@charlescol/schema-manager)\n![Build Status](https://github.com/charlescol/schema-manager/actions/workflows/npm-publish.yml/badge.svg)\n![npm downloads](https://img.shields.io/npm/dm/@charlescol/schema-manager)\n![License](https://img.shields.io/github/license/charlescol/schema-manager)\n\n## You might find it useful if:\n\n- You’re maintaining schema files (Protobuf, Avro, JSON) across multiple services or repositories\n- Some of these schemas depend on each other, and managing those dependencies manually has become error-prone\n- You want to avoid redundancy, and track versions cleanly through Git\n- You need to publish schemas to a registry (e.g., Confluent) as part of your CI/CD\n- You’re looking for a lightweight solution — no GitOps, no operators, no Maven plugins\n\n---\n\n## What Does It Do?\n\n**Schema Manager** is a Node.js-based CLI and SDK that:\n\n- Resolves dependencies explicitly using a versions.json file, avoiding duplication across versions\n- Applies topological sorting on the dependency graph to publish in the correct order\n- Applies schema transformations (e.g., injecting packages, resolving import paths)\n- Publishes schemas to a supported registry (currently: **Confluent Schema Registry**)\n\n**Sample Repo Demo**: [Try the example here in just a few minutes.](https://github.com/charlescol/schema-manager-example)\n**Guide**: [Scenario Example](#scenario-example)\n\n---\n\n### Introduction\n\nIn modern microservices architectures, separating concerns is essential for achieving scalability and maintainability. Managing schema files (e.g., Avro, Protobuf, JSON) across multiple services can quickly become complex and error-prone, especially when each microservice maintains its own schemas and handles their publication to a schema registry (e.g., Confluent Schema Registry).\n\n**Schema Manager** addresses these challenges by centralizing schema management and delegation. It maintains schema files in a centralized Git repository and handles deployment to the target Schema Registry (typically through a CI/CD pipeline).\n\nBy enforcing this separation of concerns, Schema Manager simplifies schema management across services and ensures **a clear separation of responsibilities**, allowing microservices to remain lightweight and focused.\n\nFor a more comprehensive explanation of how Schema Manager integrates into modern Kafka-oriented applications involving multiple microservices and languages, refer to the [Example of Integration with Schema Manager](#example-of-integration-with-schema-manager) section. This section covers schema publication, as well as the generation and distribution of models across services.\n\n---\n\n## Table of Contents\n\n- [Quick Start](#quick-start)\n- [Supported Schema Formats](#supported-schema-formats)\n- [Available Registries](#available-registries)\n- [How It Works](#how-it-works)\n- [Scenario Example](#scenario-example)\n  - [File Structure](#file-structure)\n  - [versions.json for topic1](#versionsjson-for-topic1)\n  - [versions.json for topic2](#versionsjson-for-topic2)\n  - [Additional Remarks](#additional-remarks)\n- [Example Schema Content](#example-schema-content)\n- [Building Schemas](#building-schemas)\n- [Publish to the Registry](#publish-to-the-registry)\n- [Full Code](#full-code)\n- [Schema Content](#schema-content)\n  - [Protobuf](#protobuf)\n  - [Avro](#avro)\n  - [Json](#json)\n- [Future Plans and Roadmap](#future-plans-and-roadmap)\n- [Example of Typical Integration with Schema Manager](#example-of-typical-integration-with-schema-manager)\n  - [Benefits](#benefits)\n- [Contributing](#contributing)\n- [License](#license)\n\n---\n\n### Quick Start\n\nThe Schema Manager is distributed via NPM:\n\n```bash\nnpm install @charlescol/schema-manager\n```\n\nThe minimal requirement is to initiate an npm project within your repository and install the package. The schema manager capabilities can be used programmatically in a JavaScript script file (example [here](#full-code)).\n\n```bash\nmkdir schema-manager-example \u0026\u0026 cd schema-manager-example # Create the folder\nnpm init -y \u0026\u0026 npm install @charlescol/schema-manager # Initiate the project with the Schema-Manager package\n```\n\n**Sample Repo Demo**: [Try the example here in just a few minutes.](https://github.com/charlescol/schema-manager-example)\n\n---\n\n## Supported Schema Formats\n\n| **Config**   | **Supported Formats** | **Description**                                                                            |\n| ------------ | --------------------- | ------------------------------------------------------------------------------------------ |\n| **Avro**     | `.avro`, `.avsc`      | Parses Avro schema files, supports extracting dependencies from `.avro` and `.avsc` files. |\n| **Protobuf** | `.proto`              | Parses Protobuf schema files, supports extracting dependencies from `.proto` files.        |\n| **Json**     | `.json`               | Parses JSON schema files, supports extracting dependencies from `.json` files.             |\n\nPlease refer to the [Parser Documentation](how-to/create-parser.md) for more details on how to contribute and create a parser.\n\n---\n\n## Available Registries\n\n| **Registry**                  | **Class Name**      | **Supported Registries** |\n| ----------------------------- | ------------------- | ------------------------ |\n| **Confluent Schema Registry** | `ConfluentRegistry` | Confluent Kafka          |\n\nPlease refer to the [Registry Documentation](how-to/create-registry.md) for more details on how to contribute and create a registry.\n\n---\n\n### How It Works\n\n1. **Organize Your Schemas:**\n\n   - Place your files in versioned directories (e.g., `v1`, `v2`) and map them in `versions.json`. Refer to the [versions file structure](#versions-file-structure) for details on how to structure the `versions.json` file.\n\n2. **Build Schemas:**\n\n   - Schema Manager produces a `build` directory with all the schemas grouped into folders. Each folder contains the transformed schema with the appropriate namespaces and dependencies.\n\n3. **Automated Registration:**\n\n   - Schema Manager processes the `build` folder to register the schemas in the target Schema Registry in the correct order based on dependencies.\n\n## Scenario Example\n\nConsider a system managing a set of Protobuf schemas for an event-driven architecture. Each schema has multiple versions and dependencies.\n\nIf you don’t already have a project set up, you can create one with the following commands:\n\n```bash\nmkdir schema-manager-example\ncd schema-manager-example\nnpm init -y \u0026\u0026 npm install @charlescol/schema-manager\n```\n\nIn this example, we have two topics named \"topic1\" and \"topic2,\" as well as a common folder used for shared schemas.\n\nThe topics can contain multiple major versions of schema files. A JSON file named `versions.json` is used to manage them for each topic.\n\nIn our case, let's say that our schemas for `topic1` contain two major versions due to breaking changes in a file called `data.proto`:\n\n### File Structure\n\n```bash\nexample-schemas/\n  ├── topic1/\n  │   ├── v1/\n  │   │   ├── data.proto         # Schema for v1 data of topic1\n  │   │   └── model.proto        # Schema for v1 model (depends on topic1/v1/data.proto)\n  │   ├── v2/\n  │   │   └── data.proto         # Schema for v2 data (depends on ./common/v1/entity.proto)\n  │   └── versions.json          # Version mapping for topic1 (v1 and v2)\n  ├── topic2/\n  │   ├── v1/\n  │   │   └── data2.proto        # Schema for v1 data2 of topic2 (depends on ./common/v1/entity.proto)\n  │   └── versions.json          # Version mapping for topic2 (v1)\n  ├── common/\n  │   ├── v1/\n  │   │   └── entity.proto       # Schema for shared entity\n```\n\nIn the above structure, the folder `topic1/v2` contains only one file, even though there are three schemas in v2 (`model.proto` → `data.proto` → `entity.proto`).\n\nThis is because `versions.json` is used to resolve dependencies. If a file is the same across different versions, it doesn't need to be duplicated. The Schema Manager handles this implicit import mechanism to avoid schema duplication.\n\n### `versions.json` for `topic1`\n\n```json\n{\n  \"v1\": {\n    \"data\": \"v1/data.proto\",\n    \"model\": \"v1/model.proto\"\n  },\n  \"v2\": {\n    \"data\": \"v2/data.proto\",\n    \"model\": \"v1/model.proto\",\n    \"entity\": \"../common/v1/entity.proto\"\n  }\n}\n```\n\n### `versions.json` for `topic2`\n\n```json\n{\n  \"v1\": {\n    \"data2\": \"v1/data2.proto\",\n    \"entity\": \"../common/v1/entity.proto\"\n  }\n}\n```\n\nAdding a `versions.json` in the `common` directory is unnecessary because it is not meant to be registered independently.\n\n### Additional Remarks\n\n- Schema Manager supports any string for version names (e.g., `v1`, `v1.0`, `alpha`).\n- The schema name must be unique (case insensitive) across a given version of a topic.\n- Alternatively, a single `versions.json` file could be used to manage all the topics centrally.\n\n**Note:** Multiple examples, including those mentioned here with different schema types (Protobuf, Avro, etc.), can be found in the repository at `./examples`. Explore these to better understand schema management workflows.\n\n---\n\n### Example Schema Content\n\nHere is an example on how the model.proto should be structured. For more details on how to structure tour schema files by type, refer to the [Schema Content](#schema-content) section.\n\n#### `topic1/v1/model.proto`\n\n```protobuf\n// Information regarding the package, namespace, and import path will be completed automatically during the build process\nsyntax = \"proto3\";\n\n// Don't need to create a package.\n\n// Don't need to import the full path, only the filename is enough.\nimport \"data.proto\";\n\nmessage Model {\n  Data data = 1; // Don't need to reference the namespace, only the message name is enough.\n  string description = 2;\n  repeated string tags = 3;\n}\n```\n\n---\n\n### Building Schemas\n\nThe code below will generate a `build` directory with all the schemas grouped into folders, one per topic and version.\n\n```typescript\nimport { ConfigType, ConfluentRegistry, Manager } from '@charlescol/schema-manager';\nimport * as path from 'path';\n\n(async () =\u003e {\n  const SCHEMA_REGISTRY_URL = process.env.SCHEMA_REGISTRY_URL || 'http://localhost:8081';\n  const SCHEMA_DIR = path.resolve(__dirname, '../schemas');\n\n  // Create the schema registry\n  const registry = new ConfluentRegistry({\n    schemaRegistryUrl: SCHEMA_REGISTRY_URL,\n  });\n\n  // Create the manager\n  const manager = new Manager({\n    schemaRegistry: registry,\n    configType: ConfigType.PROTOBUF, // Specify the config type\n  });\n  await manager.build(SCHEMA_DIR); // Build the schemas\n})();\n```\n\n**Build Directory Structure:**\n\n```bash\nbuild/\n  ├── topic1/\n  │   ├── v1/\n  │   │   ├── data.proto\n  │   │   └── model.proto # Depends on data.proto\n  │   ├── v2/\n  │   │   └── data.proto # Depends on entity.proto\n  │   │   └── model.proto # Depends on data.proto\n  │   │   └── entity.proto\n  ├── topic2/\n  │   ├── v1/\n  │   │   └── data2.proto # Depends on entity.proto\n  │   │   └── entity.proto\n```\n\nEach generated file is now ready to be registered in the schema registry.\n\n#### Generated `topic1/v1/model.proto`\n\n```protobuf\nsyntax = \"proto3\";\n\npackage topic1.v1; // Resolved automatically\n\nimport \"topic1/v1/data.proto\"; // Resolved automatically\n\nmessage Model {\n  topic1.v1.Data data = 1; // Resolved automatically\n  string description = 2;\n  repeated string tags = 3;\n}\n```\n\nBy default, the namespace is generated by replacing `/` with `.` in the path. This can be customized by providing a custom namespace builder function to the manager.\n\n---\n\n### Publish to the Registry\n\nOnce the schemas are built, they can be deployed in the registry.\n\n```typescript\nfunction subjectBuilder(filepath: string): string {\n  const [topic, version, filename] = filepath.split('/'); // Extract topic and version\n  const filenameWithoutExt = filename?.split('.')[0] || ''; // Extract the filename without extension\n  return `${topic}.${filenameWithoutExt}.${version}`; // Return the constructed subject\n}\n\nawait manager.register(subjectBuilder); // Register the schemas\n```\n\nThe `subjectBuilder` function generates the subject name for each schema registered in the schema registry. The subject is a unique identifier used by the registry to track a schema. The above implementation is a custom strategy but you can also follow a `TopicNameStrategy` or a `RecordNameStrategy` to generate the subject name.\n\nThe `register` function builds a dependency graph based on schema dependencies and registers them in the correct order using a topological sorting algorithm.\n\n#### Example of Subject Names Generated by `subjectBuilder`\n\n- `topic1/v1/data.proto` → `topic1.data.v1`\n- `topic1/v1/model.proto` → `topic1.model.v1`\n- `topic1/v2/data.proto` → `topic1.data.v2`\n\n**After the registration process, a `subjects.txt` file is generated in the root directory to keep track of the subjects registered in the schema registry.** This file can be used to generate constants during code generation process.\n\n#### Example of Optimal Registration Order (Based on Dependencies)\n\n1. `topic1/v1/data.proto`\n2. `topic1/v2/entity.proto`\n3. `topic2/v1/entity.proto`\n4. `topic1/v2/data.proto`\n5. `topic2/v2/data2.proto`\n6. `topic1/v1/model.proto`\n7. `topic1/v2/model.proto`\n\n---\n\n### Full Code\n\nBelow is the complete code example for the steps above.\n\n#### `publish-schemas.ts`\n\n```typescript\nimport { ConfigType, ConfluentRegistry, Manager } from '@charlescol/schema-manager';\nimport * as path from 'path';\n\n(async () =\u003e {\n  const SCHEMA_REGISTRY_URL = process.env.SCHEMA_REGISTRY_URL || 'http://localhost:8081';\n  const SCHEMA_DIR = path.resolve(__dirname, '../schemas');\n\n  const registry = new ConfluentRegistry({\n    // Create the schema registry\n    schemaRegistryUrl: SCHEMA_REGISTRY_URL,\n  });\n\n  const manager = new Manager({\n    // Create the manager\n    schemaRegistry: registry,\n    configType: ConfigType.PROTOBUF,\n  });\n  await manager.build(SCHEMA_DIR); // Build the schemas\n  await manager.register(subjectBuilder); // register the schemas\n})();\n\n// Function to provide, used to build the subject for each schema file.\nfunction subjectBuilder(filepath: string): string {\n  const [topic, version, filename] = filepath.split('/');\n  const filenameWithoutExt = filename?.split('.')[0] || '';\n  return `${topic}.${filenameWithoutExt}.${version}`; // Return the constructed subject\n}\n```\n\nIf the file above is saved as `publish-schemas.ts`, you can run it with the following command to compile and execute it:\n\n```bash\ntsc \u0026\u0026 node dist/publish-schemas.js\n```\n\n---\n\n## Schema Content\n\nMany details are dynamically resolved during the build process using the `versions.json` file. This setup avoids code redundancy and enables dynamic schema transformations.\n\n### Protobuf\n\n- **Package Handling**: The package is automatically included during the build process and does not need to be specified in the schema.\n- **Imports**: Import statements should reference only the file name (key specified in the `versions.json` file + `.proto`) without the full path (e.g., `import \"entity.proto\";` ✅, not `import \"common/v1/entity.proto\";` ❌).\n- **External Imports**: External imports (e.g., `google/protobuf/timestamp.proto`) remain unchanged.\n- **Object Names**: Internal object references do not need to include the package name (e.g., `Entity` ✅, not `common.v1.Entity` ❌).\n\n```protobuf\n// topic1/v1/data.proto\nsyntax = \"proto3\";\n\n// No need to create a package.\n\nimport \"entity.proto\"; // Use only the file name\nimport \"google/protobuf/timestamp.proto\"; // External imports are unchanged\n\nmessage Data {\nstring additional_info = 1;\ngoogle.protobuf.Timestamp created_at = 2; // External imports are unchanged\nrepeated Entity entities = 3; // Internal object names exclude the package name\n}\n```\n\n---\n\n### Avro\n\n- **Namespace Handling**: Namespaces are automatically included during the build process and do not need to be specified in the schema.\n- **Object Names**: Object references do not need to include the namespace name (e.g., `Entity` ✅, not `common.v1.Entity` ❌).\n\n```json\n{\n  \"type\": \"record\",\n  \"name\": \"Model\",\n  \"fields\": [\n    {\n      \"name\": \"description\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"entities\",\n      \"type\": {\n        \"type\": \"array\",\n        \"items\": \"Entity\"\n      }\n    }\n  ]\n}\n```\n\n---\n\n### Json\n\n- **Id Handling**: Ids are automatically included (based on the relative file path) during the build process and do not need to be specified in the schema.\n- **Object Names**: Object references do not need to include the namespace name (e.g., `data` ✅, not `topic1.v2.data` ❌).\n\n```json\n{\n  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n  \"title\": \"model\",\n  \"description\": \"A model object\",\n  \"type\": \"object\",\n  \"properties\": {\n    \"data\": {\n      \"$ref\": \"data\"\n    },\n    \"description\": {\n      \"type\": \"string\"\n    },\n    \"tags\": {\n      \"type\": \"array\",\n      \"items\": {\n        \"type\": \"string\"\n      }\n    }\n  }\n}\n```\n\n---\n\n## Future Plans and Roadmap\n\nWe aim to extend Schema Manager with the following features:\n\n- **Support for Additional Schema Registries**: Expanding beyond Confluent Schema Registry to support other platforms.\n- **Expanded Format Support**: Adding parsers for more schema types beyond Protobuf, Avro, and JSON.\n- **Complex Transformations**: Enabling advanced schema transformations, such as adding options in Protobuf files to control future code generation.\n\n---\n\n## Example of Typical Integration with Schema Manager\n\nManaging schema files doesn’t need to be overengineered.\n\nJust keep your schemas in a Git repository (shared or dedicated),\ntrigger a CI/CD pipeline when changes are made (e.g., on tag or PR merge),\nand use Schema Manager to:\n\n- Validate and build your schema structure\n- Publish to your registry over HTTP\n- Generate and distribute code to target languages (e.g., Java, Rust, TypeScript)\n\n---\n\n### Benefits\n\n- **Simplified Management**: Centralized schemas reduce duplication and inconsistencies.\n- **Automation**: CI/CD integration ensures schema updates are handled automatically.\n- **Compatibility**: Versioned models guarantee consistency across services.\n\n---\n\n## Contributing\n\nWe welcome contributions! If you have suggestions or improvements, please open an issue or submit a pull request.\n\nRefer to the [Contributing Guide](CONTRIBUTING.md) and [How-To Documentation](how-to/overview.md) for details on contributing.\n\n---\n\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharlescol%2Fschema-manager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcharlescol%2Fschema-manager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharlescol%2Fschema-manager/lists"}