{"id":22725358,"url":"https://github.com/ovotech/castle","last_synced_at":"2025-05-16T07:07:02.793Z","repository":{"id":36468826,"uuid":"224130576","full_name":"ovotech/castle","owner":"ovotech","description":"Framework for building Kafka and avro based apps in typescript","archived":false,"fork":false,"pushed_at":"2025-03-08T08:21:22.000Z","size":1980,"stargazers_count":79,"open_issues_count":25,"forks_count":18,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-05-15T19:42:47.853Z","etag":null,"topics":["company-ovo"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovotech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-26T07:36:41.000Z","updated_at":"2025-05-01T18:45:21.000Z","dependencies_parsed_at":"2024-06-18T16:59:27.637Z","dependency_job_id":"985133ad-3aeb-42f9-a991-d7ae26efaed1","html_url":"https://github.com/ovotech/castle","commit_stats":{"total_commits":189,"total_committers":12,"mean_commits":15.75,"dds":"0.35449735449735453","last_synced_commit":"72f282c972026b3b9aac7fc41392a79bedaad77a"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcastle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcastle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcastle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovotech%2Fcastle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovotech","download_url":"https://codeload.github.com/ovotech/castle/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254485065,"owners_count":22078767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["company-ovo"],"created_at":"2024-12-10T16:09:25.801Z","updated_at":"2025-05-16T07:06:57.783Z","avatar_url":"https://github.com/ovotech.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Castle\n\nA framework around [kafka.js](https://github.com/tulios/kafkajs) to transparently use [Schema Registry](https://www.confluent.io/confluent-schema-registry/) and create an application that consumes, produces, and reacts to different kafka topics. Supports consumption in batches or in parallel. Statically define and verify the schemas / message types in TypeScript\n\nPackages:\n\n- [@ovotech/avro-kafkajs](packages/avro-kafkajs) - wrapper around [kafka.js](https://github.com/tulios/kafkajs) to use [Schema Registry](https://www.confluent.io/confluent-schema-registry/)\n- [@ovotech/avro-ts](packages/avro-ts) - Generate typescript from avro schemas. Without cli dependencies.\n- [@ovotech/avro-ts-cli](packages/avro-ts-cli) - A cli interface for `@ovotech/avro-ts`\n- [@ovotech/blaise](packages/blaise/README.md) | - [@ovotech/castle](https://npmjs.com/@ovotech/castle) combined with [@ovotech/avro-mock-generator](https://npmjs.com/@ovotech/avro-mock-generator)\n- [@ovotech/castle](packages/castle) - core service server\n- [@ovotech/castle-cli](packages/castle-cli) - cli for interacting with kafka and [Schema Registry](https://www.confluent.io/confluent-schema-registry/)\n- [@ovotech/castle-stream](packages/castle-stream) - A castle implementation using streams as input instead of kafka consumers. Useful backfillers and the like.\n\n## Usage\n\n```shell\nyarn add @ovotech/castle\n```\n\n\u003e [examples/simple.ts](packages/castle/examples/simple.ts)\n\n```typescript\nimport { createCastle, produce, consumeEachMessage } from '@ovotech/castle';\nimport { Event, EventSchema } from './avro';\n\n// Define producers as pure functions\n// With statically setting the typescript types and avro schemas\nconst mySender = produce\u003cEvent\u003e({ topic: 'my-topic-1', schema: EventSchema });\n\n// Define consumers as pure functions\n// With statically setting which types it will accept\nconst eachEvent = consumeEachMessage\u003cEvent\u003e(async ({ message }) =\u003e {\n  console.log(message.value);\n});\n\nconst main = async () =\u003e {\n  const castle = createCastle({\n    schemaRegistry: { uri: 'http://localhost:8081' },\n    kafka: { brokers: ['localhost:29092'] },\n    consumers: [{ topic: 'my-topic-1', groupId: 'my-group-1', eachMessage: eachEvent }],\n  });\n\n  // Start all consumers and producers\n  await castle.start();\n\n  await mySender(castle.producer, [{ value: { field1: 'my-string' } }]);\n};\n\nmain();\n```\n\nYou can connect to multiple topics, each of which is will have its own independent consumer group.\nYou can also pass SSL and SASL authentication, as well as authenticating the schema registry.\nMore about castle package in [packages/castle/README.md](packages/castle/README.md)\n\n## Command line usage\n\n```bash\nyarn global add @ovotech/castle-cli\n```\n\nYou can read about the various commands available with\n\n```bash\ncastle --help\n```\n\nThere are 4 main subcommand groups:\n\n- **castle topic**: Subcommands for searching and manipulating kafka topics, as well as producing and consuming events from them.\n- **castle schema**: Subcommands for getting schema versions of kafka topics.\n- **castle config**: Subcommands to create / edit connection configs to kafka brokers and schema registers, that can be used by other commands.\n- **castle group**: Subcommands to manipulate consumer group offsets.\n\nYou can configure access to the kafka to your server named `uat`, if you have the tls key, cert and certificate authority as text files. The schema registry is set as a url, any username and password can be set with a url provided auth like: `http://user:pass@localhost:8081`. The config file is saved to `$HOME/.castle-cli/` folder.\n\n```bash\ncastle config set uat --kafka-broker localhost:3203 --key private.pem --ca ca.pem --cert cert.pem --schema-registry http://localhost:8081\n```\n\nAfter that is set you can use it in any command by stating `--config uat` (or `-C uat`):\n\n```\ncastle schema search my-topic --config uat\ncastle schema show my-topic-full-name --config uat\ncastle topic search my-topic --config uat\ncastle topic consume my-topic-full-name --config uat\n```\n\nUsing it without a specified config would connect to the default local kafka server.\n\n## Castle also holds a collection of avro logical types, separate into descrete packages\n\n- [@ovotech/avro-timestamp-millis](packages/avro-timestamp-millis) - Date stored as the number of milliseconds since epoch\n- [@ovotech/avro-epoch-days](packages/avro-epoch-days) - Date stored as the number of days since epoch\n- [@ovotech/avro-decimal](packages/avro-decimal) - Decimal object value as raw bytes\n\n## Running the tests\n\nYou can run the tests with:\n\n```bash\nyarn test\n```\n\n### Coding style (linting, etc) tests\n\nStyle is maintained with prettier and eslint\n\n```\nyarn lint\n```\n\n## Deployment\n\nDeployment is preferment by lerna automatically on merge / push to main, but you'll need to bump the package version numbers yourself. Only updated packages with newer versions will be pushed to the npm registry.\n\n## Contributing\n\nHave a bug? File an issue with a simple example that reproduces this so we can take a look \u0026 confirm.\n\nWant to make a change? Submit a PR, explain why it's useful, and make sure you've updated the docs (this file) and the tests (see [test folder](test)).\n\n## License\n\nThis project is licensed under Apache 2 - see the [LICENSE](LICENSE) file for details\n\n## Why Castle?\n\nIt's Kafka's greatest work :)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fcastle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovotech%2Fcastle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovotech%2Fcastle/lists"}