Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ovotech/castle

Framework for building Kafka and avro based apps in typescript
https://github.com/ovotech/castle

company-ovo

Last synced: 5 days ago
JSON representation

Framework for building Kafka and avro based apps in typescript

Awesome Lists containing this project

README 

        
# Castle



A framework around [kafka.js](https://github.com/tulios/kafkajs) to transparently use [Schema Registry](https://www.confluent.io/confluent-schema-registry/) and create an application that consumes, produces, and reacts to different kafka topics. Supports consumption in batches or in parallel. Statically define and verify the schemas / message types in TypeScript



Packages:



- [@ovotech/avro-kafkajs](packages/avro-kafkajs) - wrapper around [kafka.js](https://github.com/tulios/kafkajs) to use [Schema Registry](https://www.confluent.io/confluent-schema-registry/)

- [@ovotech/avro-ts](packages/avro-ts) - Generate typescript from avro schemas. Without cli dependencies.

- [@ovotech/avro-ts-cli](packages/avro-ts-cli) - A cli interface for `@ovotech/avro-ts`

- [@ovotech/blaise](packages/blaise/README.md) | - [@ovotech/castle](https://npmjs.com/@ovotech/castle) combined with [@ovotech/avro-mock-generator](https://npmjs.com/@ovotech/avro-mock-generator)

- [@ovotech/castle](packages/castle) - core service server

- [@ovotech/castle-cli](packages/castle-cli) - cli for interacting with kafka and [Schema Registry](https://www.confluent.io/confluent-schema-registry/)

- [@ovotech/castle-stream](packages/castle-stream) - A castle implementation using streams as input instead of kafka consumers. Useful backfillers and the like.



## Usage



```shell

yarn add @ovotech/castle

```



> [examples/simple.ts](packages/castle/examples/simple.ts)



```typescript

import { createCastle, produce, consumeEachMessage } from '@ovotech/castle';

import { Event, EventSchema } from './avro';



// Define producers as pure functions

// With statically setting the typescript types and avro schemas

const mySender = produce({ topic: 'my-topic-1', schema: EventSchema });



// Define consumers as pure functions

// With statically setting which types it will accept

const eachEvent = consumeEachMessage(async ({ message }) => {

  console.log(message.value);

});



const main = async () => {

  const castle = createCastle({

    schemaRegistry: { uri: 'http://localhost:8081' },

    kafka: { brokers: ['localhost:29092'] },

    consumers: [{ topic: 'my-topic-1', groupId: 'my-group-1', eachMessage: eachEvent }],

  });



  // Start all consumers and producers

  await castle.start();



  await mySender(castle.producer, [{ value: { field1: 'my-string' } }]);

};



main();

```



You can connect to multiple topics, each of which is will have its own independent consumer group.

You can also pass SSL and SASL authentication, as well as authenticating the schema registry.

More about castle package in [packages/castle/README.md](packages/castle/README.md)



## Command line usage



```bash

yarn global add @ovotech/castle-cli

```



You can read about the various commands available with



```bash

castle --help

```



There are 4 main subcommand groups:



- **castle topic**: Subcommands for searching and manipulating kafka topics, as well as producing and consuming events from them.

- **castle schema**: Subcommands for getting schema versions of kafka topics.

- **castle config**: Subcommands to create / edit connection configs to kafka brokers and schema registers, that can be used by other commands.

- **castle group**: Subcommands to manipulate consumer group offsets.



You can configure access to the kafka to your server named `uat`, if you have the tls key, cert and certificate authority as text files. The schema registry is set as a url, any username and password can be set with a url provided auth like: `http://user:pass@localhost:8081`. The config file is saved to `$HOME/.castle-cli/` folder.



```bash

castle config set uat --kafka-broker localhost:3203 --key private.pem --ca ca.pem --cert cert.pem --schema-registry http://localhost:8081

```



After that is set you can use it in any command by stating `--config uat` (or `-C uat`):



```

castle schema search my-topic --config uat

castle schema show my-topic-full-name --config uat

castle topic search my-topic --config uat

castle topic consume my-topic-full-name --config uat

```



Using it without a specified config would connect to the default local kafka server.



## Castle also holds a collection of avro logical types, separate into descrete packages



- [@ovotech/avro-timestamp-millis](packages/avro-timestamp-millis) - Date stored as the number of milliseconds since epoch

- [@ovotech/avro-epoch-days](packages/avro-epoch-days) - Date stored as the number of days since epoch

- [@ovotech/avro-decimal](packages/avro-decimal) - Decimal object value as raw bytes



## Running the tests



You can run the tests with:



```bash

yarn test

```



### Coding style (linting, etc) tests



Style is maintained with prettier and eslint



```

yarn lint

```



## Deployment



Deployment is preferment by lerna automatically on merge / push to main, but you'll need to bump the package version numbers yourself. Only updated packages with newer versions will be pushed to the npm registry.



## Contributing



Have a bug? File an issue with a simple example that reproduces this so we can take a look & confirm.



Want to make a change? Submit a PR, explain why it's useful, and make sure you've updated the docs (this file) and the tests (see [test folder](test)).



## License



This project is licensed under Apache 2 - see the [LICENSE](LICENSE) file for details



## Why Castle?



It's Kafka's greatest work :)