https://github.com/eifinger/kafka-scheduler

A scalable, distributed Kafka message scheduler with only a Kafka cluster as a dependency
https://github.com/eifinger/kafka-scheduler

apache-kafka hacktoberfest hacktoberfest2023 java jobrunr kafka

Last synced: 3 months ago
JSON representation

A scalable, distributed Kafka message scheduler with only a Kafka cluster as a dependency

• Host: GitHub
• URL: https://github.com/eifinger/kafka-scheduler
• Owner: eifinger
• License: apache-2.0
• Created: 2021-01-17T19:52:18.000Z (over 4 years ago)
• Default Branch: main
• Last Pushed: 2023-01-29T16:32:20.000Z (over 2 years ago)
• Last Synced: 2024-12-24T08:22:56.611Z (4 months ago)
• Topics: apache-kafka, hacktoberfest, hacktoberfest2023, java, jobrunr, kafka
• Language: Java
• Homepage:
• Size: 327 KB
• Stars: 3
• Watchers: 3
• Forks: 1
• Open Issues: 3
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Codeowners: .github/CODEOWNERS

Awesome Lists containing this project

README

        
# kafka-scheduler

A scalable, distributed Kafka message scheduler with only a Kafka cluster as a dependency.

## Why?

Some applications or microservices run without any database or external dependency

only sending and receiving messages through Kafka.

If they have to schedule some tasks for example

* Send out a reminder email every day if a user has not verified their email

* Generate an export every 30 minutes

* ...

the application has to make sure that the task is executed exactly once.

Even if the application moves to another node on a Kubernetes cluster.

The usual way to handle persistent jobs would be to use Quartz, Hangfire or Jobrunr.

But all of them depend on a database.

Why should I have to introduce a database into my stack if I can "just" stay with Kafka?

## How?

Kafka-scheduler lets you schedule a Kafka message with a desired key, value and headers and

when you want to receive them on which topic. This way your application receives the message

as a trigger and can execute the business logic on the desired schedule.

### OneTimeCommand

A fire-and-forget command will be triggered at a certain point in the future.

```mermaid

sequenceDiagram

    autonumber

    note right of Application: Topic: one-time-commands
Headers
kafka-scheduler-key: myKey
kafka-scheduler-topic: mytopic
kafka-scheduler-when:2011-12-03T10:15:30+01:00

    Application->>Kafka-Scheduler: key:value

    Kafka-Scheduler->>Kafka-Scheduler: Schedule

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

```

#### Headers

|        Header         |                           Description                            |

|:---------------------:|:----------------------------------------------------------------:|

| kafka-scheduler-key   | The kafka message key used as a unique id so it can be cancelled |

| kafka-scheduler-value |         The Kafka message value you want to receive back         |

| kafka-scheduler-when  |                 A ISO-6801 configured Timestamp                  |

#### Key and Value

| Field |                           Description                            |

|-------|:----------------------------------------------------------------:|

| Key   | The kafka message key used as a unique id so it can be cancelled |

| Value |         The Kafka message value you want to receive back         |

### RecurringCommand

A command which will be triggered on a CRON schedule

```mermaid

sequenceDiagram

    autonumber

    note right of Application: Topic: recurring-commands
Headers
kafka-scheduler-key: myKey
kafka-scheduler-topic: mytopic
kafka-scheduler-cron: */5 * * * * *

    Application->>Kafka-Scheduler: key:value

    Kafka-Scheduler->>Kafka-Scheduler: Schedule

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

```

#### Headers

|        Header         |                           Description                            |

|:---------------------:|:----------------------------------------------------------------:|

| kafka-scheduler-key   | The kafka message key used as a unique id so it can be cancelled |

| kafka-scheduler-value |         The Kafka message value you want to receive back         |

| kafka-scheduler-cron  |        A cron expression with extended syntax support]           |

#### Key and Value

| Field |                           Description                            |

|-------|:----------------------------------------------------------------:|

| Key   | The kafka message key used as a unique id so it can be cancelled |

| Value |         The Kafka message value you want to receive back         |

### FixedRateCommand

A command which a defined duration between runs

```mermaid

sequenceDiagram

    autonumber

    note right of Application: Topic: fixed-rate-commands
Headers
kafka-scheduler-key: myKey
kafka-scheduler-topic: mytopic
kafka-scheduler-period: PT5S

    Application->>Kafka-Scheduler: key:value

    Kafka-Scheduler->>Kafka-Scheduler: Schedule

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

    note right of Application: Topic: mytopic

    Kafka-Scheduler->>Application: mykey:value

```

#### Headers

|         Header         |                           Description                            |

|:----------------------:|:----------------------------------------------------------------:|

| kafka-scheduler-key    | The kafka message key used as a unique id so it can be cancelled |

| kafka-scheduler-value  |         The Kafka message value you want to receive back         |

| kafka-scheduler-period |                       A ISO-8601 Duration                        |

#### Key and Value

| Field |                           Description                            |

|-------|:----------------------------------------------------------------:|

| Key   | The kafka message key used as a unique id so it can be cancelled |

| Value |         The Kafka message value you want to receive back         |

### Cancelling Commands

All commands can be cancelled by sending a Kafka tombstone message

with the same key which was used to schedule the command.

```mermaid

sequenceDiagram

    autonumber

    note right of Application: Topic: one-time-commands
Headers
kafka-scheduler-key: myKey
kafka-scheduler-topic: mytopic
kafka-scheduler-when:2011-12-03T10:15:30+01:00

    Application->>Kafka-Scheduler: key:value

    Kafka-Scheduler->>Kafka-Scheduler: Schedule

    note right of Application: Topic: one-time-commands
Headers

    Application->>Kafka-Scheduler: key:null

    Kafka-Scheduler->>Kafka-Scheduler: Cancel scheduled command

```

## Client SDK

TBD

## Detailed Documentation

TBD

# Other Kafka scheduler implementations

[kafka-message-scheduler](https://github.com/etf1/kafka-message-scheduler) -

Awesome implementation by [@etf](https://github.com/etf) in Go. Only supports OneTimeCommands but has

memory optimizations and an [Admin UI](https://github.com/etf1/kafka-message-scheduler-admin).

[high-available-task-scheduling](https://github.com/cbenaveen/high-available-task-scheduling) -

Implementation by [@cbenaveen](https://github.com/cbenaveen) using Kafka Streams.

# Acknowledgements

[@rdehuyss](https://github.com/rdehuyss) for creating [JobRunr](https://github.com/jobrunr/jobrunr)

the easiest to use scheduling library in the JVM space.