Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/message-db/message-db

Microservice native message and event store for Postgres
https://github.com/message-db/message-db
event-driven event-sourcing event-store event-stream message-queue postgres pub-sub
Last synced: 30 days ago
JSON representation
Microservice native message and event store for Postgres
Host: GitHub
URL: https://github.com/message-db/message-db
Owner: message-db
License: mit
Created: 2019-12-06T01:17:38.000Z (almost 5 years ago)
Default Branch: master
Last Pushed: 2024-04-13T02:28:42.000Z (7 months ago)
Last Synced: 2024-10-01T21:24:03.126Z (about 1 month ago)
Topics: event-driven, event-sourcing, event-store, event-stream, message-queue, postgres, pub-sub
Language: Shell
Homepage: http://docs.eventide-project.org/user-guide/message-db/
Size: 260 KB
Stars: 1,620
Watchers: 26
Forks: 62
Open Issues: 22
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.md
- License: MIT-License.txt
Awesome Lists containing this project

awesome-list - message-db - db | 1345 | (Shell)
jimsghstars - message-db/message-db - Microservice native message and event store for Postgres (Shell)
README

        ![Message DB](http://docs.eventide-project.org/message-db-logo-90x105.png)

# Message DB

**Microservice Native Event Store and Message Store for Postgres**

A fully-featured event store and message store implemented in PostgreSQL for Pub/Sub, Event Sourcing, Messaging, and Evented Microservices applications.

## Features

- Pub/Sub

- JSON message data

- Event streams

- Stream categories

- Metadata

- Message queues

- Message storage

- Consumer groups

- Service host

- Administration tools

- Reports

## Rationale

An event sourcing and Pub/Sub message store built on Postgres for simple cloud or local hosting. A minimalist implementation of the essential features of tools like [Event Store](https://eventstore.org) or [Kafka](https://kafka.apache.org), with built-in support for messaging patterns like Pub/Sub, and consumer patterns like consumer groups.

Message DB was extracted from the [Eventide Project](http://docs.eventide-project.org) to make it easier for users to write clients in the language of their choosing.

## User Guide

A complete user guide is available on the Eventide Project docs site:

[http://docs.eventide-project.org/user-guide/message-db/](http://docs.eventide-project.org/user-guide/message-db/)

## Installation

Message DB can be installed either as a Ruby Gem, an NPM package, or can simply be cloned from this repository.

### Git Clone

``` bash

git clone [email protected]:message-db/message-db.git

```

### As a Ruby Gem

``` bash

gem install message-db

```

### As an NPM Module

``` bash

npm install @eventide/message-db

```

## Create the Postgres Database

Running the database installation script creates the database, schema, table, indexes, functions, views, types, a user role, and limit the user's privileges to the message store's public interface.

### Requirements

Make sure that your default Postgres user has administrative privileges.

### From the Git Clone

The installation script is in the `database` directory of the cloned repo. Change directory to the `message-db` directory where you cloned the repo, and run the script:

``` bash

database/install.sh

```

### From the Ruby Executable

If you installed Message DB via RubyGems, a database installation Ruby executable will be installed with the `message-db` gem.

The executable will be in the gem executable search path and may also be executed through bundler:

``` bash

bundle exec mdb-create-db

```

For more information about Ruby executables installed with the `message-db` Ruby Gem, see the Eventide docs on the administration tools that are bundled with the gem:

[http://docs.eventide-project.org/user-guide/message-db/tools.html](http://docs.eventide-project.org/user-guide/message-db/tools.html)

### From the NPM Module

The `message-db` NPM module doesn't ship with any special tooling other than the bundled scripts.

To execute the installation script, navigate to the directory where the `message-db` module is installed and run the script:

``` bash

install.sh

```

### Database Name

By default, the database creation tool will create a database named `message_store`.

If you prefer either a different database name, you can override the name using the `DATABASE_NAME` environment variable.

``` bash

DATABASE_NAME=some_other_database database/install.sh

```

### Uninstalling the Database

If you need to drop the database (for example, on a local dev machine):

``` bash

database/uninstall.sh

```

If you're upgrading a previous version of the database:

``` bash

database/update.sh

```

## API Overview

The message store provides an interface of Postgres server functions that can be used with any programming language or through the `psql` command line tool.

Interaction with the underlying store through the Postgres server functions ensures correct writing and reading messages, streams, and categories.

### Write a Message

Write a JSON-formatted message to a named stream, optionally specifying JSON-formatted metadata and an expected version number.

``` sql

write_message(

  id varchar,

  stream_name varchar,

  type varchar,

  data jsonb,

  metadata jsonb DEFAULT NULL,

  expected_version bigint DEFAULT NULL

)

```

#### Returns

Position of the message written.

#### Arguments

| Name | Description | Type | Default | Example |

| --- | --- | --- | --- | --- |

| id | UUID of the message being written | varchar | | a5eb2a97-84d9-4ccf-8a56-7160338b11e2 |

| stream_name | Name of stream to which the message is written | varchar | | someStream-123 |

| type | The type of the message | varchar | | Withdrawn |

| data | JSON representation of the message body | jsonb | | {"someAttribute": "some value"} |

| metadata (optional) | JSON representation of the message metadata | jsonb | NULL | {"metadataAttribute": "some meta data value"} |

| expected_version (optional) | Version that the stream is expected to be when the message is written | bigint | NULL | 11 |

#### Usage

``` sql

SELECT write_message('a11e9022-e741-4450-bf9c-c4cc5ddb6ea3', 'someStream-123', 'SomeMessageType', '{"someAttribute": "some value"}', '{"metadataAttribute": "some meta data value"}');

```

```

-[ RECORD 1 ]-+--

write_message | 0

```

Example: [https://github.com/message-db/message-db/blob/master/database/write-test-message.sh](https://github.com/message-db/message-db/blob/master/database/write-test-message.sh)

### Get Messages from a Stream

Retrieve messages from a single stream, optionally specifying the starting position, the number of messages to retrieve, and an additional condition that will be appended to the SQL command's WHERE clause.

``` sql

get_stream_messages(

  stream_name varchar,

  position bigint DEFAULT 0,

  batch_size bigint DEFAULT 1000,

  condition varchar DEFAULT NULL

)

```

#### Arguments

| Name | Description | Type | Default | Example |

| --- | --- | --- | --- | --- |

| stream_name | Name of stream to retrieve messages from | varchar | | someStream-123 |

| position (optional) | Starting position of the messages to retrieve | bigint | 0 | 11 |

| batch_size (optional) | Number of messages to retrieve | bigint | 1000 | 111 |

| condition (optional) | SQL condition to filter the batch by | varchar | NULL | messages.time >= current_time |

#### Usage

``` sql

SELECT * FROM get_stream_messages('someStream-123', 0, 1000, condition => 'messages.time >= current_time');

```

```

-[ RECORD 1 ]---+---------------------------------------------------------

id              | 4b96f09e-104a-4b1f-b198-5b3b46cf1d06

stream_name     | someStream-123

type            | SomeType

position        | 0

global_position | 1

data            | {"attribute": "some value"}

metadata        | {"metaAttribute": "some meta value"}

time            | 2019-11-24 17:56:09.71594

-[ RECORD 2 ]---+---------------------------------------------------------

id              | d94e79e3-cdda-49a3-9aad-ce5d70a5edd7

stream_name     | someStream-123

type            | SomeType

position        | 1

global_position | 2

data            | {"attribute": "some value"}

metadata        | {"metaAttribute": "some meta value"}

time            | 2019-11-24 17:56:09.75969

```

Example: [https://github.com/message-db/message-db/blob/master/test/get-stream-messages/get-stream-messages.sh](https://github.com/message-db/message-db/blob/master/test/get-stream-messages/get-stream-messages.sh)

### Get Messages from a Category

Retrieve messages from a category of streams, optionally specifying the starting position, the number of messages to retrieve, the correlation category for Pub/Sub, consumer group parameters, and an additional condition that will be appended to the SQL command's WHERE clause.

``` sql

CREATE OR REPLACE FUNCTION get_category_messages(

  category_name varchar,

  position bigint DEFAULT 0,

  batch_size bigint DEFAULT 1000,

  correlation varchar DEFAULT NULL,

  consumer_group_member bigint DEFAULT NULL,

  consumer_group_size bigint DEFAULT NULL,

  condition varchar DEFAULT NULL

)

```

#### Arguments

| Name | Description | Type | Default | Example |

| --- | --- | --- | --- | --- |

| category_name | Name of the category to retrieve messages from | varchar | | someCategory |

| position (optional) | Global position to start retrieving messages from | bigint | 1 | 11 |

| batch_size (optional) | Number of messages to retrieve | bigint | 1000 | 111 |

| correlation (optional) | Category or stream name recorded in message metadata's `correlationStreamName` attribute to filter the batch by | varchar | NULL | someCorrelationCategory |

| consumer_group_member (optional) | The zero-based member number of an individual consumer that is participating in a consumer group | bigint | NULL | 1 |

| consumer_group_size (optional) | The size of a group of consumers that are cooperatively processing a single category | bigint | NULL | 2 |

| condition (optional) | SQL condition to filter the batch by | varchar | NULL | messages.time >= current_time |

#### Usage

``` sql

SELECT * FROM get_category_messages('someCategory', 1, 1000, correlation => 'someCorrelationCategory', consumer_group_member => 1, consumer_group_size => 2, condition => 'messages.time >= current_time');

```

```

-[ RECORD 1 ]---+---------------------------------------------------------

id              | 28d8347f-677e-4738-b6b9-954f1b15463b

stream_name     | someCategory-123

type            | SomeType

position        | 0

global_position | 111

data            | {"attribute": "some value"}

metadata        | {"correlationStreamName": "someCorrelationCategory-123"}

time            | 2019-11-24 17:51:49.836341

-[ RECORD 2 ]---+---------------------------------------------------------

id              | 57894da7-680b-4483-825c-732dcf873e93

stream_name     | someCategory-456

type            | SomeType

position        | 1

global_position | 1111

data            | {"attribute": "some value"}

metadata        | {"correlationStreamName": "someCorrelationCategory-123"}

time            | 2019-11-24 17:51:49.879011

```

Note: Where `someStream-123` is a _stream name_, `someStream` is a _category_. Reading the `someStream` category retrieves messages from all streams whose names start with `someStream` and are followed by an ID, or where `someStream` is the whole stream name.

Example: [https://github.com/message-db/message-db/blob/master/test/get-category-messages/get-category-messages.sh](https://github.com/message-db/message-db/blob/master/test/get-category-messages/get-category-messages.sh)

### Full API Reference

- [write_message](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#write-a-message)

- [get_stream_messages](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-messages-from-a-stream)

- [get_category_messages](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-messages-from-a-category)

- [get_last_stream_message](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-last-message-from-a-stream)

- [stream_version](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-stream-version-from-a-stream)

- [id](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-the-id-from-a-stream-name)

- [cardinal_id](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-the-cardinal-id-from-a-stream-name)

- [category](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-the-category-from-a-stream-name)

- [is_category](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#determine-whether-a-stream-name-is-a-category)

- [acquire_lock](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#acquire-a-lock-for-a-stream-name)

- [hash_64](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#calculate-a-64-bit-hash-for-a-stream-name)

- [message_store_version](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-message-store-database-schema-version)

## Structure

The message store is a single table named `messages`.

## Messages Table

| Column | Description | Type | Default | Nullable |

| --- | --- | --- | --- | --- |

| id | Identifier of a message record | UUID | gen_random_uuid() | No |

| stream_name | Name of stream to which the message belongs | varchar | | No |

| type | The type of the message | varchar | | No |

| position | The ordinal position of the message in its stream. Position is gapless. | bigint | | No |

| global_position | Primary key. The ordinal position of the message in the entire message store. Global position may have gaps. | bigint | | No |

| data | Message payload | jsonb | NULL | Yes |

| metadata | Message metadata | jsonb | NULL | Yes |

| time | Timestamp when the message was written. The timestamp does not include a time zone. | timestamp | now() AT TIME ZONE 'utc' | No |

## Indexes

| Name | Columns | Unique | Note |

| --- | --- | --- | --- |

| messages_id | id | Yes | Enforce uniqueness as secondary key |

| messages_stream | stream_name, position | Yes | Ensures uniqueness of position number in a stream |

| messages_category | category(stream_name), global_position, category(metadata->>'correlationStreamName') | No | Used when retrieving by category name |

## Database

By default, the message store database is named `message_store`.

## Schema

All message store database objects are contained within a schema named `message_store`.

## User/Role

A role named `message_store` is created. The `message_store` role is given the `LOGIN` attribute, but no password is assigned. A password [can be assigned to the role](https://www.postgresql.org/docs/current/sql-alterrole.html), or the `message_store` role can be [granted to another Postgres user](https://www.postgresql.org/docs/current/role-membership.html).

## Source Code

View complete source code at:

[https://github.com/message-db/message-db/tree/master/database](https://github.com/message-db/message-db/tree/master/database)

## License

The Postgres Message Store is released under the [MIT License](https://github.com/message-db/message-db/blob/master/MIT-License.txt).