{"id":13528224,"url":"https://github.com/message-db/message-db","last_synced_at":"2025-05-14T13:06:02.050Z","repository":{"id":41249724,"uuid":"226219329","full_name":"message-db/message-db","owner":"message-db","description":"Microservice native message and event store for Postgres","archived":false,"fork":false,"pushed_at":"2024-04-13T02:28:42.000Z","size":266,"stargazers_count":1636,"open_issues_count":23,"forks_count":63,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-04-11T06:01:48.708Z","etag":null,"topics":["event-driven","event-sourcing","event-store","event-stream","message-queue","postgres","pub-sub"],"latest_commit_sha":null,"homepage":"http://docs.eventide-project.org/user-guide/message-db/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/message-db.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"MIT-License.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-06T01:17:38.000Z","updated_at":"2025-04-10T03:26:08.000Z","dependencies_parsed_at":"2024-01-10T20:11:20.679Z","dependency_job_id":"15856fab-d214-4198-8434-b251927215d9","html_url":"https://github.com/message-db/message-db","commit_stats":{"total_commits":375,"total_committers":11,"mean_commits":34.09090909090909,"dds":0.07199999999999995,"last_synced_commit":"25da82b044f94416202ac3daa1866791b385badc"},"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/message-db%2Fmessage-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/message-db%2Fmessage-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/message-db%2Fmessage-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/message-db%2Fmessage-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/message-db","download_url":"https://codeload.github.com/message-db/message-db/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253683339,"owners_count":21947118,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-driven","event-sourcing","event-store","event-stream","message-queue","postgres","pub-sub"],"created_at":"2024-08-01T06:02:20.089Z","updated_at":"2025-05-14T13:06:02.012Z","avatar_url":"https://github.com/message-db.png","language":"Shell","funding_links":[],"categories":["Shell","Libraries and Frameworks"],"sub_categories":["Databases"],"readme":"![Message DB](http://docs.eventide-project.org/message-db-logo-90x105.png)\n\n# Message DB\n\n**Microservice Native Event Store and Message Store for Postgres**\n\nA fully-featured event store and message store implemented in PostgreSQL for Pub/Sub, Event Sourcing, Messaging, and Evented Microservices applications.\n\n## Features\n\n- Pub/Sub\n- JSON message data\n- Event streams\n- Stream categories\n- Metadata\n- Message queues\n- Message storage\n- Consumer groups\n- Service host\n- Administration tools\n- Reports\n\n## Rationale\n\nAn event sourcing and Pub/Sub message store built on Postgres for simple cloud or local hosting. A minimalist implementation of the essential features of tools like [Event Store](https://eventstore.org) or [Kafka](https://kafka.apache.org), with built-in support for messaging patterns like Pub/Sub, and consumer patterns like consumer groups.\n\nMessage DB was extracted from the [Eventide Project](http://docs.eventide-project.org) to make it easier for users to write clients in the language of their choosing.\n\n## User Guide\n\nA complete user guide is available on the Eventide Project docs site:\n\n[http://docs.eventide-project.org/user-guide/message-db/](http://docs.eventide-project.org/user-guide/message-db/)\n\n## Installation\n\nMessage DB can be installed either as a Ruby Gem, an NPM package, or can simply be cloned from this repository.\n\n### Git Clone\n\n``` bash\ngit clone git@github.com:message-db/message-db.git\n```\n\n### As a Ruby Gem\n\n``` bash\ngem install message-db\n```\n\n### As an NPM Module\n\n``` bash\nnpm install @eventide/message-db\n```\n\n## Create the Postgres Database\n\nRunning the database installation script creates the database, schema, table, indexes, functions, views, types, a user role, and limit the user's privileges to the message store's public interface.\n\n### Requirements\n\nMake sure that your default Postgres user has administrative privileges.\n\n### From the Git Clone\n\nThe installation script is in the `database` directory of the cloned repo. Change directory to the `message-db` directory where you cloned the repo, and run the script:\n\n``` bash\ndatabase/install.sh\n```\n\n### From the Ruby Executable\n\nIf you installed Message DB via RubyGems, a database installation Ruby executable will be installed with the `message-db` gem.\n\nThe executable will be in the gem executable search path and may also be executed through bundler:\n\n``` bash\nbundle exec mdb-create-db\n```\n\nFor more information about Ruby executables installed with the `message-db` Ruby Gem, see the Eventide docs on the administration tools that are bundled with the gem:\n\n[http://docs.eventide-project.org/user-guide/message-db/tools.html](http://docs.eventide-project.org/user-guide/message-db/tools.html)\n\n### From the NPM Module\n\nThe `message-db` NPM module doesn't ship with any special tooling other than the bundled scripts.\n\nTo execute the installation script, navigate to the directory where the `message-db` module is installed and run the script:\n\n``` bash\ninstall.sh\n```\n\n### Database Name\n\nBy default, the database creation tool will create a database named `message_store`.\n\nIf you prefer either a different database name, you can override the name using the `DATABASE_NAME` environment variable.\n\n``` bash\nDATABASE_NAME=some_other_database database/install.sh\n```\n\n### Uninstalling the Database\n\nIf you need to drop the database (for example, on a local dev machine):\n\n``` bash\ndatabase/uninstall.sh\n```\n\nIf you're upgrading a previous version of the database:\n\n``` bash\ndatabase/update.sh\n```\n\n## API Overview\n\nThe message store provides an interface of Postgres server functions that can be used with any programming language or through the `psql` command line tool.\n\nInteraction with the underlying store through the Postgres server functions ensures correct writing and reading messages, streams, and categories.\n\n### Write a Message\n\nWrite a JSON-formatted message to a named stream, optionally specifying JSON-formatted metadata and an expected version number.\n\n``` sql\nwrite_message(\n  id varchar,\n  stream_name varchar,\n  type varchar,\n  data jsonb,\n  metadata jsonb DEFAULT NULL,\n  expected_version bigint DEFAULT NULL\n)\n```\n\n#### Returns\n\nPosition of the message written.\n\n#### Arguments\n\n| Name | Description | Type | Default | Example |\n| --- | --- | --- | --- | --- |\n| id | UUID of the message being written | varchar | | a5eb2a97-84d9-4ccf-8a56-7160338b11e2 |\n| stream_name | Name of stream to which the message is written | varchar | | someStream-123 |\n| type | The type of the message | varchar | | Withdrawn |\n| data | JSON representation of the message body | jsonb | | {\"someAttribute\": \"some value\"} |\n| metadata (optional) | JSON representation of the message metadata | jsonb | NULL | {\"metadataAttribute\": \"some meta data value\"} |\n| expected_version (optional) | Version that the stream is expected to be when the message is written | bigint | NULL | 11 |\n\n#### Usage\n\n``` sql\nSELECT write_message('a11e9022-e741-4450-bf9c-c4cc5ddb6ea3', 'someStream-123', 'SomeMessageType', '{\"someAttribute\": \"some value\"}', '{\"metadataAttribute\": \"some meta data value\"}');\n```\n\n```\n-[ RECORD 1 ]-+--\nwrite_message | 0\n```\n\nExample: [https://github.com/message-db/message-db/blob/master/database/write-test-message.sh](https://github.com/message-db/message-db/blob/master/database/write-test-message.sh)\n\n### Get Messages from a Stream\n\nRetrieve messages from a single stream, optionally specifying the starting position, the number of messages to retrieve, and an additional condition that will be appended to the SQL command's WHERE clause.\n\n``` sql\nget_stream_messages(\n  stream_name varchar,\n  position bigint DEFAULT 0,\n  batch_size bigint DEFAULT 1000,\n  condition varchar DEFAULT NULL\n)\n```\n\n#### Arguments\n\n| Name | Description | Type | Default | Example |\n| --- | --- | --- | --- | --- |\n| stream_name | Name of stream to retrieve messages from | varchar | | someStream-123 |\n| position (optional) | Starting position of the messages to retrieve | bigint | 0 | 11 |\n| batch_size (optional) | Number of messages to retrieve | bigint | 1000 | 111 |\n| condition (optional) | SQL condition to filter the batch by | varchar | NULL | messages.time \u003e= current_time |\n\n#### Usage\n\n``` sql\nSELECT * FROM get_stream_messages('someStream-123', 0, 1000, condition =\u003e 'messages.time \u003e= current_time');\n```\n\n```\n-[ RECORD 1 ]---+---------------------------------------------------------\nid              | 4b96f09e-104a-4b1f-b198-5b3b46cf1d06\nstream_name     | someStream-123\ntype            | SomeType\nposition        | 0\nglobal_position | 1\ndata            | {\"attribute\": \"some value\"}\nmetadata        | {\"metaAttribute\": \"some meta value\"}\ntime            | 2019-11-24 17:56:09.71594\n-[ RECORD 2 ]---+---------------------------------------------------------\nid              | d94e79e3-cdda-49a3-9aad-ce5d70a5edd7\nstream_name     | someStream-123\ntype            | SomeType\nposition        | 1\nglobal_position | 2\ndata            | {\"attribute\": \"some value\"}\nmetadata        | {\"metaAttribute\": \"some meta value\"}\ntime            | 2019-11-24 17:56:09.75969\n```\n\nExample: [https://github.com/message-db/message-db/blob/master/test/get-stream-messages/get-stream-messages.sh](https://github.com/message-db/message-db/blob/master/test/get-stream-messages/get-stream-messages.sh)\n\n### Get Messages from a Category\n\nRetrieve messages from a category of streams, optionally specifying the starting position, the number of messages to retrieve, the correlation category for Pub/Sub, consumer group parameters, and an additional condition that will be appended to the SQL command's WHERE clause.\n\n``` sql\nCREATE OR REPLACE FUNCTION get_category_messages(\n  category_name varchar,\n  position bigint DEFAULT 0,\n  batch_size bigint DEFAULT 1000,\n  correlation varchar DEFAULT NULL,\n  consumer_group_member bigint DEFAULT NULL,\n  consumer_group_size bigint DEFAULT NULL,\n  condition varchar DEFAULT NULL\n)\n```\n\n#### Arguments\n\n| Name | Description | Type | Default | Example |\n| --- | --- | --- | --- | --- |\n| category_name | Name of the category to retrieve messages from | varchar | | someCategory |\n| position (optional) | Global position to start retrieving messages from | bigint | 1 | 11 |\n| batch_size (optional) | Number of messages to retrieve | bigint | 1000 | 111 |\n| correlation (optional) | Category or stream name recorded in message metadata's `correlationStreamName` attribute to filter the batch by | varchar | NULL | someCorrelationCategory |\n| consumer_group_member (optional) | The zero-based member number of an individual consumer that is participating in a consumer group | bigint | NULL | 1 |\n| consumer_group_size (optional) | The size of a group of consumers that are cooperatively processing a single category | bigint | NULL | 2 |\n| condition (optional) | SQL condition to filter the batch by | varchar | NULL | messages.time \u003e= current_time |\n\n#### Usage\n\n``` sql\nSELECT * FROM get_category_messages('someCategory', 1, 1000, correlation =\u003e 'someCorrelationCategory', consumer_group_member =\u003e 1, consumer_group_size =\u003e 2, condition =\u003e 'messages.time \u003e= current_time');\n```\n\n```\n-[ RECORD 1 ]---+---------------------------------------------------------\nid              | 28d8347f-677e-4738-b6b9-954f1b15463b\nstream_name     | someCategory-123\ntype            | SomeType\nposition        | 0\nglobal_position | 111\ndata            | {\"attribute\": \"some value\"}\nmetadata        | {\"correlationStreamName\": \"someCorrelationCategory-123\"}\ntime            | 2019-11-24 17:51:49.836341\n-[ RECORD 2 ]---+---------------------------------------------------------\nid              | 57894da7-680b-4483-825c-732dcf873e93\nstream_name     | someCategory-456\ntype            | SomeType\nposition        | 1\nglobal_position | 1111\ndata            | {\"attribute\": \"some value\"}\nmetadata        | {\"correlationStreamName\": \"someCorrelationCategory-123\"}\ntime            | 2019-11-24 17:51:49.879011\n```\n\nNote: Where `someStream-123` is a _stream name_, `someStream` is a _category_. Reading the `someStream` category retrieves messages from all streams whose names start with `someStream` and are followed by an ID, or where `someStream` is the whole stream name.\n\nExample: [https://github.com/message-db/message-db/blob/master/test/get-category-messages/get-category-messages.sh](https://github.com/message-db/message-db/blob/master/test/get-category-messages/get-category-messages.sh)\n\n### Full API Reference\n\n- [write_message](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#write-a-message)\n- [get_stream_messages](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-messages-from-a-stream)\n- [get_category_messages](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-messages-from-a-category)\n- [get_last_stream_message](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-last-message-from-a-stream)\n- [stream_version](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-stream-version-from-a-stream)\n- [id](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-the-id-from-a-stream-name)\n- [cardinal_id](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-the-cardinal-id-from-a-stream-name)\n- [category](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-the-category-from-a-stream-name)\n- [is_category](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#determine-whether-a-stream-name-is-a-category)\n- [acquire_lock](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#acquire-a-lock-for-a-stream-name)\n- [hash_64](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#calculate-a-64-bit-hash-for-a-stream-name)\n- [message_store_version](http://docs.eventide-project.org/user-guide/message-db/server-functions.html#get-message-store-database-schema-version)\n\n## Structure\n\nThe message store is a single table named `messages`.\n\n## Messages Table\n\n| Column | Description | Type | Default | Nullable |\n| --- | --- | --- | --- | --- |\n| id | Identifier of a message record | UUID | gen_random_uuid() | No |\n| stream_name | Name of stream to which the message belongs | varchar | | No |\n| type | The type of the message | varchar | | No |\n| position | The ordinal position of the message in its stream. Position is gapless. | bigint | | No |\n| global_position | Primary key. The ordinal position of the message in the entire message store. Global position may have gaps. | bigint | | No |\n| data | Message payload | jsonb | NULL | Yes |\n| metadata | Message metadata | jsonb | NULL | Yes |\n| time | Timestamp when the message was written. The timestamp does not include a time zone. | timestamp | now() AT TIME ZONE 'utc' | No |\n\n## Indexes\n\n| Name | Columns | Unique | Note |\n| --- | --- | --- | --- |\n| messages_id | id | Yes | Enforce uniqueness as secondary key |\n| messages_stream | stream_name, position | Yes | Ensures uniqueness of position number in a stream |\n| messages_category | category(stream_name), global_position, category(metadata-\u003e\u003e'correlationStreamName') | No | Used when retrieving by category name |\n\n## Database\n\nBy default, the message store database is named `message_store`.\n\n## Schema\n\nAll message store database objects are contained within a schema named `message_store`.\n\n## User/Role\n\nA role named `message_store` is created. The `message_store` role is given the `LOGIN` attribute, but no password is assigned. A password [can be assigned to the role](https://www.postgresql.org/docs/current/sql-alterrole.html), or the `message_store` role can be [granted to another Postgres user](https://www.postgresql.org/docs/current/role-membership.html).\n\n## Source Code\n\nView complete source code at:\n\n[https://github.com/message-db/message-db/tree/master/database](https://github.com/message-db/message-db/tree/master/database)\n\n## License\n\nThe Postgres Message Store is released under the [MIT License](https://github.com/message-db/message-db/blob/master/MIT-License.txt).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmessage-db%2Fmessage-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmessage-db%2Fmessage-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmessage-db%2Fmessage-db/lists"}