https://github.com/gazette/core

Build platforms that flexibly mix SQL, batch, and stream processing paradigms
https://github.com/gazette/core

brokers event-sourcing golang stream-processing streaming-platforms

Last synced: about 1 month ago
JSON representation

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

• Host: GitHub
• URL: https://github.com/gazette/core
• Owner: gazette
• License: mit
• Created: 2017-10-20T18:54:20.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2024-02-06T12:49:33.000Z (5 months ago)
• Last Synced: 2024-02-06T14:22:36.216Z (5 months ago)
• Topics: brokers, event-sourcing, golang, stream-processing, streaming-platforms
• Language: Go
• Homepage: https://gazette.dev
• Size: 63.9 MB
• Stars: 504
• Watchers: 33
• Forks: 49
• Open Issues: 25
• Metadata Files:
  • Readme: README.rst
  • Changelog: CHANGELOG.rst
  • Contributing: CONTRIBUTING.md
  • License: LICENSE

Lists

• awesome-streaming - Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms. (Table of Contents / Data Pipeline)
• data-engineering-collection - Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms. (Data Ingestion)
• awesome-repositories - gazette/core - Build platforms that flexibly mix SQL, batch, and stream processing paradigms (Go)
• awesome-repositories - gazette/core - Build platforms that flexibly mix SQL, batch, and stream processing paradigms (Go)
• awesome-bigdata - Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms. (Data Ingestion)
• awesome-bigdata - Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms. (Data Ingestion)
• awesome-bigdata - Gazette - Distributed streaming infrastructure built on cloud storage which makes it easy to mix and match batch and streaming paradigms. (Data Ingestion)

README

        

.. image:: docs/_static/logo_with_text.svg

   :alt: Gazette Logo

.. image:: https://github.com/gazette/core/workflows/Gazette%20Continuous%20Integration/badge.svg

   :target: https://github.com/gazette/core/actions?query=workflow%3A%22Gazette+Continuous+Integration%22

   :alt: Gazette Continuous Integration

.. image:: https://godoc.org/go.gazette.dev/core?status.svg

   :target: https://godoc.org/go.gazette.dev/core

   :alt: GoDoc

.. image:: https://img.shields.io/badge/slack-@gazette/dev-yellow.svg?logo=slack

   :target: https://join.slack.com/t/gazette-dev/shared_invite/enQtNjQxMzgyNTEzNzk1LTU0ZjZlZmY5ODdkOTEzZDQzZWU5OTk3ZTgyNjY1ZDE1M2U1ZTViMWQxMThiMjU1N2MwOTlhMmVjYjEzMjEwMGQ

   :alt: Slack

.. image:: https://goreportcard.com/badge/github.com/gazette/core

   :target: https://goreportcard.com/report/github.com/gazette/core

   :alt: Go Report Card

Overview

=========

Gazette makes it easy to build platforms that flexibly mix *SQL*, *batch*,

and *millisecond-latency streaming* processing paradigms. It enables teams,

applications, and analysts to work from a common catalog of data in the way

that's most convenient **to them**. Gazette's core abstraction is a "journal"

-- a streaming append log that's represented using regular files in a BLOB

store (i.e., S3).

The magic of this representation is that journals are simultaneously a

low-latency data stream *and* a collection of immutable, organized files

in cloud storage (aka, a data lake) -- a collection which can be directly

integrated into familiar processing tools and SQL engines.

Atop the journal *broker service*, Gazette offers a powerful *consumers

framework* for building streaming applications in Go. Gazette has served

production use cases for nearly five years, with deployments scaled to

millions of streamed records per second.

Where to Start

===============

- Official documentation_, with tutorials and examples.

- API Godocs_.

.. _documentation: https://gazette.dev

.. _Godocs: https://godoc.org/go.gazette.dev/core