Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ccamel/go-graphql-subscription-example
☝️ go-graphql subscription over Kafka/Redis/NSQ example
https://github.com/ccamel/go-graphql-subscription-example
event-stream events go graphql graphql-subscriptions kafka kafka-consumer nsq prometheus reactivex redis websocket
Last synced: about 1 month ago
JSON representation
☝️ go-graphql subscription over Kafka/Redis/NSQ example
- Host: GitHub
- URL: https://github.com/ccamel/go-graphql-subscription-example
- Owner: ccamel
- License: mit
- Created: 2019-02-10T14:58:45.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2024-07-11T13:48:28.000Z (about 2 months ago)
- Last Synced: 2024-07-11T15:53:38.665Z (about 2 months ago)
- Topics: event-stream, events, go, graphql, graphql-subscriptions, kafka, kafka-consumer, nsq, prometheus, reactivex, redis, websocket
- Language: Go
- Homepage:
- Size: 1.11 MB
- Stars: 40
- Watchers: 3
- Forks: 6
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-ccamel - ccamel/go-graphql-subscription-example - ☝️ go-graphql subscription over Kafka/Redis/NSQ example (Go)
README
# go-graphql-subscription-example
[](https://github.com/ccamel/go-graphql-subscription-example/actions/workflows/lint.yml)
[](https://github.com/ccamel/go-graphql-subscription-example/actions/workflows/build.yml)
[](https://goreportcard.com/report/github.com/ccamel/go-graphql-subscription-example)
[](https://codeclimate.com/github/ccamel/go-graphql-subscription-example/maintainability)
[](https://sonarcloud.io/dashboard?id=ccamel_go-graphql-subscription-example)
[](https://sonarcloud.io/dashboard?id=ccamel_go-graphql-subscription-example)
[](https://stackshare.io/ccamel/go-graphql-subscription-example)
[](https://gitmoji.carloscuesta.me)
[](https://magefile.org)
[](https://github.com/ccamel/go-graphql-subscription-example/blob/master/LICENSE)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fccamel%2Fgo-graphql-subscription-example?ref=badge_shield)
> Project that demonstrates [graphQL] [subscriptions (over Websocket)](https://github.com/apollographql/subscriptions-transport-ws/blob/v0.9.4/PROTOCOL.md) to consume pre-configured topics from different kinds of
> stream sources like [Apache Kafka](https://kafka.apache.org/), [redis](https://redis.io/), [NSQ](https://nsq.io)...
## Purpose
This repository implements a simple service allowing clients to consume messages from a topics/channels through a [graphQL](https://graphql.org/) subscription endpoint.
This particular example demonstrates how to perform basic operations such as:
- serve a [graphiQL](https://github.com/graphql/graphiql) page
- expose a [Prometheus](https://prometheus.io/) endpoint
- implement a subscription resolver using WebSocket transport (compliant with [Apollo v0.9.16 protocol](https://github.com/apollographql/subscriptions-transport-ws/blob/v0.9.16/PROTOCOL.md))
- implement custom [graphQL] _scalars_
- consumer following kind of stream sources:
- [Apache Kafka](https://kafka.apache.org/) -
an open-source stream-processing software which aims to provide a unified,
high-throughput, low-latency platform for handling real-time data feeds.
- [Redis Streams](https://redis.io/) -
an open source, in-memory data structure store, message broker with [streaming](https://redis.io/topics/streams-intro) capabilities.
- [NSQ](https://nsq.io) -
a realtime distributed messaging platform designed to operate at scale.
- process messages using [reactive streams](http://reactivex.io/)
- filter messages using an expression evaluator
- ...
## Pre-requisites
**Requires Go 1.14.x** or above, which support Go modules. Read more about them [here](https://github.com/golang/go/wiki/Modules).
## Build
The project comes with a `magefile.go`, so all the main activities can be performed by [mage](https://magefile.org).
:warning: The source code provided is incomplete - build needs a code generation phase, especially for the embedding of the static resources.
To build the project, simply invoke the `build` target:
```sh
mage build
```
Alternately, the project can be build by [docker](https://www.docker.com/):
```sh
mage docker
```
Command will produce the image `ccamel/go-graphql-subscription-example`.
## How to play with it using Kafka?
### 1. Start Kafka server
At first, kafka must be started. See [official documentation](https://kafka.apache.org/quickstart) for more.
Kafka uses [ZooKeeper](https://zookeeper.apache.org/) so you need to first start a ZooKeeper server if you don't already have one.
```sh
> bin/zookeeper-server-start.sh config/zookeeper.properties
```
Now start the Kafka server:
```sh
> bin/kafka-server-start.sh config/server.properties
```
### 2. Create topics
For the purpose of the demo, some topics shall be created. So, let's create 2 topics named `topic-a` and `topic-b`,
with a single partition and only one replica:
```sh
> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic-a
> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic-b
```
### 3. Start the GraphQL server
The configuration is pretty straightforward:
```sh
> ./go-graphql-subscription-example --help
Usage:
go-graphql-subscription-example [flags]
Flags:
-h, --help help for go-graphql-subscription-example
--port uint16 The listening port (default 8000)
--source string The URI of the source to connect to
--topics strings The list of topics/stream names that subscribers can consume (default [foo])
```
Run the application which exposes the 2 previously created topics to subscribers:
```sh
> ./go-graphql-subscription-example --topics topic-a,topic-b
```
Alternately, if the docker image has been previously built, the container can be started this way:
```sh
> docker run -ti --rm -p 8000:8000 ccamel/go-graphql-subscription-example --topics topic-a,topic-b
```
### 4. Subscribe
The application exposes a graphql endpoint through which clients can receive messages coming from a kafka topic.
Navigate to `http://localhost:8000/graphiql` URL and submit the subscription to the topic `topic-a`.
```graphql
subscription {
event(on: "topic-a")
}
```
The offset id to consume from can also be specified. Negative values have a special meaning:
- `-1`: the most recent offset available for a partition (end)
- `-2`: the least recent offset available for a partition (beginning)
```graphql
subscription {
event(on: "topic-a", at: -1)
}
```
Additionally, a filter expression can be specified. The events consumed are then only ones matching the given predicate.
You can refer to [antonmedv/expr] for an overview of the syntax to use to write predicates.
```graphql
subscription {
event(
on: "topic-a",
at: -1,
matching: "value > 8"
)
}
```
### 5. Push messages
Run the producer and then type a few messages into the console to send to Kafka. Note that messages shall be
[JSON objects](https://www.json.org/).
```sh
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-a
{ "message": "hello world !", "value": 14 }
```
The message should be displayed on the browser.
## How to play with it using Redis?
⚠️ Redis implementation does not support offsets (i.e. the capability to resume at some point in time).
### 1. Start Redis
At first, a redis server (at least v5.0) must be started. See [official documentation](https://redis.io/download) for more.
### 2. Start the GraphQL server
Run the application which exposes the 2 previously created topics to subscribers:
```sh
> ./go-graphql-subscription-example --source redis://6379?name=foo --topics topic-a,topic-b
```
Alternately, if the docker image has been previously built, the container can be started this way:
```sh
> docker run -ti --rm -p 8000:8000 ccamel/go-graphql-subscription-example --source redis://6379?name=foo --topics topic-a,topic-b
```
### 3. Subscribe
The application exposes a graphql endpoint through which clients can receive messages coming from a redis stream.
Navigate to `http://localhost:8000/graphiql` URL and submit the subscription to the topic `topic-a`.
```graphql
subscription {
event(on: "topic-a")
}
```
Additionally, a filter expression can be specified. The events consumed are then only ones matching the given predicate.
You can refer to [antonmedv/expr] for an overview of the syntax to use to write predicates.
```graphql
subscription {
event(
on: "topic-a",
matching: "message contains \"hello\""
)
}
```
### 4. Push messages
Start the `redis-cli` and then use the `XADD` command to send the messages to the Redis stream.
```sh
> redis-cli
127.0.0.1:6379> XADD topic-a * message "hello world !" "value" "14"
```
The message should be displayed on the browser.
## How to play with it using NSQ?
⚠️ NSQ implementation does not support offsets (i.e. the capability to resume at some point in time).
### 1. Start NSQ
At first, NSQ must be started. See [official documentation](https://nsq.io/overview/quick_start.html) for more.
```sh
> nsqlookupd
> nsqd --lookupd-tcp-address=127.0.0.1:4160
> nsqadmin --lookupd-http-address=127.0.0.1:4161
```
### 2. Start the GraphQL server
Run the application which exposes the 2 previously created topics to subscribers:
```sh
> ./go-graphql-subscription-example --source nsq: --topics topic-a,topic-b
```
Alternately, if the docker image has been previously built, the container can be started this way:
```sh
> docker run -ti --rm -p 8000:8000 ccamel/go-graphql-subscription-example --source nsq: --topics topic-a,topic-b
```
### 3. Subscribe
The application exposes a graphql endpoint through which clients can receive messages coming from a redis stream.
Navigate to `http://localhost:8000/graphiql` URL and submit the subscription to the topic `topic-a`.
```graphql
subscription {
event(on: "topic-a")
}
```
Additionally, a filter expression can be specified. The events consumed are then only ones matching the given predicate.
You can refer to [antonmedv/expr] for an overview of the syntax to use to write predicates.
```graphql
subscription {
event(
on: "topic-a",
matching: "message contains \"hello\""
)
}
```
### 4. Push messages
Publish a message to the topic `topic-a` by using the command line below:
```sh
> curl -d '{ "message": "hello world !", "value": 14 }' 'http://127.0.0.1:4151/pub?topic=topic-a'
```
The message should be displayed on the browser.
## Stack
### Technical
This application mainly uses:
- **GraphQL**
↳ [graph-gophers/graphql-go](https://github.com/graph-gophers/graphql-go)
↳ [graph-gophers/graphql-transport-ws](https://github.com/graph-gophers/graphql-transport-ws)
↳ [graphql/graphiql](https://github.com/graphql/graphiql)
- **Prometheus**
↳ [prometheus/client_golang](https://github.com/prometheus/client_golang)
- **Kafka**
↳ [segment-integrations/connect-kafka](https://github.com/segment-integrations/connect-kafka)
- **Redis**
↳ [robinjoseph08/redisqueue](https://github.com/robinjoseph08/redisqueue)
- **NSQ**
↳ [nsqio/go-nsq](https://github.com/nsqio/go-nsq)
- **Expression language**
↳ [antonmedv/expr]
- **Design Patterns**
↳ [ReactiveX/RxGo v2](https://github.com/ReactiveX/RxGo/tree/v2.0.0)
- **CLI**
↳ [spf13/cobra](https://github.com/spf13/cobra)
- **Log**
↳ [rs/zerolog](https://github.com/rs/zerolog)
### Project
- **Build**
↳ [mage](https://magefile.org)
- **Linter**
↳ [golangci-lint](https://github.com/golangci/golangci-lint)
## License
[MIT] © [Chris Camel]
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fccamel%2Fgo-graphql-subscription-example?ref=badge_large)
[antonmedv/expr]: https://github.com/antonmedv/expr
[graphQL]: https://graphql.org/