https://github.com/cloud8421/oredev-genstage-demo
Elixir GenStage demo application for Oredev 2017 conference
https://github.com/cloud8421/oredev-genstage-demo
elixir genstage oredev pubsub
Last synced: 1 day ago
JSON representation
Elixir GenStage demo application for Oredev 2017 conference
- Host: GitHub
- URL: https://github.com/cloud8421/oredev-genstage-demo
- Owner: cloud8421
- Created: 2017-10-28T11:05:04.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2017-11-07T07:59:50.000Z (over 8 years ago)
- Last Synced: 2025-06-13T21:47:11.393Z (about 1 year ago)
- Topics: elixir, genstage, oredev, pubsub
- Language: Elixir
- Size: 41 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Oredev
This application shows how [GenStage](https://github.com/elixir-lang/gen_stage) can be used to model consumption of events from CouchDB.
CouchDB exposes an http endpoint that returns all changes made to the database, 1 json line per event. This endpoint can be streamed to replay the entire database history (for more details see ). Any database change done while tailing the changes feed is instantly streamed to the consumer.
This application processes the CouchDB changes feed for a given database of Oredev sessions (see Setup - Step 1 to see how to create it) and calculates two schedules: one with events per day, another with events per topic. Schedules update almost instantly for each change done in CouchDB.
## Structure
The application contains two event processing implementations:
- A traditional pub sub implementation built on top of `Registry`. For each change captured by `Oredev.Changes.Feed`, an event is broadcasted asynchronously. Two subscribers, `Oredev.Subscriber.DailySchedule` and `Oredev.Subscriber.TopicSchedule` listen to these events to update their state.
- A `gen_stage` based implementation. Every time `Oredev.Changes.Feed` captures an event, it publishes it synchronously to a producer, which is consumed by `Oredev.Consumer.DailySchedule` and `Oredev.Consumer.TopicSchedule`.
## Key differences
- When using pub sub, the feed publishes as soon as it receives an event. Being pub sub completely asynchronous, the feed can ingest events without any concern for the ability of the rest of the system to process them.
- When using the producer, the feed blocks to publish the event. This means that unless the producer accepts the new event, the feed can't move on and process a new event from CouchDB. In other words, the speed of ingestion is directly influenced by the speed of processing.
## Setup
Assuming you have a working, local installation of CouchDB >= 2.0.
#### Step 1 - create and populate a CouchDB database
You can use the provided `make reset-db` task to create a `oredev` database populated with the schedule data for the conference (~150 sessions).
If you visit you should be able to see the list of changes applied to the newly created db (in form of ids and revisions).
#### Step 2 - dependencies
Assuming you have Elixir >= 1.5 installed, you can:
- `mix deps.get` to install dependencies
- `iex -S mix` to start the application
#### Step 3 - run the demo
You can use `Oredev.DbSupervisor.start_child("oredev")` to start processing data from the `oredev` database created in step 1.
Note that the application logs at `log/dev/{info,error}.log`, so make sure you tail them in a separate window to see if anything goes wrong.
If you see any `tcp` errors, you can: `Application.stop(:oredev); Application.ensure_all_started(:oredev); Oredev.DbSupervisor.start_child("oredev")` to stop the application and restart the process.
#### Step 4 - see how the system is affected
You can call `Oredev.Changes.Supervisor.healthcheck("oredev")` to see the pending message count for consumers and subscribers.