https://github.com/amundsno/wikiproject-live-ranking

Proof-of-concept for real-time stream processing using Kafka, Python, and Server-Sent Events (SSE).
https://github.com/amundsno/wikiproject-live-ranking

docker kafka microservices python server-sent-events

Last synced: 4 months ago
JSON representation

Proof-of-concept for real-time stream processing using Kafka, Python, and Server-Sent Events (SSE).

• Host: GitHub
• URL: https://github.com/amundsno/wikiproject-live-ranking
• Owner: amundsno
• License: mit
• Created: 2025-03-24T17:20:24.000Z (7 months ago)
• Default Branch: main
• Last Pushed: 2025-03-24T17:49:24.000Z (7 months ago)
• Last Synced: 2025-06-04T21:55:00.564Z (5 months ago)
• Topics: docker, kafka, microservices, python, server-sent-events
• Language: Python
• Homepage:
• Size: 12.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# WikiProject Live Ranking

Real-time ranking of WikiProjects based on Wikipedia edit activity.

## Quick Start

```sh

git clone https://github.com/amundsno/wikiproject-live-ranking.git

cd wikiproject-live-ranking

docker compose up -d --build

```

![wikiproject-live-ranking](https://github.com/user-attachments/assets/15a6475d-c7e3-4b34-9216-711c433a369f)

## Project Overview

This demo project explores real-time processing of stream events using **Kafka** and **microservices in Python**. All services run as Docker containers:

1. **Kafka**  

   Single-node controller and broker in KRaft mode (no ZooKeeper).

2. `wikistream-subscriber`  

   Subscribes to [WikiMedia's public stream of recent change events](https://stream.wikimedia.org/v2/stream/recentchange), filters *real user edits* on *main articles* of the *English Wikipedia*, and sends them to the `wiki-stream` Kafka topic.

3. `wikistream-enricher`  

   Listens to `wiki-stream`, fetches related WikiProjects from Wikipedia's API, and forwards enriched events to the `wiki-stream-enriched` topic.

4. `sse-streamer`  

   Exposes events in the `wiki-stream-enriched` topic via **Server-Sent Events (SSE)** in real-time.

5. `nginx`  

   Serves the UI (frontend) and proxies SSE requests to `sse-streamer`.

### Tech Stack

- **Backend:** Kafka, Python

- **Frontend:** TypeScript, HTML/CSS, SSE (Server-Sent Events)

- **Proxy & Static Hosting:** Nginx

- **Infrastructure:** Docker

## Contributing

PRs and suggestions are welcome! Feel free to fork, modify, and improve this project (MIT license).