{"id":25365745,"url":"https://github.com/fernando24164/real-time-ingestion","last_synced_at":"2026-05-19T04:13:43.951Z","repository":{"id":277046809,"uuid":"931156492","full_name":"fernando24164/real-time-ingestion","owner":"fernando24164","description":"Real time ingestion with FastAPI","archived":false,"fork":false,"pushed_at":"2025-06-28T17:34:36.000Z","size":1927,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-28T18:37:30.178Z","etag":null,"topics":["analytics","data-ingestion","fastapi","insights","microservices","personalization","python","real-time","rest-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fernando24164.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-11T20:04:52.000Z","updated_at":"2025-06-28T17:34:39.000Z","dependencies_parsed_at":null,"dependency_job_id":"191554dd-01d5-4756-a013-fd4687747ce8","html_url":"https://github.com/fernando24164/real-time-ingestion","commit_stats":null,"previous_names":["fernando24164/real-time-ingestion"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/fernando24164/real-time-ingestion","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fernando24164%2Freal-time-ingestion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fernando24164%2Freal-time-ingestion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fernando24164%2Freal-time-ingestion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fernando24164%2Freal-time-ingestion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fernando24164","download_url":"https://codeload.github.com/fernando24164/real-time-ingestion/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fernando24164%2Freal-time-ingestion/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265815944,"owners_count":23833045,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data-ingestion","fastapi","insights","microservices","personalization","python","real-time","rest-api"],"created_at":"2025-02-14T23:49:43.484Z","updated_at":"2026-05-19T04:13:43.942Z","avatar_url":"https://github.com/fernando24164.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"media/real-time-logo.png\" alt=\"Real Time Ingestion Logo\" width=\"300\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/fernando24164/real-time-ingestion/actions/workflows/run-tests.yml\"\u003e\n    \u003cimg src=\"https://github.com/fernando24164/real-time-ingestion/actions/workflows/run-tests.yml/badge.svg\" alt=\"Tests Status\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.python.org/downloads/release/python-3130/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/python-3.13-blue.svg\" alt=\"Python 3.13\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://fastapi.tiangolo.com/\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/FastAPI-0.121.1-009688.svg\" alt=\"FastAPI\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# 🎮 Real-Time Game Store Ingestion API\n\n**Real-Time Game Store Ingestion API** how to handle **real-time data ingestion** using **FastAPI**, **PostgreSQL**, and **Redis**, while providing a robust REST API for managing and analyzing customer interactions in a game store.\n\n## Architecture\n![Architecture](media/game-store-c4.png)\n\n---\n\n## 🛠️ Tech Stack\n\n- **Backend**: [FastAPI](https://fastapi.tiangolo.com/) - High-performance Python web framework.\n- **Database**: [PostgreSQL](https://www.postgresql.org/) - Relational database for persistent storage.\n- **Cache**: [Redis](https://redis.io/) - In-memory data store for fast access to recent views.\n- **Monitoring**: [Prometheus](https://prometheus.io/) \u0026 [Grafana](https://grafana.com/) - Metrics collection and visualization.\n- **Containerization**: [Docker](https://www.docker.com/) - Simplified deployment and scaling.\n\n---\n\n## 🛠️ Event Ingestion Strategy\n\n### Strategy Overview\n\nThe **Real-Time Game Store Ingestion API** employs a **hybrid ingestion strategy** to handle customer interaction events efficiently. This strategy combines **Redis** for real-time caching and **PostgreSQL** for persistent storage.\n\n### How It Works\n\n1. **Redis for Real-Time Caching**:\n   - Events such as \"VIEW,\" \"ADD_TO_CART,\" and \"PURCHASE\" are immediately pushed to Redis.\n   - Redis stores the **last viewed games** for each user in a **list data structure**, enabling quick retrieval for personalized recommendations.\n   - The list is trimmed to a fixed size (e.g., 10 items) to optimize memory usage.\n\n2. **PostgreSQL for Persistent Storage**:\n   - All events are asynchronously written to PostgreSQL for long-term storage and analytics.\n   - This ensures that historical data is available for generating insights, such as user preferences, engagement scores, and platform usage.\n\n3. **Background Task Processing**:\n   - Event ingestion is queued as a background task, allowing the API to respond quickly to clients while processing continues asynchronously.\n   - This approach minimizes latency and ensures a smooth user experience.\n\n4. **Monitoring and Metrics**:\n   - Prometheus tracks ingestion performance, including request rates, error rates, and processing times.\n   - Grafana visualizes these metrics, helping to identify bottlenecks and optimize the system.\n\n---\n\n### Why This Strategy?\n\nThis strategy is designed to balance **speed**, **scalability**, and **data integrity**:\n\n- **Speed**: Redis provides low-latency access to frequently accessed data, such as the last viewed games.\n- **Scalability**: By offloading real-time data to Redis and using PostgreSQL for analytics, the system can handle high event volumes without overwhelming the database.\n- **Data Integrity**: PostgreSQL ensures that all events are stored reliably, even if Redis data is evicted due to memory constraints.\n\n---\n\n### Alternative Strategies\n\nIf your use case differs, consider these alternatives:\n\n1. **Event Streaming**:\n   - Stream events to multiple consumers.\n   - Ideal for systems requiring **real-time analytics** or **multi-service event processing**.\n   - Trade-off: Higher complexity and operational overhead.\n\n2. **In-Memory Only**:\n   - Store all events in-memory for ultra-low latency.\n   - Suitable for **ephemeral data** or **short-lived sessions**.\n   - Trade-off: No persistent storage; data loss on server restarts.\n\n3. **Direct Database Writes**:\n   - Write events directly to PostgreSQL or another relational database.\n   - Simplifies architecture but may introduce **write bottlenecks** under high load.\n\n4. **NoSQL Databases**:\n   - Use a NoSQL database for flexible schema and high write throughput.\n   - Suitable for systems with **unstructured data** or **horizontal scaling needs**.\n   - Trade-off: May lack advanced querying capabilities for relational data.\n\n---\n\n## 📂 Project Structure\n\n```\nreal-time-ingestion/\n├── app/\n│   ├── api/                # API endpoints and routing\n│   ├── core/               # Core configurations and utilities\n│   ├── db/                 # Database and Redis connection setup\n│   ├── models/             # SQLAlchemy models for the database\n│   ├── schemas/            # Pydantic schemas for request/response validation\n│   ├── services/           # Business logic and service layer\n│   ├── metrics.py          # Prometheus metrics integration\n│   └── main.py             # FastAPI application entry point\n├── config/                 # Configuration files for Prometheus and Grafana\n├── scripts/                # Database seeding scripts\n├── tests/                  # Unit and integration tests\n├── docker-compose.yml      # Docker Compose configuration\n├── pyproject.toml          # Python dependencies\n└── README.md               # Project documentation\n```\n\n---\n\n## 🛠️ Installation\n\n### Prerequisites\n\n- Python 3.9+\n- Docker \u0026 Docker Compose\n\n### Steps\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/fernando24164/real-time-ingestion.git\n   cd real-time-ingestion\n   ```\n\n2. **Install Dependencies**:\n   ```bash\n   uv sync --locked\n   ```\n\n3. **Run with Docker**:\n   ```bash\n   docker-compose up -d\n   ```\n\n4. **Seed the Database**:\n   ```bash\n   python scripts/seed_database.py\n   ```\n\n5. **Access the API**:\n   - Swagger UI: [http://localhost:8000/docs](http://localhost:8000/docs)\n   - Prometheus Metrics: [http://localhost:8001/metrics](http://localhost:8001/metrics)\n   - Grafana Dashboard: [http://localhost:3000](http://localhost:3000) (Default login: `admin/admin`)\n\n---\n\n## 📊 Monitoring \u0026 Metrics\n\n- **Prometheus**: Collects API metrics like request count, duration, and error rates.\n- **Grafana**: Visualize metrics with pre-configured dashboards.\n\n### Example Metrics\n\n- **Requests per Minute by Endpoint**\n- **95th Percentile Response Time**\n- **Error Rate**\n- **Average Active Requests**\n\n---\n\n## 🔬 Testing\n\nRun the test suite using `pytest`:\n\n```bash\npytest tests/\n```\n\n---\n\n## 📚 API Endpoints\n\n### Ingestion API\n- **POST** `/api/v1/ingest`: Ingest customer interaction data.\n\n### Customer Insights\n- **GET** `/api/v1/customer/insights`: Generate personalized insights for a user.\n\n---\n\n## 🛠️ Manage Database Migrations\n\nUse Alembic for database migrations:\n\n1. **Initialize Migrations**:\n   ```bash\n   alembic init --template async migrations\n   ```\n\n2. **Create a New Migration**:\n   ```bash\n   alembic revision --autogenerate -m \"Add new feature\"\n   ```\n\n3. **Apply Migrations**:\n   ```bash\n   alembic upgrade head\n   ```\n\n4. **Rollback Migrations**:\n   ```bash\n   alembic downgrade -1\n   ```\n\n---\n\n## 📈 Data Model\n\n![Data Model](media/data-model.svg)\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! Please follow these steps:\n\n1. Fork the repository.\n2. Create a new branch (`feature/my-feature`).\n3. Commit your changes.\n4. Push to the branch.\n5. Open a pull request.\n\n---\n\n## 📄 License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n---\n\n## 🌟 Acknowledgments\n\n- Special thanks to the FastAPI, PostgreSQL, and Redis communities.\n- [Tiangolo FastAPI template](https://github.com/fastapi/full-stack-fastapi-template)\n- [Real world example in FastAPI deprecated but good to take a look](https://github.com/nsidnev/fastapi-realworld-example-app)\n\n---\n\n## 📬 Contact\n\nFor questions or feedback, feel free to reach out:\n\n- **GitHub**: [fernando24164](https://github.com/fernando24164)\n\n---\n\n⭐ If you find this project helpful, please give it a star ⭐\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffernando24164%2Freal-time-ingestion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffernando24164%2Freal-time-ingestion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffernando24164%2Freal-time-ingestion/lists"}