{"id":29551521,"url":"https://github.com/luca-02/ecommerce-kafka-analytics","last_synced_at":"2026-02-06T04:32:46.885Z","repository":{"id":303989866,"uuid":"1017412276","full_name":"Luca-02/ecommerce-kafka-analytics","owner":"Luca-02","description":"A distributed system for real-time e-commerce event analysis using Apache Kafka.","archived":false,"fork":false,"pushed_at":"2025-10-25T23:53:33.000Z","size":289,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-26T01:16:18.343Z","etag":null,"topics":["cloud-computing","docker","fault-tolerance","high-availability","k8s","kafka","microservices","python","tls"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Luca-02.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-07-10T13:47:24.000Z","updated_at":"2025-10-25T23:54:00.000Z","dependencies_parsed_at":"2025-07-10T20:59:07.526Z","dependency_job_id":"90b75493-6344-4376-b5cb-a215657a9165","html_url":"https://github.com/Luca-02/ecommerce-kafka-analytics","commit_stats":null,"previous_names":["luca-02/ecommerce-kafka-analytics"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Luca-02/ecommerce-kafka-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luca-02%2Fecommerce-kafka-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luca-02%2Fecommerce-kafka-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luca-02%2Fecommerce-kafka-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luca-02%2Fecommerce-kafka-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Luca-02","download_url":"https://codeload.github.com/Luca-02/ecommerce-kafka-analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Luca-02%2Fecommerce-kafka-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29150670,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T02:39:25.012Z","status":"ssl_error","status_checked_at":"2026-02-06T02:37:22.784Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-computing","docker","fault-tolerance","high-availability","k8s","kafka","microservices","python","tls"],"created_at":"2025-07-18T04:06:14.651Z","updated_at":"2026-02-06T04:32:46.879Z","avatar_url":"https://github.com/Luca-02.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# E-commerce Kafka Analytics System\n\nA distributed system for real-time e-commerce event analysis using Apache Kafka, developed for the Cloud Computing Technologies course (A.Y 2024/25).\n\n## 📋 Project Overview\n\nThis project implements an event-driven architecture for real-time e-commerce data analysis. The system simulates user behavior on an e-commerce platform, captures events through Apache Kafka, and processes them to generate analytics stored in Firebase Firestore.\n\n### System Architecture\n\nThe system is composed of three main components:\n\n1. **Event Simulator Service** - Simulates user behavior and generates events  \n2. **Apache Kafka Cluster** - Message broker for event management  \n3. **Analytics Service** - Processes events and generates statistics  \n\n## 🏗️ System Components\n\n### Event Simulator Service\n\nThe simulation service generates realistic e-commerce events:\n\n- **Supported Event Types**:\n  - `session_started` - User session start  \n  - `category_viewed` - Category viewed  \n  - `product_viewed` - Product viewed  \n  - `product_added_to_cart` - Product added to cart  \n  - `product_removed_from_cart` - Product removed from cart  \n  - `purchase` - Purchase completed  \n  - `session_ended` - User session end  \n\n- **Features**:\n  - Multi-process simulation for high concurrency  \n  - Synthetic data generation (users, products, categories)  \n  - Configurable probabilistic behaviors  \n  - Realistic user geolocation  \n\n### Apache Kafka Cluster\n\nSecure Kafka cluster with:\n\n- **Configuration**:\n  - 3 Kafka brokers for high availability  \n  - Topic partition replication  \n  - Replication factor: 3  \n  - Min ISR: 2  \n\n- **Security**:\n  - Encryption in Transit: SSL/TLS  \n  - Authentication: SASL/SCRAM-SHA-256  \n  - Authorization: Kafka ACLs for access control  \n  - Custom SSL certificates  \n\n### Analytics Service\n\nEvent processing service with:\n\n- **Processing**:\n  - Multi-threaded Kafka consumer  \n  - Event-type specific processing patterns  \n  - Real-time statistical aggregations  \n  - Concurrency management with worker pool  \n\n- **Storage**:\n  - Firebase Firestore for persistence  \n  - Collections organized by data type  \n  - Aggregations by country, date, product  \n  - Lifetime and temporal statistics  \n\n## 🚀 Deployment and Configuration\n\n### Prerequisites\n\n- Docker \u0026 Docker Compose  \n- Python 3.11+  \n- Firebase Project  \n- Make  \n\n### Initial Setup\n\n1. **Clone the repository**:\n    ```bash\n    git clone https://github.com/Luca-02/ecommerce-kafka-analytics.git\n    cd ecommerce-kafka-analytics\n    ```\n\n2. **Initialization**:\n    ```bash\n    make init\n    ```\n    This command:\n    - Creates Python virtual environments  \n    - Installs dependencies  \n    - Generates SSL certificates and Kafka configurations  \n\n3. **Firebase Configuration**:\n   - Place the `serviceAccountKey.json` file in `secret/`  \n   - Configure environment variables in the `.env` file  \n\n### Deployment Modes\n\n1. **Development (with Firebase Emulator)**\n    ```bash\n    # Start Firebase Emulator\n    make firebase-start\n    \n    # In another terminal\n    make compose-dev-up\n    ```\n\n2. **Full Production**\n    ```bash\n    make compose-up\n    ```\n\n3. **Kafka Cluster Only**\n    ```bash\n    make kafka-up\n    ```\n\n4. **Services Only (Producer/Consumer)**\n    ```bash\n    make services-up\n    ```\n\n### Kubernetes Deployment\n\nFor deployment in Kubernetes run:\n```bash\nbash ./scripts/k8s-start.sh\n```\n\n## 🧪 Testing and Security\n\n### Security Tests\n\nThe project includes \"attacker\" services to test security:\n```bash\nmake attacker-up\n```\n\nThis starts:\n- Producer with invalid certificates\n- Producer with incorrect credentials\n- Consumer with invalid certificates\n- Consumer with incorrect credentials\n\n### Monitoring\n\n- **Kafka UI**: Available at `http://localhost:8888`\n- **Firebase Emulator UI**: Available at `http://localhost:4000`\n- **Logs**: Available through `docker-compose logs`\n\n## 🔧 Technologies Used\n\n- **Python**: Main programming language\n- **Apache Kafka**: Message streaming platform\n- **Firebase Firestore**: NoSQL database for analytics\n- **Docker**: Containerization\n- **Kubernetes**: Orchestration\n- **Strimzi**: Kafka operator for Kubernetes","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluca-02%2Fecommerce-kafka-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluca-02%2Fecommerce-kafka-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluca-02%2Fecommerce-kafka-analytics/lists"}