{"id":26515424,"url":"https://github.com/mana9512/twitterflow","last_synced_at":"2026-04-12T13:46:00.064Z","repository":{"id":281287426,"uuid":"944822729","full_name":"mana9512/TwitterFlow","owner":"mana9512","description":"Real-Time Twitter Data Processing and Insights ","archived":false,"fork":false,"pushed_at":"2025-03-19T19:28:07.000Z","size":133,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-19T20:32:05.497Z","etag":null,"topics":["elasticsearch","kafka","spring","springcloud"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mana9512.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-08T03:19:53.000Z","updated_at":"2025-03-19T19:28:11.000Z","dependencies_parsed_at":"2025-03-08T04:20:26.526Z","dependency_job_id":null,"html_url":"https://github.com/mana9512/TwitterFlow","commit_stats":null,"previous_names":["mana9512/twitterflow"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mana9512%2FTwitterFlow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mana9512%2FTwitterFlow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mana9512%2FTwitterFlow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mana9512%2FTwitterFlow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mana9512","download_url":"https://codeload.github.com/mana9512/TwitterFlow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244745751,"owners_count":20503050,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","kafka","spring","springcloud"],"created_at":"2025-03-21T06:15:45.628Z","updated_at":"2026-04-12T13:46:00.017Z","avatar_url":"https://github.com/mana9512.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TwitterFlow\n## Microservices Architecture Project\nThis project is a microservices architecture built with Java 17, Spring Boot, Kafka, ElasticSearch, and Docker. It streams data from Twitter, processes it via Kafka, stores it in ElasticSearch and PostgreSQL, and exposes query services via REST APIs.\n\n\n\n## Overview\nThis project is a microservices architecture built with Java 11, Spring Boot, Kafka, ElasticSearch, and Docker. It streams data from Twitter, processes it via Kafka, stores it in ElasticSearch and PostgreSQL, and exposes query services via REST APIs.\n\n## Key Technologies\n- **Java 11**\n- **Spring Boot**\n- **Kafka** (for streaming data)\n- **ElasticSearch** (for data indexing and querying)\n- **PostgreSQL** (for relational data storage)\n- **Docker \u0026 Docker Compose** (for containerization)\n- **Keycloak** (for authentication and authorization)\n- **Prometheus \u0026 Grafana** (for monitoring)\n- **ELK Stack** (for log aggregation)\n- **Eureka \u0026 Spring Cloud Gateway** (for service discovery and API gateway)\n\n## Architecture Diagram\n![project_architecture](https://github.com/user-attachments/assets/100322bd-13f9-4dfa-8069-15e37cb976fb)\n\n\n## Microservices Overview\n1. **Twitter to Kafka Service**\n   - Streams real-time data from Twitter using Twitter4J.\n   - Publishes data to Kafka topics.\n\n2. **Kafka to ElasticSearch Service**\n   - Consumes Kafka data and indexes it into ElasticSearch.\n\n3. **Elastic Query Service**\n   - Provides REST APIs to query data from ElasticSearch.\n\n4. **Kafka Streams Service**\n   - Processes streaming data and publishes results to a new Kafka topic.\n\n5. **Analytics Service**\n   - Consumes processed data and stores it in PostgreSQL.\n\n6. **Elastic Query Web Client**\n   - A simple front-end for querying data.\n\n## Getting Started\n### Prerequisites\n- Java 11\n- Docker \u0026 Docker Compose\n- Maven\n\n### Running Services\n1. **Start Dependencies**\n```bash\ndocker-compose up -d\n```\n\n2. **Build and Run Services**\n```bash\nmvn clean install\njava -jar target/\u003cservice-name\u003e.jar\n```\n\n### Access Services\n- **Kafka UI**: `http://localhost:9000`\n- **ElasticSearch**: `http://localhost:9200`\n- **Grafana**: `http://localhost:3000`\n\n## Security\n- Authentication and authorization are managed via Keycloak using OAuth2 and OpenID Connect.\n\n## Monitoring \u0026 Logging\n- Metrics are captured using Prometheus and visualized in Grafana.\n- Logs are aggregated using ELK Stack (ElasticSearch, Logstash, Kibana).\n\n## API Documentation\n- Swagger UI available at: `http://localhost:\u003cport\u003e/swagger-ui/index.html`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmana9512%2Ftwitterflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmana9512%2Ftwitterflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmana9512%2Ftwitterflow/lists"}