{"id":21663394,"url":"https://github.com/charlyjazz/chat-firebase-apache-camel","last_synced_at":"2025-04-11T23:43:08.655Z","repository":{"id":40379167,"uuid":"463338307","full_name":"CharlyJazz/Chat-Firebase-Apache-Camel","owner":"CharlyJazz","description":"Chat Message Monorepo Architecture Solution using Kafka, Terraform, GKE, GCP,  DockerHub,Kubernetes, Docker, Apache Camel, Python, FastAPI, Cassandra and Firestore, Next JS, React","archived":false,"fork":false,"pushed_at":"2023-12-27T23:39:08.000Z","size":906,"stargazers_count":21,"open_issues_count":7,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T19:40:05.702Z","etag":null,"topics":["aggregator","apache-camel","apache-kafka","api-rest","architecture-component","chat-application","docker","docker-compose","enterprise-architecture","fastapi","java","k8s","kubernetes","microservices","monorepo","openapi","python","react","rest","swagger"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CharlyJazz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-02-24T23:55:22.000Z","updated_at":"2024-11-29T14:58:19.000Z","dependencies_parsed_at":"2023-02-16T03:45:45.622Z","dependency_job_id":"f0f71c12-93f3-44ea-9f55-f68cbc1d1cd1","html_url":"https://github.com/CharlyJazz/Chat-Firebase-Apache-Camel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CharlyJazz%2FChat-Firebase-Apache-Camel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CharlyJazz%2FChat-Firebase-Apache-Camel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CharlyJazz%2FChat-Firebase-Apache-Camel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CharlyJazz%2FChat-Firebase-Apache-Camel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CharlyJazz","download_url":"https://codeload.github.com/CharlyJazz/Chat-Firebase-Apache-Camel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248497902,"owners_count":21113982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aggregator","apache-camel","apache-kafka","api-rest","architecture-component","chat-application","docker","docker-compose","enterprise-architecture","fastapi","java","k8s","kubernetes","microservices","monorepo","openapi","python","react","rest","swagger"],"created_at":"2024-11-25T10:21:35.813Z","updated_at":"2025-04-11T23:43:08.616Z","avatar_url":"https://github.com/CharlyJazz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Realtime Chat using Apache Camel, Cassandra and Firestore\n\nChat system architecture leveraging Apache Camel's aggregation pattern. This architecture incorporates a mechanism to buffer incoming messages for a specific timeout duration, measured in seconds. Subsequently, the system aggregates these buffered messages based on their unique chat IDs. The purpose of this aggregation step is to optimize Firestore usage by reducing the number of write operations. The aggregated message data is then efficiently dispatched to Firestore, with the primary goal of minimizing Firestore billing costs.\n\nThe chat pagination utilize Cassandra instead of Firestore. This strategic decision serves a dual purpose:\n\n- It alleviates the load on Firestore's read operations\n- Harnesses the inherent advantages of Cassandra for optimizing the partitioning using the `chat_id` as the partition key.\n\n[![Demo Video](https://github.com/CharlyJazz/Chat-Firebase-Apache-Camel/assets/12489333/79618746-367e-474c-bc16-07b2da7693dc)](https://user-images.githubusercontent.com/12489333/273462854-31d8e470-4169-4ac7-a397-327987f885f3.mov)\n\nThe next content of this paper explain the sistem using a Architectural Map Features approach.\n\n## Problem\n\nCreate a chat that makes spam difficult, optimizes availability, and manages reading and writing operations in Firestore to reduce expenses.\n\n## The Features are:\n\n1. Create User\n2. Login\n3. Get the list of users\n4. User send messages to other user\n5. Create a new chat room between to users: The user A will find the user B and start a chat.\n6. User get realtime message updating\n\n## Mapping Features to Architectural Components\n\n- Feature Mapping 1: The users management and authentication (features 1, 2 and 3) will be handle by a microservice called **Auth Microservice**\n  - The microservice will have a store.\n  - The microservice will use JWT for the Authentication.\n- Feature Mapping 2. Chat creation and Message creation/retrieving (features 4 and 5) will be handle by a microservice called **Chat Microservice**\n  - The microservice will have a store.\n- Feature Mapping 3. **Chat Microservice** will take care of sending the messages to a queue proccesing in other to satisfy the feature 6\n\n## Realtime solution to satisfy feature 6\n\nInstead of using Firestore or another cloud-based solution for a complete chat solution, we are going to use Firestore only to update the UI with the `aggregated news messages`.\n\nThe `aggregated news messages` refers to a set of messages sent by a user. Instead of displaying each message one by one, there will be a server that will group all the related messages sent by a user into a list. This list will then be retrieved for the chat..\n\n![Architecture diagram](https://user-images.githubusercontent.com/12489333/166072172-482250b2-93f7-4787-9652-3826054cc817.png)\n\nThis is how it looks like in Firestore:\n\n![Firestore Image of the messages](https://user-images.githubusercontent.com/12489333/269821581-987ba838-92c0-493a-bc08-5352205597cf.png)\n\n## Architectural Components to explicit implementation details\n\n- The auth and chat microservices will be written in Python using FastAPI.\n- The message aggregating process will be handled by Apache Camel using SpringBoot on Java.\n- User databases will be Postgres.\n- Chat database will be Cassandra to create a fast partition for each chat and fast pagination based on Time UUID.\n- The `aggregated news messages` will be saved into Firestore and only will retrieve for realtime\n- Chat messages queue processing will be handle by Kafka using a Python Kafka Consumer.\n- Auth validation in Python microservices will be handled by a Python shared library called `auth-validator`\n\n## Chat messages pagination and retrieving of the first page\n\nThe Chat will send a HTTP request to the Chat Microserver `/chat/\u003cid\u003e?time=\u003ctime-uuid\u003e\u0026quantity` to retrieve a set of messages\n\n- Query parameter **quatity** will be a integer to know the max size of messages to get\n- Query parameter **time** will be the UUID time of the last message that the chat have to get messages before that time\n- Both query parameter are optional. When there are not quantity you get 10. When there are not time you get the latest messages.\n\n## Development Enviroment Details\n\n- We going to use [tilt](https://tilt.dev/) in order to run the infraestructure on Kubernetes for development purposes.\n- There is a Docker Compose for development purposes.\n- For Python you need Pipenv and version 3.9.7\n- For Java you need:\n\n```\njava 17.0.2 2022-01-18 LTS\nJava(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)\nJava HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)\n```\n\n## Running the project using Docker Compose:\n\n```bash\ndocker-compose rm -svf \u0026\u0026 docker-compose up\n```\n\nIt should create 9 containers:\n\n```\n1.) chat-firebase-apache-camel_zookeeper_1\n2.) chat-firebase-apache-camel_cassandra_1\n3.) chat-firebase-apache-camel_auth_db_1\n4.) chat-firebase-apache-camel_kafka_1\n5.) chat-firebase-apache-camel_consumer_1\n6.) chat-firebase-apache-camel_auth_microservice_1\n7.) chat-firebase-apache-camel_apache_camel_microservices_1\n8.) chat-firebase-apache-camel_chat_microservices_1\n9.) front_end_next_js_1\n```\n\nThen you can open `http://localhost:3000/` and start using the chat.\n\n## Run tests in Apache Camel\n\n```bash\nmvn test -X\n```\n\n## How to run the microservices manually?\n\n- Chat Microservice:\n\n```bash\ncd chat-microservice \u0026\u0026 pipenv run server\n```\n\n- Auth Microservice:\n\n```bash\ncd auth-microservice \u0026\u0026 pipenv run server\n```\n\n- You will need Maven to run the Apache Camel Spring Boot server\n\n```bash\ncd apache-camel-service-bus \u0026\u0026 mvn package \u0026\u0026 mvn spring-boot:run\n```\n\n## Docker Development Worflow\n\n### Main command to run docker-compose\n\n```bash\ndocker-compose rm -svf \u0026\u0026 docker-compose up\n```\n\n### Migrate using Alembic in the Auth Microservice\n\nIn order to use the auth microservice you must migrate using alembic to create the user table. Access to the container and run:\n\n```bash\npipenv shell\nalembic upgrade head\n```\n\nIt should print something like:\n\n```bash\ndict_keys(['user'])\nINFO  [alembic.runtime.migration] Context impl PostgresqlImpl.\nINFO  [alembic.runtime.migration] Will assume transactional DDL.\nINFO  [alembic.runtime.migration] Running upgrade  -\u003e fe1a63894533, create user table\nINFO  [alembic.runtime.migration] Running upgrade fe1a63894533 -\u003e d3c6204b9f3c, username unique\n```\n\n### Run Kafka CLI:\n\nCreate a topic:\n\n```bash\ndocker exec -it CONTAINER_ID kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic charlytest\n```\n\nList of topics:\n\n```bash\ndocker exec -it CONTAINER_ID kafka-topics.sh --bootstrap-server localhost:9092 --list\n```\n\nGet topics information\n\n```bash\ndocker exec -it CONTAINER_ID kafka-topics.sh --bootstrap-server localhost:9092 --describe\n```\n\nConsole Producer using key and acks\n\nIn this case 12345 is the key and hello de message content (Useful to put all the messages in the same partition using the user_id)\n\n```bash\ndocker exec -it CONTAINER_ID kafka-console-producer.sh  --bootstrap-server localhost:9092 --topic charlytest --producer-property acks=all --property parse.key=true --property key.separator=:\n\n\u003e\u003e 12345:hello\n```\n\n### Access to Cassandra:\n\n```bash\ndocker exec -it CONTAINER_ID cqlsh\n```\n\n### Get the network information\n\n```bash\ndocker network ls | grep \"camel\"\n```\n\n### Firebase integration for Firestore.\n\nYou need to download the `service_account_key.json ` from your Firebase project add it to the `aggregated-messages-consumer` folder.\n\nAlso you need to update the .env file in the `frontend-dev` folder to looks like:\n\n```bash\n# MICROSERVICES\nNEXT_PUBLIC_AUTH_MICROSERVICE = 'http://0.0.0.0:8000'\nNEXT_PUBLIC_CHAT_MICROSERVICE = 'http://0.0.0.0:9000'\n# FIREBASE\nNEXT_PUBLIC_API_KEY=\"...\"\nNEXT_PUBLIC_AUTH_DOMAIN=\"...\"\nNEXT_PUBLIC_PROJECT_ID=\"...\"\nNEXT_PUBLIC_STORAGE_BUCKET=\"...\"\nNEXT_PUBLIC_MESSAGING_SENDER_ID=\"...\"\nNEXT_PUBLIC_APP_ID=\"...\"\n```\n\n### Github Actions secrets required\n\n- DOCKERHUB_TOKEN\n- DOCKERHUB_USERNAME\n- GCP_PROJECT\n- GKE_SA_KEY\n- GKE_ZONE\n- GKE_CLUSTER\n- GOOGLE_CREDENTIALS\n\n### Terraform\n\n- Create project\n- `gcloud auth application-default set-quota-project $PROJECT_NAME`\n- `gcloud config set project $PROJECT_NAME`\n- `terraform init`\n- Comment `gke_secrets.tf` and `kafka.tf`\n- `terraform apply`\n- `gcloud container clusters get-credentials chat2-405718-gke --region us-central1`\n- Set `gke_config_context` in `terraform.tfvars`\n- Uncomment `gke_secrets.tf` and `kafka.tf`\n- `terraform apply`\n- ./apply_k8s_configs.sh\n- Set in a .env `NEXT_PUBLIC_AUTH_MICROSERVICE` and `NEXT_PUBLIC_CHAT_MICROSERVICE` using the load balancing external IP\n- Go to firebase and create the project and a web app for this gcp project and fill the .env\n- Run ` kubectl create secret generic frontend-secrets --from-env-file=.env`\n- https://github.com/google-github-actions/auth/blob/main/docs/EXAMPLES.md to set GOOGLE_CREDENTIALS secret\n- GKE_SA_KEY gcloud iam service-accounts keys create key.json --iam-account=SA_EMAIL\n- export GKE_SA_KEY=$(cat key.json | base64)\n- use key.json in secret GKE_SA_KEY\n\n### License\n\nApache License 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharlyjazz%2Fchat-firebase-apache-camel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcharlyjazz%2Fchat-firebase-apache-camel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcharlyjazz%2Fchat-firebase-apache-camel/lists"}