{"id":20823373,"url":"https://github.com/snithish/advanced-databases_apachekafka","last_synced_at":"2026-03-19T17:41:52.073Z","repository":{"id":52299178,"uuid":"226839781","full_name":"snithish/advanced-databases_ApacheKafka","owner":"snithish","description":"Project to showcase Streaming Databases - Apache Kafka for Advanced Database (https://cs.ulb.ac.be/public/teaching/infoh415)","archived":false,"fork":false,"pushed_at":"2022-12-08T03:17:18.000Z","size":5157,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-26T02:03:08.758Z","etag":null,"topics":["kafka","kafka-sql","kafka-streams"],"latest_commit_sha":null,"homepage":null,"language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snithish.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-09T10:00:08.000Z","updated_at":"2019-12-11T19:00:33.000Z","dependencies_parsed_at":"2023-01-25T04:00:58.482Z","dependency_job_id":null,"html_url":"https://github.com/snithish/advanced-databases_ApacheKafka","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/snithish/advanced-databases_ApacheKafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snithish%2Fadvanced-databases_ApacheKafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snithish%2Fadvanced-databases_ApacheKafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snithish%2Fadvanced-databases_ApacheKafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snithish%2Fadvanced-databases_ApacheKafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snithish","download_url":"https://codeload.github.com/snithish/advanced-databases_ApacheKafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snithish%2Fadvanced-databases_ApacheKafka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29185344,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-07T00:44:15.062Z","status":"online","status_checked_at":"2026-02-07T02:00:07.217Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kafka","kafka-sql","kafka-streams"],"created_at":"2024-11-17T22:18:21.365Z","updated_at":"2026-02-07T03:30:54.115Z","avatar_url":"https://github.com/snithish.png","language":"PLpgSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stream Databases and Kafka\nThis repository includes artifacts required to compare an RDBMS like postgres with Kafka \nwe primarily look at the cost of adding triggers to achieve CEP like a Stream processing system does\nand analyze the cost to insertion speed and we then try to see how decoupling ingestion and processing is\nbeneficial with kafka as a storage system and KSQL (Kafka Streams) for processing. \nThanks for [meetup.com](https://www.meetup.com/) for providing a [Streaming API for RSVPs](http://stream.meetup.com/2/rsvps)\n\n## Data Generation\n1. We use Streaming API from [meetup.com](https://www.meetup.com/) for our experiment\n2. Clone this repository\n3. Run `save.py` to save data from API\n    ```shell script\n    cd datagen\n    pip3 install requests\n    python3 save.py\n    ```\n4. Once we have sufficient data we can transfer this to a GCS bucket with name `$BUCKET_NAME` \nfor use in benchmark\n\n## Bootstrap GCP VM\nPS. Ensure that the VM have sufficient cores and memory to run kafka and consumers\nin parallel eg. 15 cores 50GB memory\n\n1. Run\n    ```shell script\n    sudo apt update \u0026\u0026 sudo apt upgrade\n    ```\n2. Copy the data generated from previous step from GCS Bucket into VM\n    ```shell script\n    gsutil cp -R gs://$BUCKET_NAME .\n    ```\n3. Install `docker` as described here [Install Docker via Convenience Script](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-using-the-convenience-script)\n    ```shell script\n    curl -fsSL https://get.docker.com -o get-docker.sh\n    sudo sh get-docker.sh\n    sudo usermod -aG docker $USER\n    ```\n4. Install `docker-compose`\n    ```shell script\n    sudo curl -L \"https://github.com/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)\" -o /usr/local/bin/docker-compose\n    sudo chmod +x /usr/local/bin/docker-compose\n    ```\n5. Install pip3 to install dependencies\n    ```shell script\n    sudo apt install python3-pip\n    ```\n6. Clone this repository onto the VM\n    ```shell script\n    git clone https://github.com/snithish/advanced-databases_ApacheKafka.git\n    ```\n7. Start containers\n    ```shell script\n    cd advanced-databases_ApacheKafka\n    docker-compose up -d\n    ```\n8. **(Optionally)** Setup remote port forwarding to view the control center, to be run in work station\n    ensure that SSH has been setup up to connect to instances [GCP - Connecting to Instances](https://cloud.google.com/compute/docs/instances/connecting-advanced#thirdpartytools)\n    ```shell script\n    ssh -L 5000:localhost:9021 [USERNAME]@[EXTERNAL_IP_ADDRESS]\n    ```\n9. Move `datafile` to `datagen/data.txt`\n\n***\n   \n## Benchmarking Postgres\n1. Setup database\n   ```shell script\n   docker-compose exec postgres psql -Ubenchmark -f /datagen/initdb.sql\n   ```\n2. We need to install `psycopg2` to connect to postgres\n   ```shell script\n   pip3 install psycopg2-binary \n   ``` \n3. Run benchmark\n   ```shell script\n   python3 ingest_postgres.sql\n   ```\n4. Remove trigger from `initdb.sql` one at a time and repeat the steps above\n\n***\n\n## Benchmarking Kafka\n1. Create a new topic to inject data for benchmark\n    ```shell script\n    docker-compose exec broker kafka-topics --create \\\n      --zookeeper zookeeper:2181 \\\n      --replication-factor 1 --partitions 60 \\\n      --topic meetup\n    ```\n2. Open multi tabs or panes, suggested to use a terminal multiplexer like [TMux](https://github.com/tmux/tmux/wiki) or open multiple SSH connections and run\n    ```shell script\n    docker-compose exec ksqldb-cli ksql http://ksqldb-server:8088\n    ```\n3. Create `stream` from `kafka topic` Refer to: `kafka-processing.sql`\n4. Install `kafka-python` to act as producer for kafka\n    ```shell script\n    pip3 install kafka-python\n    ```\n5. Execute the other queries in multiple panes / windows\n6. Run benchmark\n    ```shell script\n    python3 ingest_kafka.py\n    ```\n7. Multiplex and run multiple producers to see high throughput\n8. Repeat steps 4 - 6 for various query combinations","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnithish%2Fadvanced-databases_apachekafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnithish%2Fadvanced-databases_apachekafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnithish%2Fadvanced-databases_apachekafka/lists"}