{"id":13340486,"url":"https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project","last_synced_at":"2025-03-11T18:31:19.745Z","repository":{"id":174297502,"uuid":"652036835","full_name":"hanydief/Kafka_Producer_Consumer_to_SQL_Project","owner":"hanydief","description":"Serverless Data Platform: Creating a Kafka producer.py, consumer.py \u0026 PgAdmin SQL DB","archived":false,"fork":false,"pushed_at":"2023-06-10T22:42:06.000Z","size":819,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-10-24T06:27:47.525Z","etag":null,"topics":["gitbash","pgadmin4","upstash","visual-studio"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hanydief.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-06-10T22:07:49.000Z","updated_at":"2023-12-05T09:42:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"2bdb0fe8-534c-435e-9e84-e9141f834303","html_url":"https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project","commit_stats":null,"previous_names":["hanydief/kafka_producer_consumer_to_sql_project"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanydief%2FKafka_Producer_Consumer_to_SQL_Project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanydief%2FKafka_Producer_Consumer_to_SQL_Project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanydief%2FKafka_Producer_Consumer_to_SQL_Project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hanydief%2FKafka_Producer_Consumer_to_SQL_Project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hanydief","download_url":"https://codeload.github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243090296,"owners_count":20234762,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gitbash","pgadmin4","upstash","visual-studio"],"created_at":"2024-07-29T19:23:00.776Z","updated_at":"2025-03-11T18:31:19.713Z","avatar_url":"https://github.com/hanydief.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka_Producer_Consumer_to_SQL_Project\nCreating a Kafka producer.py, consumer.py \u0026amp; PgAdmin SQL DB\n\n# Required installations if not existing on the same environment:\n- pip install kafka-python\n- conda install -c conda-forge kafka-python \n- conda install -c anaconda sqlalchemy\n- conda install -c anaconda psycopg2\n\n# What the project about:\n- Created Upstash account\n- Created New single replica Cluster\n- producer.py Python base code \n\n![KafkaProducer_upstash](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/KafkaProducer_upstash.png)\n\n- On VS created a producer.py to publish random message to Kafka every 5 seconds\n\n![kafkaProducer_VS](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/kafkaProducer_VS.png)\n\n- consumer.py Python base code \n\n![KafkaConsumer_upstash](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/KafkaConsumer_upstash.png)\n\n- On VS created a consumer.py to fitch the producer.py random messages from Kafka every 2 seconds\n\n![kafkaConsumer_VS](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/kafkaConsumer_VS.png)\n\n- After activating the environment: $ conda activate kafka-env\n- On GitBash: $ python producer.py \n\n![kafkaProducer_GitBash](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/kafkaProducer_GitBash.png)\n\n- After activating the environment: $ conda activate kafka-env\n- On GitBash: $ python consumer.py \n\n![kafkaConsumer_GitBash](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/kafkaConsumer_GitBash.png)\n\n- using engine Pushing all produced data thru consumer.py to PgAdmin SQL Database\n\n![kafkaConsumer_PGAdmin_SQL](https://github.com/hanydief/Kafka_Producer_Consumer_to_SQL_Project/blob/main/Outputs/kafkaConsumer_PGAdmin_SQL.png)\n\n\n# Benificial links:\nhttps://upstash.com/?gclid=CjwKCAjwvpCkBhB4EiwAujULMu611BMTjPj1328j0xQ8lKUNu55HbCBBtwRG8ysGnVgpFEML33sTRxoCWcMQAvD_BwE\nhttps://pypi.org/project/psycopg2-binary/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanydief%2FKafka_Producer_Consumer_to_SQL_Project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhanydief%2FKafka_Producer_Consumer_to_SQL_Project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhanydief%2FKafka_Producer_Consumer_to_SQL_Project/lists"}