{"id":26658872,"url":"https://github.com/denisecase/buzzline-04-case","last_synced_at":"2025-04-11T14:09:36.362Z","repository":{"id":274075700,"uuid":"897593281","full_name":"denisecase/buzzline-04-case","owner":"denisecase","description":"Adding live visualizations to streaming data applications","archived":false,"fork":false,"pushed_at":"2025-02-01T19:18:42.000Z","size":124,"stargazers_count":0,"open_issues_count":0,"forks_count":25,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-25T10:17:03.755Z","etag":null,"topics":["animation","data","kafka","matplotlib","python","streaming"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denisecase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-02T22:34:06.000Z","updated_at":"2025-02-01T19:18:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"d5fb0a71-932b-4bad-8b9f-dd1dd8741a95","html_url":"https://github.com/denisecase/buzzline-04-case","commit_stats":null,"previous_names":["denisecase/buzzline-04-case"],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fbuzzline-04-case","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fbuzzline-04-case/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fbuzzline-04-case/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fbuzzline-04-case/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denisecase","download_url":"https://codeload.github.com/denisecase/buzzline-04-case/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248413704,"owners_count":21099346,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["animation","data","kafka","matplotlib","python","streaming"],"created_at":"2025-03-25T10:17:08.491Z","updated_at":"2025-04-11T14:09:36.339Z","avatar_url":"https://github.com/denisecase.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# buzzline-04-case\n\nWe can analyze and visualize different types of streaming data as the information arrives.\n\nThe producers don't change from buzzline-03-case - they write the same information to a Kafka topic, except the csv producer for the smart smoker has been modified to not run continuously. It will stop after reading all the rows in the CSV file. \nThe consumers have been enhanced to add visualization. \n\nThis project uses matplotlib and its animation capabilities for visualization. \n\nIt generates three applications:\n\n1. A basic producer and consumer that exchange information via a dynamically updated file. \n2. A JSON producer and consumer that exchange information via a Kafka topic. \n3. A CSV producer and consumer that exchange information via a different Kafka topic. \n\nAll three applications produce live charts to illustrate the data. \n\n## Task 1. Use Tools from Module 1 and 2\n\nBefore starting, ensure you have completed the setup tasks in \u003chttps://github.com/denisecase/buzzline-01-case\u003e and \u003chttps://github.com/denisecase/buzzline-02-case\u003e first. \nPython 3.11 is required. \n\n## Task 2. Copy This Example Project and Rename\n\nOnce the tools are installed, copy/fork this project into your GitHub account\nand create your own version of this project to run and experiment with. \nFollow the instructions in [FORK-THIS-REPO.md](https://github.com/denisecase/buzzline-01-case/docs/FORK-THIS-REPO.md).\n\nOR: For more practice, add these example scripts or features to your earlier project. \nYou'll want to check requirements.txt, .env, and the consumers, producers, and util folders. \nUse your README.md to record your workflow and commands. \n    \n\n## Task 3. Manage Local Project Virtual Environment\n\nFollow the instructions in [MANAGE-VENV.md](https://github.com/denisecase/buzzline-01-case/docs/MANAGE-VENV.md) to:\n1. Create your .venv\n2. Activate .venv\n3. Install the required dependencies using requirements.txt.\n\n## Task 4. Start Zookeeper and Kafka (2 Terminals)\n\nIf Zookeeper and Kafka are not already running, you'll need to restart them.\nSee instructions at [SETUP-KAFKA.md] to:\n\n1. Start Zookeeper Service ([link](https://github.com/denisecase/buzzline-02-case/blob/main/docs/SETUP-KAFKA.md#step-7-start-zookeeper-service-terminal-1))\n2. Start Kafka ([link](https://github.com/denisecase/buzzline-02-case/blob/main/docs/SETUP-KAFKA.md#step-8-start-kafka-terminal-2))\n\n---\n\n## Task 5. Start a Basic (File-based, not Kafka) Streaming Application\n\nThis will take two terminals:\n\n1. One to run the producer which writes to a file in the data folder. \n2. Another to run the consumer which reads from the dynamically updated file. \n\n### Producer Terminal\n\nStart the producer to generate the messages. \n\nIn VS Code, open a NEW terminal.\nUse the commands below to activate .venv, and start the producer. \n\nWindows:\n\n```shell\n.venv\\Scripts\\activate\npy -m producers.basic_json_producer_case\n```\n\nMac/Linux:\n```zsh\nsource .venv/bin/activate\npython3 -m producers.basic_json_producer_case\n```\n\n### Consumer Terminal\n\nStart the associated consumer that will process and visualize the messages. \n\nIn VS Code, open a NEW terminal in your root project folder. \nUse the commands below to activate .venv, and start the consumer. \n\nWindows:\n```shell\n.venv\\Scripts\\activate\npy -m consumers.basic_json_consumer_case\n```\n\nMac/Linux:\n```zsh\nsource .venv/bin/activate\npython3 -m consumers.basic_json_consumer_case\n```\n\n### Review the Application Code\n\nReview the code for both the producer and the consumer. \nUnderstand how the information is generated, written to a file, and read and processed. \nReview the visualization code to see how the live chart is produced. \nWhen done, remember to kill the associated terminals for the producer and consumer. \n\n\n---\n\n## Task 6. Start a (Kafka-based) JSON Streaming Application\n\nThis will take two terminals:\n\n1. One to run the producer which writes to a Kafka topic. \n2. Another to run the consumer which reads from that Kafka topic.\n\nFor each one, you will need to: \n1. Open a new terminal. \n2. Activate your .venv.\n3. Know the command that works on your machine to execute python (e.g. py or python3).\n4. Know how to use the -m (module flag to run your file as a module).\n5. Know the full name of the module you want to run. \n   - Look in the producers folder for json_producer_case.\n   - Look in the consumers folder for json_consumer_case.\n\n\n### Review the Application Code\n\nReview the code for both the producer and the consumer. \nUnderstand how the information is generated and written to a Kafka topic, and consumed from the topic and processed. \nReview the visualization code to see how the live chart is produced. \n\nCompare the non-Kafka JSON streaming application to the Kafka JSON streaming application.\nBy organizing code into reusable functions, which functions can be reused? \nWhich functions must be updated based on the sharing mechanism? \nWhat new functions/features must be added to work with a Kafka-based streaming system?\n\nWhen done, remember to kill the associated terminals for the producer and consumer. \n\n---\n\n## Task 7. Start a (Kafka-based) CSV Streaming Application\n\nThis will take two terminals:\n\n1. One to run the producer which writes to a Kafka topic. \n2. Another to run the consumer which reads from that Kafka topic.\n\nFor each one, you will need to: \n1. Open a new terminal. \n2. Activate your .venv.\n3. Know the command that works on your machine to execute python (e.g. py or python3).\n4. Know how to use the -m (module flag to run your file as a module).\n5. Know the full name of the module you want to run. \n   - Look in the producers folder for csv_producer_case.\n   - Look in the consumers folder for csv_consumer_case.\n\n### Review the Application Code\n\nReview the code for both the producer and the consumer. \nUnderstand how the information is generated and written to a Kafka topic, and consumed from the topic and processed. \nReview the visualization code to see how the live chart is produced. \n\nCompare the JSON application to the CSV streaming application.\nBy organizing code into reusable functions, which functions can be reused? \nWhich functions must be updated based on the type of data?\nHow does the visualization code get changed based on the type of data and type of chart used?\nWhich aspects are similar between the different types of data? \n\nWhen done, remember to kill the associated terminals for the producer and consumer. \n\n---\n\n## Possible Explorations\n\n- JSON: Process messages in batches of 5 messages.\n- JSON:Limit the display to the top 3 authors.\n- Modify chart appearance.\n- Stream a different set of data and visualize the custom stream with an appropriate chart. \n- How do we find out what types of charts are available? \n- How do we find out what attributes and colors are available?\n\n---\n\n## Later Work Sessions\nWhen resuming work on this project:\n1. Open the folder in VS Code. \n2. Start the Zookeeper service.\n3. Start the Kafka service.\n4. Activate your local project virtual environment (.env).\n\n## Save Space\nTo save disk space, you can delete the .venv folder when not actively working on this project.\nYou can always recreate it, activate it, and reinstall the necessary packages later. \nManaging Python virtual environments is a valuable skill. \n\n## License\nThis project is licensed under the MIT License as an example project. \nYou are encouraged to fork, copy, explore, and modify the code as you like. \nSee the [LICENSE](LICENSE.txt) file for more.\n\n## Live Chart Examples\n\nLive Bar Chart (JSON file streaming)\n\n![Basic JSON (file-exchange)](images/live_bar_chart_basic_example.jpg)\n\nLive Bar Chart (Kafka JSON streaming)\n\n![JSON (Kafka)](images/live_bar_chart_example.jpg)\n\nLive Line Chart with Alert (Kafka CSV streaming)\n\n![CSV (Kafka)](images/live_line_chart_example.jpg)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fbuzzline-04-case","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenisecase%2Fbuzzline-04-case","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fbuzzline-04-case/lists"}