{"id":50508858,"url":"https://github.com/denisecase/streaming-03-analytics","last_synced_at":"2026-06-02T18:31:11.814Z","repository":{"id":356962643,"uuid":"1214511918","full_name":"denisecase/streaming-03-analytics","owner":"denisecase","description":"Streaming data analytics: validate and summarize messages.","archived":false,"fork":false,"pushed_at":"2026-05-10T16:16:47.000Z","size":101,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-10T18:21:32.323Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://denisecase.github.io/streaming-03-analytics/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/denisecase.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-18T17:18:01.000Z","updated_at":"2026-05-10T16:16:52.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/denisecase/streaming-03-analytics","commit_stats":null,"previous_names":["denisecase/streaming-03-analytics"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/denisecase/streaming-03-analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fstreaming-03-analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fstreaming-03-analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fstreaming-03-analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fstreaming-03-analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/denisecase","download_url":"https://codeload.github.com/denisecase/streaming-03-analytics/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/denisecase%2Fstreaming-03-analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33833277,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-02T02:00:07.132Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-02T18:31:11.128Z","updated_at":"2026-06-02T18:31:11.793Z","avatar_url":"https://github.com/denisecase.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# streaming-03-analytics\n\n[![API Reference](https://img.shields.io/badge/API--Utils-datafun--streaming-purple)](https://denisecase.github.io/datafun-streaming/api/)\n[![Workflow Guide](https://img.shields.io/badge/Pro--Guide-pro--analytics--02-green)](https://denisecase.github.io/pro-analytics-02/workflow-b-apply-example-project/)\n[![Python 3.14](https://img.shields.io/badge/python-3.14%2B-blue?logo=python)](./pyproject.toml)\n[![MIT](https://img.shields.io/badge/license-see%20LICENSE-yellow.svg)](./LICENSE)\n\n\u003e Streaming data analytics: validate and summarize messages.\n\nStreaming analytics requires working with data in motion\nand distributed, scalable systems.\nThis course builds capabilities through working projects.\nIn the age of generative AI, durable skills are grounded in real work:\nsetting up a professional environment,\nreading and running code,\nunderstanding the logic,\nand pushing work to a shared repository.\nEach project follows the structure of professional Python projects.\nWe learn by doing.\n\n## This Project\n\nThis project focuses on analytics performed as messages are consumed.\n\nThe project uses Kafka to move sales messages from a producer to a consumer.\nThe consumer reads each message, validates required fields, computes derived values,\nwrites processed records to CSV, and logs running summary statistics.\n\nThis module adds validation and message-by-message analytics to the streaming workflow.\n\nThe goal is to see how each incoming message can be checked, transformed,\nand summarized without waiting for a batch process.\n\n## Working Files\n\nYou'll work with just these areas:\n\n- **data/** - input data and generated output files\n- **docs/** - the project narrative and documentation\n- **src/streaming/** - producer, consumer, and supporting code\n- **pyproject.toml** - update authorship \u0026 links\n- **zensical.toml** - update authorship \u0026 links\n\n## Instructions\n\nFollow the\n[step-by-step workflow guide](https://denisecase.github.io/pro-analytics-02/workflow-b-apply-example-project/)\nto complete:\n\n1. Phase 1. **Start \u0026 Run**\n2. Phase 2. **Change Authorship**\n3. Phase 3. **Read \u0026 Understand**\n4. Phase 4. **Modify**\n5. Phase 5. **Apply**\n\n## Challenges\n\nChallenges are expected.\nSometimes instructions may not quite match your operating system.\nWhen issues occur, share screenshots, error messages, and details about what you tried.\nWorking through issues is part of implementing professional projects.\n\n## Success\n\nAfter completing Phase 1. **Start \u0026 Run**, you'll have your own GitHub project\nrunning with Kafka.\n\nUse four named terminals:\n\n1. **kafka** - keep the Kafka message broker running\n2. **topics** - create, list, or reset Kafka topics\n3. **producer** - run the project and producer\n4. **consumer** - run the consumer\n\nAfter the producer and consumer run successfully, you should see:\n\n```shell\n========================\nConsumer executed successfully!\n========================\n```\n\nA new file `project.log` will appear in the root project folder\nand processed data will appear in data/output/.\n\n## Command Reference\n\nThe commands below are used in the workflow guide above.\nThey are provided here for convenience.\n\n**Important:** the first few times you run a project,\nfollow the guide with the **complete instructions**.\n\n\u003cdetails\u003e\n\u003csummary\u003eShow command reference\u003c/summary\u003e\n\n### In a machine terminal (open in your `Repos` folder)\n\nAfter you get a copy of this repo in your own GitHub account,\nopen a machine terminal in your `Repos` folder:\n\n```bash\n# Replace username with YOUR GitHub username.\ngit clone https://github.com/username/streaming-03-analytics\n\ncd streaming-03-analytics\ncode .\n```\n\n### In VS Code Terminal 1: Start Kafka (kafka)\n\nFor full instructions see\n[**start kafka**](https://denisecase.github.io/pro-analytics-02/kafka/start-kafka/).\n\nIf any command fails,\nrepeat the steps at\n[**install kafka**](https://denisecase.github.io/pro-analytics-02/kafka/install-kafka/)\nuntil starting up is reliable.\n\nOpen a new VS Code terminal. Rename it `kafka`.\nIf running Windows, specify the terminal type as **wsl** or\ntype `wsl`.\nRun the commands one at a time.\n\nStep 1. Verify Java and PATH\n\n```bash\necho \"$JAVA_HOME\"\n\n\"$JAVA_HOME/bin/java\" --version\n```\n\nStep 2. Rebuild ClusterID (as needed)\n\n```bash\ncd ~/kafka\n\nrm -rf /tmp/kraft-combined-logs\n\nKAFKA_CLUSTER_ID=\"$(bin/kafka-storage.sh random-uuid)\"\n\necho \"Cluster ID: $KAFKA_CLUSTER_ID\"\n\nbin/kafka-storage.sh format --standalone -t \"$KAFKA_CLUSTER_ID\" -c config/server.properties\n```\n\nStep 3. Start kafka server (keep running)\n\n```bash\ncd ~/kafka\n\nbin/kafka-server-start.sh config/server.properties\n```\n\n### In VS Code terminal 2: Create Topic (topics)\n\nFor full instructions see\n[**create topic**](https://denisecase.github.io/pro-analytics-02/kafka/create-topic/).\n\nThe topic name must match the name defined in your\n`.env` file (copy `.env.example` to `.env`).\n\nOpen another VS Code terminal. Rename it `topics`.\nIf running Windows, specify the terminal type as **wsl** or\ntype `wsl`.\nRun the commands one at a time.\n\n```bash\ncd ~/kafka\n\nbin/kafka-topics.sh --create \\\n  --bootstrap-server localhost:9092 \\\n  --partitions 1 \\\n  --replication-factor 1 \\\n  --topic streaming-03-analytics-case\n```\n\n### In VS Code Terminal 3: Run Project and Producer (producer)\n\nOpen another VS Code terminal. Rename it `producer`.\nIf running Windows, use **PowerShell**.\nRun the commands one at a time.\n\n```shell\n# reset uv cache only if/when you start getting strange dependency errors\n# uv cache clean\n\nuv self update\nuv python pin 3.14\nuv sync --extra dev --extra docs --upgrade\n\nuvx pre-commit install\n\ngit add -A\nuvx pre-commit run --all-files\n# repeat if changes were made\ngit add -A\nuvx pre-commit run --all-files\n\n# run the producer\nclear\nuv run python -m streaming.kafka_producer_case\n\n# do chores\nuv run ruff format .\nuv run ruff check . --fix\nuv run python -m pyright\nuv run python -m pytest\nuv run python -m zensical build\n\n# save progress\ngit add -A\ngit commit -m \"update\"\ngit push -u origin main\n```\n\n### In VS Code Terminal 4: Run Consumer (consumer)\n\nOpen another VS Code terminal. Rename it `consumer`.\nIf running Windows, use **PowerShell**.\nRun the commands one at a time.\nClear the terminal, then start the consumer.\n\n```shell\nclear\nuv run python -m streaming.kafka_consumer_case\n```\n\nTo start fresh, see\n[manage topics](https://denisecase.github.io/pro-analytics-02/kafka/manage-topics/)\nto delete the topic and recreate it.\n\n\u003c/details\u003e\n\n## Notes\n\n- Use the **UP ARROW** and **DOWN ARROW** in the terminal to scroll through past commands.\n- Use `CTRL+f` to find (and replace) text within a file.\n- You do not need to add to or modify `tests/`. They are provided for example only.\n- Many files are silent helpers. Explore as you like, but nothing is required.\n- You do NOT not to understand everything; understanding builds naturally over time.\n\n## Troubleshooting \u003e\u003e\u003e or\n\nIf you see something like this in your terminal: `\u003e\u003e\u003e` or `...`\nYou accidentally started Python interactive mode.\nIt happens.\nPress `Ctrl+c` (both keys together) or `Ctrl+Z` then `Enter` on Windows.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fstreaming-03-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdenisecase%2Fstreaming-03-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdenisecase%2Fstreaming-03-analytics/lists"}