{"id":25717563,"url":"https://github.com/daniel-elston/real-time-reddit-scalable-processing","last_synced_at":"2026-05-12T13:43:01.606Z","repository":{"id":278450364,"uuid":"920643827","full_name":"Daniel-Elston/real-time-reddit-scalable-processing","owner":"Daniel-Elston","description":"Scaling NLP processing pipelines with Dask and PySpark, utilising Apache Kafka real-time data streaming, for optimal LLM training","archived":false,"fork":false,"pushed_at":"2025-02-24T15:35:31.000Z","size":3222,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-24T16:40:34.423Z","etag":null,"topics":["apache-kafka","dask-distributed","embeddings","llm","llm-training","nlp","pyspark","scalability"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Daniel-Elston.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-22T14:21:57.000Z","updated_at":"2025-02-24T15:35:34.000Z","dependencies_parsed_at":"2025-02-24T16:41:13.853Z","dependency_job_id":null,"html_url":"https://github.com/Daniel-Elston/real-time-reddit-scalable-processing","commit_stats":null,"previous_names":["daniel-elston/real-time-reddit-sentiment","daniel-elston/real-time-reddit-scalable-processing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Daniel-Elston/real-time-reddit-scalable-processing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Elston%2Freal-time-reddit-scalable-processing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Elston%2Freal-time-reddit-scalable-processing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Elston%2Freal-time-reddit-scalable-processing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Elston%2Freal-time-reddit-scalable-processing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Daniel-Elston","download_url":"https://codeload.github.com/Daniel-Elston/real-time-reddit-scalable-processing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Elston%2Freal-time-reddit-scalable-processing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32942140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-12T09:19:52.626Z","status":"ssl_error","status_checked_at":"2026-05-12T09:17:33.438Z","response_time":102,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","dask-distributed","embeddings","llm","llm-training","nlp","pyspark","scalability"],"created_at":"2025-02-25T15:38:37.866Z","updated_at":"2026-05-12T13:43:01.589Z","avatar_url":"https://github.com/Daniel-Elston.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# README\n\n### Goal\nBuild scalable text data processing pipelines for efficient model training with Dask and PySpark, utilising Apache Kafka for real-time data streaming.\n\n---\n\n### Results\n1. Results can be found in the ``reports/result.xlsx`` file.\n2. This is partition number 2 of resultant Dask processing 4 partitions.\n\n---\n\n### **QRG:**\n\n1. Load up docker app\n2. Load 2 separate WSL terminals (T1 and T2)\n3. In T1, run ``docker-compose up --build``\n4. Open file ``config/settings`` and adjust the Config to either 'extract', 'transform' or 'results'\n5. Once all images are running, in T2, run ``python main.py``\n6. Data is streamed in temrinal but also saved: ``data/temp/reddit_comments.json``\n7. Sample result of PySpark and Dask processing can be found as SDOs in ``data/results/*.xlsx``\n\n---\n\n### **Requirements:**\n\n- WSL2\n- Ubuntu 24.04\n- Python 3.12.*\n\n---\n\n### Ensure Java Runtime Env:\n\n``sudo apt update``\n\n``sudo apt install openjdk-11-jdk``\n\n``readlink -f $(which java)``\n\n``export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64``\n\n**Add lines to .zshrc:**\n\n``echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64' \u003e\u003e ~/.bashrc``\n\n``echo 'export PATH=$JAVA_HOME/bin:$PATH' \u003e\u003e ~/.bashrc``\n\n---\n\n### Apologies \u0026 Disclaimer\n\nThis project streams real-time Reddit comments, which are generated by users across various subreddits. As a result, some content may include offensive, inappropriate, or controversial language. Please note that I do not endorse or control the content of these comments.\n\nI sincerely apologise for any offensive material that may appear during the data stream. If you come across content that is particularly concerning, I encourage you to report it directly to Reddit through their moderation tools.\n\nThank you for your understanding.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel-elston%2Freal-time-reddit-scalable-processing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaniel-elston%2Freal-time-reddit-scalable-processing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel-elston%2Freal-time-reddit-scalable-processing/lists"}