{"id":15958263,"url":"https://github.com/taiwotman/pysparkstreaming","last_synced_at":"2026-03-18T17:21:15.998Z","repository":{"id":149643106,"uuid":"147985556","full_name":"taiwotman/PySparkStreaming","owner":"taiwotman","description":"Demonstrate pyspark structured programming using template design pattern","archived":false,"fork":false,"pushed_at":"2018-09-14T17:21:37.000Z","size":21,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T09:44:48.654Z","etag":null,"topics":["analytics","data","pyspark","streaming","wordcount"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/taiwotman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-09T02:12:39.000Z","updated_at":"2022-10-07T15:24:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"5a3dfc51-1e88-4af5-9ac6-0f2f99e9fc04","html_url":"https://github.com/taiwotman/PySparkStreaming","commit_stats":{"total_commits":13,"total_committers":2,"mean_commits":6.5,"dds":0.07692307692307687,"last_synced_commit":"c3aa96501841836b09319dab6b81163c8eac409f"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/taiwotman/PySparkStreaming","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taiwotman%2FPySparkStreaming","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taiwotman%2FPySparkStreaming/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taiwotman%2FPySparkStreaming/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taiwotman%2FPySparkStreaming/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/taiwotman","download_url":"https://codeload.github.com/taiwotman/PySparkStreaming/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/taiwotman%2FPySparkStreaming/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28547219,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T14:59:57.589Z","status":"ssl_error","status_checked_at":"2026-01-18T14:59:46.540Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","data","pyspark","streaming","wordcount"],"created_at":"2024-10-07T14:02:28.903Z","updated_at":"2026-01-18T18:32:22.050Z","avatar_url":"https://github.com/taiwotman.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Pyspark Streaming\n\n**An example PySpark project.**\n\nThe project demonstrate pyspark structured programming using template design pattern\n\n**Upgrade python**\n\n    curl https://bootstrap.pypa.io/get-pip.py | python\n\n\n**Running Spark job with spark-submit on Command line**\n\n    mkdir ./dist\n    cp ./src/main.py ../dist/\n    cd ./src \u0026\u0026 zip -x main.py -r ../dist/wordcount .\n\n    source venv/bin/activate\n\n**To run this PySpark Streaming application, execute the following command from your $SPARK_HOME folder:**\n\n    ./bin/spark-submit wordcount.py localhost 9999\n\n**To begin the streaming, on the command line, type the following netcat command:**\n\n    nc -lk 9999\n\n**Then, start typing your events, for example:**\n\nFor the first second, type\n\n        apple apple apple\n\nFor the second second, type\n    \n        orange orange apple\n\nWait a second; for the fourth second, type\n    \n        mango mango mang\n\n\n**Reference**\n\nStructured Streaming Programming Guide: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#creating-streaming-dataframes-and-streaming-datasets\n\nCreating a PySpark project with pytest, pyenv, and egg files: https://medium.com/@mrpowers/creating-a-pyspark-project-with-pytest-pyenv-and-egg-files-d2709eb1604c\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaiwotman%2Fpysparkstreaming","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftaiwotman%2Fpysparkstreaming","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftaiwotman%2Fpysparkstreaming/lists"}