{"id":16519783,"url":"https://github.com/zonca/python-wordcount-hadoop","last_synced_at":"2025-10-28T07:31:37.153Z","repository":{"id":9464795,"uuid":"11347853","full_name":"zonca/python-wordcount-hadoop","owner":"zonca","description":null,"archived":false,"fork":false,"pushed_at":"2013-09-02T02:08:43.000Z","size":172,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-01T12:45:29.764Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zonca.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-07-11T17:06:18.000Z","updated_at":"2023-12-04T15:17:58.000Z","dependencies_parsed_at":"2022-08-25T20:42:04.349Z","dependency_job_id":null,"html_url":"https://github.com/zonca/python-wordcount-hadoop","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fpython-wordcount-hadoop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fpython-wordcount-hadoop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fpython-wordcount-hadoop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zonca%2Fpython-wordcount-hadoop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zonca","download_url":"https://codeload.github.com/zonca/python-wordcount-hadoop/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238614565,"owners_count":19501481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T16:48:22.073Z","updated_at":"2025-10-28T07:31:36.788Z","avatar_url":"https://github.com/zonca.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"hadoop python test\n==================\n\nPython simple wordcount test based on:\n\nhttp://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/\n\nand configured to run on Gordon (HPC Cluster at the San Diego Supercomputing Center):\n\nhttp://www.sdsc.edu/us/resources/gordon/gordon_hadoop.html\n\nIt uses the Hadoop streaming interface to send input and get outputs from the Python mapper and reducer.\nHDFS is setup on the local SSD flash drives on the computing nodes, output is then copied back to local space.\n\nHow to run:\n\n* clone the repository in your home folder\n* grab the input files by running `download-inputs.sh` in the `gutemberg` folder\n* run: `qsub run.sh`\n\nRun on Amazon Elastic Map Reduce with MrJob\n-------------------------------------------\n\nSee the mrjob/ folder, more details on:\n\nhttp://www.andreazonca.com/2013/09/run-hadoop-python-jobs-on-amazon-with-mrjob.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzonca%2Fpython-wordcount-hadoop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzonca%2Fpython-wordcount-hadoop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzonca%2Fpython-wordcount-hadoop/lists"}