{"id":15286258,"url":"https://github.com/marky-mark/emrclient","last_synced_at":"2026-01-04T08:51:38.581Z","repository":{"id":57426319,"uuid":"41688248","full_name":"marky-mark/emrclient","owner":"marky-mark","description":"A handy tool for emr appliances ","archived":false,"fork":false,"pushed_at":"2016-06-10T13:31:04.000Z","size":15,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-04-15T12:22:21.839Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marky-mark.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-31T16:50:24.000Z","updated_at":"2020-05-11T06:04:32.000Z","dependencies_parsed_at":"2022-09-11T05:01:11.751Z","dependency_job_id":null,"html_url":"https://github.com/marky-mark/emrclient","commit_stats":null,"previous_names":["zalando/emrclient"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marky-mark%2Femrclient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marky-mark%2Femrclient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marky-mark%2Femrclient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marky-mark%2Femrclient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marky-mark","download_url":"https://codeload.github.com/marky-mark/emrclient/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245169925,"owners_count":20571980,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-30T15:11:32.610Z","updated_at":"2026-01-04T08:51:38.538Z","avatar_url":"https://github.com/marky-mark.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"### EMR Client\n\nThis is created to view and kill applications/jobs running on yarn inside Amazon EMR or any other remote location. \nCurrently the amazon api does not include stopping jobs. Also supports adding steps to EMR and listing them. The amazon \nEMR api does not contain have calls to terminate jobs and so must be done via yarn api. A small layer of caching is also\nadded.\n\n### Pre-requisite\n\nThe api must be exposed publicly. Either assign the master ec2 box a public ip or use an ssh tunnel like below. When setting up\nthe EMR cluster the public key must be added to the cluster in order to ssh into the box (assuming runs on port 8088).\n\n    ssh -v -i \u003cKEY LOCATION\u003e -N -L 8088:\u003cDNS PUBLIC IP\u003e.eu-west-1.compute.amazonaws.com:8088 hadoop@\u003cDNS PUBLIC IP\u003e.eu-west-1.compute.amazonaws.com\n\n### Run \n\n    python3 -m emrclient\n\n### Install via pip\n\n    pip3 install --upgrade emrclient\n\n### Commands\n\nIn order to use the `list_applications` and `kill_application` the master ec2 instance of EMR cluster created must be \nassigned a public ip \n\n##### Display help\n    \n    ./emrclient --help\n\n##### Configure\n\nThe purpose of this call is to cache some of the common parameters.\nSet the master ip and port of the EMR master instance and yarn api (default 8088), this can be found in the EC2 tab. The cache\nis currently in `~/.emrclient`\n\n    emrclient configure -m \u003cMASTER IP:PORT\u003e -b \u003cS3 BUCKET\u003e -c \u003cCluster id\u003e -r \u003cREGION\u003e\n\n##### List Running Applications\n    \nOnce this is set you may list applications by running\n\n    emrclient list-applications-running\n\nAlternatively the master may be temporally overwritten by using `-m \u003cMASTER ADDRESS\u003e`\n   \n##### List Applications by State\n    \nOnce this is set you may list applications by state(RUNNING, KILLED, FAILED)\n\n    emrclient list-applications \u003cSTATE\u003e\n\nAlternatively the master may be temporally overwritten by using `-m \u003cMASTER ADDRESS\u003e`\n   \n##### Kill an Application\n\nPick an application from the list to kill\n\n    emrclient kill-application \u003cAPPLICATION ID\u003e\n    \nAlternatively the master may be temporally overwritten by using `-m \u003cMASTER ADDRESS\u003e`\n\n##### List jobs on cluster\n\nList Jobs by state. 'PENDING','RUNNING','COMPLETED','CANCELLED','FAILED','INTERRUPTED'\n\n    emrclient list-steps \u003cSTATE\u003e  \n    \nOptions\n\n* -c, --cluster-id TEXT  Overwrite region of cluster. Not cached\n* -r, --region TEXT      Overwrite region of cluster. Not cached\n\n##### Submit job\n\nPick an application from the list to kill\n\n    emrclient submit-job \u003cNAME\u003e \u003cMAIN CLASS\u003e \n    \nOptions\n\n* -f, --file TEXT        Upload the file. This will be uploaded to s3 and overwrite whatever is there\n* -b, --s3-bucket TEXT   Overwrite the s3 bucket location for the file to be uploaded to. Does not get cached\n* -s, --s3-file TEXT     s3 file for the job. Used if already uploaded\n* -c, --cluster-id TEXT  Overwrite the cluster id of EMR. Not cached\n* -r, --region TEXT      Overwrite region of cluster. Not cached\n* -a, --args TEXT        arguments for jar\n* --help                 Display help message\n    \nExample of file already up on s3\n\n    emrclient submit-job Foo Bar -a -m,yarn-cluster,-z,XXX.YYY.ZZZ:2181 -s s3://some-bucket/some-jar-0.0.1-SNAPSHOT.jar\n    \nExample of uploading file to s3 and using it\n\n    emrclient submit-job Foo Bar -a -m,yarn-cluster,-z,XXX.YYY.ZZZ:2181 -f /some/file.jar -b some-bucket\n    \n    ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarky-mark%2Femrclient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarky-mark%2Femrclient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarky-mark%2Femrclient/lists"}