{"id":16141334,"url":"https://github.com/rootsongjc/magpie","last_synced_at":"2026-03-07T06:31:39.962Z","repository":{"id":57602646,"uuid":"70464851","full_name":"rootsongjc/magpie","owner":"rootsongjc","description":"Yarn on Docker - Managing Hadoop Yarn cluster with Docker Swarm.","archived":false,"fork":false,"pushed_at":"2021-12-07T13:20:48.000Z","size":8425,"stargazers_count":37,"open_issues_count":0,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-10-19T10:12:47.199Z","etag":null,"topics":["containers","docker","hadoop","swarm","yarn"],"latest_commit_sha":null,"homepage":"https://jimmysong.io/blog/yarn-on-docker/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rootsongjc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-10T07:54:31.000Z","updated_at":"2024-04-02T17:42:52.000Z","dependencies_parsed_at":"2022-09-26T20:01:02.180Z","dependency_job_id":null,"html_url":"https://github.com/rootsongjc/magpie","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rootsongjc/magpie","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsongjc%2Fmagpie","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsongjc%2Fmagpie/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsongjc%2Fmagpie/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsongjc%2Fmagpie/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rootsongjc","download_url":"https://codeload.github.com/rootsongjc/magpie/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsongjc%2Fmagpie/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30209086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T05:23:27.321Z","status":"ssl_error","status_checked_at":"2026-03-07T05:00:17.256Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","docker","hadoop","swarm","yarn"],"created_at":"2024-10-09T23:55:05.933Z","updated_at":"2026-03-07T06:31:39.945Z","avatar_url":"https://github.com/rootsongjc.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![magpie logo](doc/img/magpie_logo_small.png)\n![Build Status](https://travis-ci.org/rootsongjc/magpie.svg?branch=master)\n\n# Magpie  \n\n\nMagpie is a command line tool for deploying and  managing the Yarn on Docker cluster.\n\nBuild and run an yarn cluster on docker, pass the config item to hadoop configuration files through docker ENV.\n\n\n### How to build the docker image and run a container step by step?\n\n*You should follow the [docker project standard](doc/docker_project_standard.md) to build your own docker image.*\n\n- **Step1**\n\n  Prepare the direcotries for yarn and chown owner to hadoop.\n\n  Download hadoop-2.6.0-cdh5.5.2.tar.gz and install it in docker image. Unzip it and remove the default hadoop configuration files.\n\n- **Step2**\n\n  Put the codec lib.so files into the hadoop native directory.\n\n  Put custom hadoop configuration files to hadoop conf directory.\n\n- **Step3**\n\n  Set the ENV and Entrypoint. \n\n- **Stop4**\n\n  Run a container with sepecific ENV.\n\n\n\n### Build image\n\nEdit Dockerfile and change the base image to your own JDK7 image.\n\n```\n./build.sh\ndocker build -t hadoop-yarn:v0.1 .\n```\n\n### Run a container\n\nFor example\n\nWith hadoop ha\n\n```\ndocker run -d -e NAMESERVICE=addmp -e ACTIVE_NAMENODE_ID=namenode29 -e STANDBY_NAMENODE_ID=namenode63 -e HA_ZOOKEEPER_QUORUM=192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181 -e YARN_ZK_DIR=rmstore -e YARN_CLUSTER_ID= yarnRM -e YARN_RM1_IP=192.168.0.3 -e YARN_RM2_IP=192.168.0.2 -e YARN_JOBHISTORY_IP=192.168.0.3 -e ACTIVE_NAMENODE_IP=192.168.0.1 -e STANDBY_NAMENODE_IP=192.168.0.2  -e HA=yes hadoop-yarn:v0.1 resourcemanager\n\ndocker run -d -e NAMESERVICE=addmp -e ACTIVE_NAMENODE_ID=namenode29 -e STANDBY_NAMENODE_ID=namenode63 -e HA_ZOOKEEPER_QUORUM=192.168.0.1:2181,192.168.0.2:2181,192.168.0.3:2181 -e YARN_ZK_DIR=rmstore -e YARN_CLUSTER_ID= yarnRM -e YARN_RM1_IP=192.168.0.3 -e YARN_RM2_IP=192.168.0.2 -e YARN_JOBHISTORY_IP=192.168.0.3 -e ACTIVE_NAMENODE_IP=192.168.0.1 -e STANDBY_NAMENODE_IP=192.168.0.2  -e HA=yes hadoop-yarn:v0.1 nodemanager\n```\n\nWithout hadoop ha\n\n```\ndocker run -d -e NANENODE_IP=192.168.0.1 -e RESOURCEMANAGER_IP=192.168.0.1 -e YARN_JOBHISTORY_IP=192.168.0.1 -e HA=no hadoop-yarn:v0.1 resourcemanager\n\ndocker run -d -e NANENODE_IP=192.168.0.1 -e RESOURCEMANAGER_IP=192.168.0.1 -e YARN_JOBHISTORY_IP=192.168.0.1 -e HA=no hadoop-yarn:v0.1 nodemanager\n```\n\n### ENV included with hadoop HA \n\n- HA (default yes)\n\n- NAMESERVICE\n\n- ACTIVE_NAMENODE_IP\n\n- STANDBY_NAMENODE_IP\n\n- ACTIVE_NAMENODE_ID\n\n- STANDBY_NAMENODE_ID\n\n- HA_ZOOKEEPER_QUORUM\n\n- YARN_ZK_DIR\n\n- YARN_CLUSTER_ID\n\n- YARN_RM1_IP\n\n- YARN_RM2_IP\n\n- YARN_JOBHISTORY_IP\n\n### ENV included without hadoop HA\n\n- NAMENDOE_IP\n\n- RESOURCEMANAGER_IP\n\n- HISTORYSERVER_IP\n\n### NodeManager resource limit\n\n- CPU_CORE_NUM\n\n- NODEMANAGER_MEMORY_MB\n\n  ​\n\n## Magpie CLI management Tool\n\n### Precondition\n- No-password login to all the active resource managers.\n- Docker container's name must contain the cluster name.\n\n### Usage\n\nUse magpie -h for help usage.\n\n```\nMagpie is  a  CLI tool to manage the Yarn on Docker cluster.\n \n Magpie can be used to inspect the docker,swarm and yarn cluster status, scale the yarn cluster and decommising\n nodemanagers or delete the existed containers.\n \n Usage:\n   magpie [command]\n \n Available Commands:\n   docker      Docker cluster management tool.\n   tool        Other management tool.\n   yarn        Yarn cluster management tool.\n \n Flags:\n       --config string   config file (default is conf/magpie.toml)\n   -t, --toggle          Help message for toggle\n \n Use \"magpie [command] --help\" for more information about a command.\n```\n\n![magpie_commands](doc/img/magpie_commands.png)\n\n### Configuration\n\nMagpie use [viper](https://github.com/spf13/viper)  to resolve the [toml](https://github.com/toml-lang/toml) configuration file. \n\nConfig file default located at ./conf/magpie.toml\n\nYou can use --config to sepcify your custom configuration file.\n\n### Configuration file example\n\nMost configuration items are inherited from docker image, you don't need to specify in the configuration file, unless you want to modify the docker environment variables in the image.\n\n```Toml\n[clusters]\n#Yarn clsuter name\ncluster_name = [\"yarn1\",\"yarn2\",\"yarn3\"]\n#Swarm master ip address\nswarm_master_ip = \"192.168.0.1\"\nswarm_master_port = \"4000\"\nnodemanager_exclude_file = \"/usr/local/hadoop/etc/hadoop/exclude.txt\"\n\n[resource_managers]\n#Yarn active resourcemanager ip address\nyarn1 = \"172.18.0.2\"\nyarn2 = \"172.18.0.3\"\nyarn3 = \"172.18.0.4\"\n\n[base_container]\n#Base container used for scale the yarn cluster \nyarn1 = \"yarn1-nm1\"\nyarn2 = \"yarn2-nm1\"\nyarn3 = \"yarn3-nm1\"\n\n[nodemanager]\n#Nodemanager default config\nHA = \"yes\"\nNAMESERVICE = \"dcnameservice\"\nACTIVE_NAMENODE_IP= \"namenode-001.test.com\"\nSTANDBY_NAMENODE_IP = \"namenode-002.test.com\"\nACTIVE_NAMENODE_ID = \"namenode1\"\nSTANDBY_NAMENODE_ID = \"namenode2\"\nHA_ZOOKEEPER_QUORUM = \"zk-001.test.com:2181,zk-002.test.com:2181,zk-003.test.com:2181\"\nYARN_ZK_DIR = \"\"\nYARN_CLUSTER_ID = \"\"\nYARN_RM1_IP = \"\"\nYARN_RM2_IP = \"\"\nYARN_JOBHISTORY_IP = \"\"\nNAMENODE_IP = \"\"\nRESOURCEMANAGER_IP = \"\"\nCPU_CORE_NUM = 4\nNODEMANAGER_MEMORY_MB = 8192\nnetwork_mode = \"mynet\"\nlimit_cpus = 5\nlimit_memory_mb = 12288\nimage = \"docker-registry:5000/library/hadoop-yarn:v0.1\"\ncmd = [\"nodemanager\"]\n```\n\n### Feature\n\n- Inspect the swarm cluster status.\n- Inspect the yarn clsuter status.\n- Inspect the docker contianer status include contianer's host config and config items.\n- Delete docker containers.\n- Offline or decomissing yarn nodemanagers.\n- Scaling yarn cluster.\n- Add a yarn create command to create a new yarn nodemanager for other yarn cluster.\n\n### Build\n\n**Build environment**\n\ngo 1.7.4 amd64\n\n**Build magpie on your own platform**\n\nEdit `Makefile` to set `GOOS` and `GOARCH`  to your target environment .\n\n```Shell\nmake\n```\n\n### TODO\n\n- Scale and flux the yarn cluster automatically\n- Nodemanager registered on resourcemanager repeatability\n\n\n### References\n\n- [Docker remote API version 1.23]( https://docs.docker.com/engine/reference/api/docker_remote_api_v1.23/)\n- [YARN RESTful API version 2.6.0]( https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html)\n- [Swarm API version 1.2.2](https://docs.docker.com/swarm/swarm-api/)\n\n### About\n\n*FYI: If you want to create a yarn cluster with multiple nodemanagers, you need a docker plugins to make the docker container on different hosts can be accessed with each others.*\nYou need a docker ipam plugin to make the continers located on different hosts can be accessed by each others. \nTry this:https://github.com/rootsongjc/docker-ipam-plugin\nYou also need a plugin to listen on docker nodes and register container's IP-hostname into a DNS server so that docker containers can recognise each other by the hostname which is the same with the container ID.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frootsongjc%2Fmagpie","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frootsongjc%2Fmagpie","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frootsongjc%2Fmagpie/lists"}