{"id":13740081,"url":"https://github.com/mesos/mr-redis","last_synced_at":"2025-05-08T19:35:50.969Z","repository":{"id":57486782,"uuid":"48480336","full_name":"mesos/mr-redis","owner":"mesos","description":"Redis on Apache Mesos","archived":true,"fork":false,"pushed_at":"2018-12-10T14:21:09.000Z","size":10788,"stargazers_count":127,"open_issues_count":20,"forks_count":32,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-11-02T19:06:00.830Z","etag":null,"topics":["framework","golang","mesos","redis-cache"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mesos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-23T09:00:42.000Z","updated_at":"2024-04-03T00:26:32.000Z","dependencies_parsed_at":"2022-09-02T00:02:53.665Z","dependency_job_id":null,"html_url":"https://github.com/mesos/mr-redis","commit_stats":null,"previous_names":["dhilipkumars/mrredis"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mesos%2Fmr-redis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mesos%2Fmr-redis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mesos%2Fmr-redis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mesos%2Fmr-redis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mesos","download_url":"https://codeload.github.com/mesos/mr-redis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224593864,"owners_count":17337185,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["framework","golang","mesos","redis-cache"],"created_at":"2024-08-03T04:00:42.276Z","updated_at":"2024-11-15T09:31:03.629Z","avatar_url":"https://github.com/mesos.png","language":"Go","funding_links":[],"categories":["Frameworks"],"sub_categories":["Storage and Serving"],"readme":"[![Go Report Card](https://goreportcard.com/badge/github.com/mesos/mr-redis)](https://goreportcard.com/report/github.com/mesos/mr-redis)\n[![GoDoc](https://godoc.org/github.com/mesos/mr-redis?status.svg)](https://godoc.org/github.com/mesos/mr-redis)\n[![Build Status](https://travis-ci.org/mesos/mr-redis.svg?branch=master)](https://travis-ci.org/mesos/mr-redis)\n# mr-redis  \n\nMesos runs Redis.\n\n\u003cimg src=\"./logo.jpg\" width=\"20%\" height=\"20%\"\u003e \n\nA minimalistic framework for Redis workload on Apache Mesos\n\nThis framework supports the following features\n\n * Creates/Maintains Single Redis-instance\n * Creates/Maintains Redis-Instances with Master-Slave setup \n * A centralized persistence layer currently enabled by etcd\n\n## Why mr-redis?\n\nAt [Huawei] (http://www.huawei.com/en/) we foresee creating, running and maintaining huge number of redis instances on our datacenters.  We intially evaluated few cluster managers for this job, but due to the specific requirements of 'redis' itself those solutions did not satisfy most of our needs.  We quickly did a POC by writing a framework exclusively for Redis on Apache Mesos. Based on the outcome we decided to initate this project and work with the opensource community to build a robust custom framework for Redis which will be useful for Huawei as well as for the rest of the world.\n\n##Project Status is Alpha.\n We have built a basic working functionality and would like to build a strong functional framework for Redis along with the community.  In the meanwhile please feel free to try this out and give us feedback by creating PR and Issues. \n\n## Who should use mr-redis \n* If your organization has a requirement for creating and maintaining huge number of redis service instances\n* If you are planning to host 'redis' as a Service \n* If redis instances need to be created in seconds and not in minutes\n* If you are already using Apache Mesos as a Resource Manager for your datacenter and want to add Redis (as a service) workload to it\n\n## Installation Instructions\n\n### Prerequisite\nFor Mr-Redis to work you will need below software packages already available in your datacenter (Cloud)\n* [Apache Mesos](http://mesos.apache.org/gettingstarted/) :- The Resource Manager of your DC to which mr-redis scheduler will connect to.\n* [Golang Dev Environment] (https://golang.org/doc/install) :- If you are planning to build mr-redis from source you will need to setup standard golang development environment. \n* [etcd](https://github.com/coreos/etcd#getting-started) or [ZooKeeper](http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html) :- mr-redis uses etcd/ZK to store its state information so a running instance of either etcd or ZK is required in your datacenter\n* [redis-server](https://hub.docker.com/r/library/redis) :- mr-redis scheduler is capable of pulling the redis docker image and executing it. (new) You no longer need to supply a Redis binary\n\nInstalling the scheduler/framework can be done in three ways\n\n### From source code (Developer)\n```\n$git clone https://github.com/mesos/mr-redis.git ${GOPATH}/src/github.com/mesos/mr-redis\n$go get github.com/tools/godep\n$cd ${GOPATH}/src/github.com/mesos/mr-redis/sched\n$godep go build .\n$cd ../exec/\n$godep go build -o MrRedisExecutor .\n$cd ../cli/\n$godep go build -o mrr .\n```\n### From RELEASE (Alpha V1)\n```\n$mkdir mr-redis\n$cd mr-redis\n$wget https://github.com/mesos/mr-redis/releases/download/v0.01-alpha/sched\n$wget https://github.com/mesos/mr-redis/releases/download/v0.01-alpha/MrRedisExecutor\n```\nLinux CLI\n```\n$wget https://github.com/mesos/mr-redis/releases/download/v0.01-alpha/mrr_linux_amd64\n```\nWindows CLI\n```\n$wget https://github.com/mesos/mr-redis/releases/download/v0.01-alpha/mrr_win_amd64.exe\n```\nDarwin (MAC) CLI\n```\n$wget https://github.com/mesos/mr-redis/releases/download/v0.01-alpha/mrr_darwin_amd64\n```\nMark them as binaries \n```\n$chmod u+x *\n```\nCreate a link or alias for ease of use, for example in Linux perform below to create a link\n```\n$ln -s mrr_linux_amd64 mrr\n```\n\n### DC/OS\nMrRedis is integrated with DC/OS's universe, it should be pretty straight forward to install it like any other package.\n```\n$dcos package install mr-redis\n```\n#### NOTE for DC/OS Users:\nUnlike ETCD, Cassandra and other database frameworks installing mr-redis (scheduler) itself will not create any redis instance in your DC/OS environment, you have to further download and use the CLI (mrr) in order to create redis instances.  \n\n#### Reaching the framework\nThe scheduler binds itself with the port 5656, so from a node within the cluster or from a node from which mrredis.mesos is resolvable try to create an instance with the below configuration. {InstanceName : \"TestInstance\"; MemoryCapacity : 100 MB; Number of Slaves : 2;}\n```\n$curl -X \"POST\" mrredis.mesos:5656/v1/CREATE/TestInstance/100/1/2\nRequest Accepted, Instance will be created.\n```\n\n#### Status of created Instance\nTry the below curl command.  This gives a json response about this instance.\n```\n$curl mrredis.mesos:5656/v1/STATUS/TestInstance\n{\"Name\":\"TestInstance\",\"Type\":\"MS\",\"Status\":\"RUNNING\",\"Capacity\":200,\"Master\":{\"IP\":\"10.11.12.123\",\"Port\":\"6382\",\"MemoryCapacity\":200,\"MemoryUsed\":1904432,\"Uptime\":1623,\"ClientsConnected\":1,\"LastSyncedToMaster\":0},\"Slaves\":[{\"IP\":\"10.11.12.121\",\"Port\":\"6381\",\"MemoryCapacity\":200,\"MemoryUsed\":834904,\"Uptime\":1619,\"ClientsConnected\":2,\"LastSyncedToMaster\":9},{\"IP\":\"10.11.12.121\",\"Port\":\"6382\",\"MemoryCapacity\":200,\"MemoryUsed\":834904,\"Uptime\":1619,\"ClientsConnected\":2,\"LastSyncedToMaster\":9}]}\n```\nOR try the CLI\n```\n$mrr status -n TestInstance\nStatus = RUNNING\nType = MS\nCapacity = 200\nMaster = 10.11.12.123:6382\n        Slave0 = 10.11.12.121:6381\n        Slave1 = 10.11.12.121:6382\n```\n\n#### Connecting to an Instance\nFrom the above json response information related to Master are as follows.\n```\n\t\"Master\": {\n\t\t\"IP\": \"10.11.12.123\",\n\t\t\"Port\": \"6382\",\n\t\t\"MemoryCapacity\": 200,\n\t\t\"MemoryUsed\": 1904432,\n\t\t\"Uptime\": 1623,\n\t\t\"ClientsConnected\": 1,\n\t\t\"LastSyncedToMaster\": 0\n\t}\n```\nYou could use any redis [client] (http://redis.io/clients) and connect to the master or use the redis-cli to test the instance.\n```\n$redis-cli -h 10.11.12.123 -p 6382\n10.11.12.123:6381\u003e set foo bar\nOK\n10.11.12.123:6381\u003e get foo\n\"bar\"\n10.11.12.123:6381\u003e exit\n```\n\n## Starting the Scheduler (not applicable to DC/OS users)\nMrRedis scheduler binary is usually referred as `sched`, the scheduler hosts a file-server which can distribute redis binary and custom Executor.\n\nThe scheduler takes only one argument which is the config file,\n\n```\n$./sched -config=\"./config.json\"\n2016/01/17 16:35:11 *****************************************************************\n2016/01/17 16:35:11 *********************Starting MrRedis-Scheduler******************\n2016/01/17 16:35:11 *****************************************************************\n2016/01/17 16:35:11 Starting the HTTP server at port 8080\n.\n.\n```\nThe configuration file should be of json format, below is an example\n```\n$cat config.json\n{\n   \"UserName\": \"ubuntu\",\n   \"FrameworkName\": \"MrRedis\",\n   \"Master\": \"127.0.0.1:5050\",\n   \"ExecutorPath\": \"./MrRedisExecutor\",\n   \"RedisImage\": \"redis:3.0-alpine\",\n```\nyou can configure Zookeeper as the KV store \n```\n   \"DBType\": \"zookeeper\",, \n   \"DBEndPoint\": \"127.0.0.1:2181\"\", \n```\nOR use ETCD\n```\n   \"DBType\": \"etcd\", \n   \"DBEndPoint\": \"127.0.0.1:2379\", \n```\n```   \n   \"LogFile\": \"stderr\",\n   \"ArtifactIP\": \"127.0.0.1\",\n   \"ArtifactPort\": \"5454\",\n   \"HTTPPort\": \"5656\"\n}\n```\nPlease substitute appropriate values with respect to your environment  for MasterIP/Port, ExecutorPath, DBEndPoint and IP address of this scheduler's VM that is accessible from the slaves for artifactIP\n\nif you want to get an empty config for you to start working on you could do this and the scheduler will print you a dummy structure for you to start working on.\n```\n$./sched -DumpEmptyConfig\n{\n   \"UserName\": \"ubuntu\",\n   \"FrameworkName\": \"MrRedis\",\n   \"Master\": \"127.0.0.1:5050\",\n   \"ExecutorPath\": \"./MrRedisExecutor\",\n   \"RedisImage\": \"redis:3.0-alpine\",\n   \"DBType\": \"etcd\",\n   \"DBEndPoint\": \"127.0.0.1:2379\",\n   \"LogFile\": \"stderr\",\n   \"ArtifactIP\": \"127.0.0.1\",\n   \"ArtifactPort\": \"5454\",\n   \"HTTPPort\": \"5656\"\n}\n```\n\n### Distribution of redis processes among Mesos Agents.\n'Distribution' : An integer representing number of 'redis-servers' can be start in One agent belonging to this Instance\nThis attribute needs to be supplied while you create an Instances, by default the distribution Value is set to 1.\n#### Scenario \nMesos cluster with three Agents A1, A2 and A3 then\n#### Case 1:\nIf you create an instance with M and 2 Slaves namely S1 \u0026 S2 (default Distribution Value)\n\n| A1 | A2 | A3 |\n| --- | --- | --- |\n| M | S1 | S1 |\n\nroughly should be the distribution,\n#### Case 2:\nIf you create an Redis Instance with M and 5 Slaves namely S1,S2,S3,S4,S5 (distribution Value is 2)\n\n| A1 | A2 | A3 |\n| --- | --- | --- |\n| M | S1 | S1 |\n| S3 | S4 | S5 |\n\nThe json file to be supplied along with 'CREATE' call should be like below\n```\ncat d.json\n{\n        \"Distribution\": 2\n}\n```\nThe cli(mrr) create sub-command has a new `-f` option to supply addition json file. If you are using curl to create instances then supply it like this\n```\ncurl http://\u003cENDPOINT\u003e/v1/CREATE/testInstance/100/1/2 -X \"POST\" --header \"Content-Type: application/json\" -d @d.json\n```\n### Creating Instances\nThis can be done in 3 ways.\n\u003cimg src=\"./CreateInstance.gif\" width=\"100%\" height=\"100%\"\u003e \n\n### Creating Multiple Redis Instances:\nHow much time does it take to create 70 single redis instances ?\n\u003cimg src=\"./MultiInstanceCreation.gif\" width=\"100%\" height=\"100%\"\u003e \n\n### Master-Slave Promotion:\nScheduler automatically promotes a slave when a master fails.\n\u003cimg src=\"./MasterSlavePromotion.gif\" width=\"100%\" height=\"100%\"\u003e \n\n## Using the CLI\nmr-redis has built-in cli for creating and destroying redis instances.\n\nCLI should first be initialized with the scheduler with this simple command\n\n```\n$mrr init http://\u003cschedulerIP\u003e:\u003cschedulerPORT\u003e\n```\nIf you want to explore other commands below is the help screen\n```\n$mrr help\nNAME:\n   mrr - MrRedis Command Line Interface\n\nUSAGE:\n   mrr [global options] command [command options] [arguments...]\n\nVERSION:\n   0.0.0\n\nCOMMANDS:\n   init, i      $mrr init \u003chttp://MrRedisEndPoint\u003e\n   create, c    Create a Redis Instance\n   status, s    Status of a Redis Instance\n   delete, d    Delete a Redis Instance\n   help, h      Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --help, -h   show help\n```\n\nHelp on a specific command\n```\n$mrr help create\nNAME:\n   mrr create - Create a Redis Instance\n\nUSAGE:\n   mrr create [command options] [arguments...]\n\nOPTIONS:\n   --name, -n           Name of the Redis Instance\n   --memory, -m \"0\"     Memory in MB\n   --slaves, -s \"0\"     Number of Slaves\n   --wait, -w           Wait for the Instance to be created (by default the command is async)\n   --file, -f           Location of the config file for this instance, to specify Distribution Value\n```\n\n## More Examples of Using the CLI\n### Example 1:\nThe cli itself will be async by default as it does not wait for the operation to complete\n\n```\n$mrr create -n testInst -m 200 -s 1\nAttempting to Creating a Redis Instance (testInst) with mem=200 slaves=1\nInstance Creation accepted..\nCheck $mrr status -n testInst for status\n```\n\n```\n$mrr status -n testInst\nStatus = RUNNING\nType = MS\nCapacity = 200\nMaster = 10.11.12.33:6389\n        Slave0 = 10.11.12.32:6380\n```\n\n### Example 2:\nIf you have a complicated Redis requirement then a simple command like the one below will result in creating one redis instance with 1 master and 50 Slaves in less than 15 secs, Simple :-)\n```\n$time mrr create -n hello50 -m 100 -s 50 -w true\nAttempting to Creating a Redis Instance (hello50) with mem=100 slaves=50\nInstance Creation accepted................\nInstance Created.\n\nreal    0m14.269s\nuser    0m0.033s\nsys     0m0.037s\n```\nTo find the status of the redis instance you have created, below is the command\n```\n$mrr status -n hello50\nStatus = RUNNING\nType = MS\nCapacity = 100\nMaster = 10.11.12.21:6380\n        Slave0 = 10.11.12.31:6381\n        Slave1 = 10.11.12.31:6383\n        Slave2 = 10.11.12.31:6384\n        Slave3 = 10.11.12.31:6385\n        Slave4 = 10.11.12.31:6382\n        Slave5 = 10.11.12.31:6386\n        Slave6 = 10.11.12.31:6387\n        Slave7 = 10.11.12.31:6388\n        Slave8 = 10.11.12.31:6391\n        Slave9 = 10.11.12.31:6392\n        Slave10 = 10.11.12.31:6390\n        Slave11 = 10.11.12.31:6389\n        Slave12 = 10.11.12.31:6393\n        Slave13 = 10.11.12.31:6394\n        Slave14 = 10.11.12.31:6395\n        Slave15 = 10.11.12.20:6380\n        Slave16 = 10.11.12.20:6381\n        Slave17 = 10.11.12.20:6383\n        Slave18 = 10.11.12.20:6384\n        Slave19 = 10.11.12.20:6387\n        Slave20 = 10.11.12.20:6385\n        Slave21 = 10.11.12.20:6386\n        Slave22 = 10.11.12.20:6382\n        Slave23 = 10.11.12.29:6380\n        Slave24 = 10.11.12.29:6381\n        Slave25 = 10.11.12.29:6382\n        Slave26 = 10.11.12.29:6384\n        Slave27 = 10.11.12.29:6385\n        Slave28 = 10.11.12.29:6383\n        Slave29 = 10.11.12.29:6387\n        Slave30 = 10.11.12.29:6386\n        Slave31 = 10.11.12.29:6389\n        Slave32 = 10.11.12.29:6391\n        Slave33 = 10.11.12.29:6392\n        Slave34 = 10.11.12.29:6388\n        Slave35 = 10.11.12.29:6390\n        Slave36 = 10.11.12.29:6394\n        Slave37 = 10.11.12.29:6395\n        Slave38 = 10.11.12.29:6393\n        Slave39 = 10.11.12.21:6383\n        Slave40 = 10.11.12.21:6384\n        Slave41 = 10.11.12.21:6386\n        Slave42 = 10.11.12.21:6385\n        Slave43 = 10.11.12.21:6387\n        Slave44 = 10.11.12.21:6388\n        Slave45 = 10.11.12.21:6390\n        Slave46 = 10.11.12.21:6389\n        Slave47 = 10.11.12.21:6391\n        Slave48 = 10.11.12.21:6381\n        Slave49 = 10.11.12.21:6382\n```\n\n### Proxy in Progress \nWe are writing a proxy that will be installed with every redis instances especially with Master-Slave setup.  The proxy should be a simple pass-through.  Should be capable of accepting new configuration changes without needing to restart it.  Below is some performance stats comparing different available proxies and ours.\n\n```\n$redis-benchmark -h \u003cIP\u003e -p \u003cPORT\u003e  -q -r 100000\n```\n\n\n\u003cimg src=\"./ProxyCompare.PNG\" width=\"100%\" height=\"100%\"\u003e \n\n\n### Contribution Guidelines\nGo already provides a nice documentation on coding conventions and guidelines; just try to adhere to that [Effective Go](https://golang.org/doc/effective_go.html) :-) \n\n##Specifically \n- Format code using go fmt, if an already prebuilt auto formatter is not their in your editor\n- We suggest using extensive comments, as this code base is still evolving\n- Try to stress on error handling as per [Effective error handling in Go](https://golang.org/doc/effective_go.html#errors) (which we ourselves have probably missed at places)\n- Please use this framework; We are looking forward for issues, and nothing greater than an issue and a fix. Nonetheless, if interested in contributing something specific, please raise an issue outright to let us know that you are doing \"this\"\n- We have not set up tests and test code yet, this is one obvious area to contribute without saying.\n\n### Future Plans\n\n- [ ] Support REDIS 3.0 Cluster \n- [ ] Support a Proxy mechanism to expose Redis Instance Endpoint\n- [ ] Build a UI for Create/Maintain/Monitor the entire redis framework\n- [ ] Benchmark Utility for testing the RedisFramework \n\n##License\nCopyright 2015 Huawei Technologies Co. Ltd.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this project except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmesos%2Fmr-redis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmesos%2Fmr-redis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmesos%2Fmr-redis/lists"}