{"id":13398999,"url":"https://github.com/gleicon/restmq","last_synced_at":"2025-04-05T12:06:23.004Z","repository":{"id":732179,"uuid":"381567","full_name":"gleicon/restmq","owner":"gleicon","description":"REST/JSON/HTTP based message queue","archived":false,"fork":false,"pushed_at":"2016-03-20T09:24:29.000Z","size":417,"stargazers_count":556,"open_issues_count":5,"forks_count":70,"subscribers_count":35,"default_branch":"master","last_synced_at":"2024-07-31T19:17:52.817Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://zenmachine.wordpress.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gleicon.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2009-11-22T13:59:46.000Z","updated_at":"2024-07-09T09:44:43.000Z","dependencies_parsed_at":"2022-07-05T13:13:05.693Z","dependency_job_id":null,"html_url":"https://github.com/gleicon/restmq","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gleicon%2Frestmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gleicon%2Frestmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gleicon%2Frestmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gleicon%2Frestmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gleicon","download_url":"https://codeload.github.com/gleicon/restmq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247332604,"owners_count":20921853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T19:00:33.329Z","updated_at":"2025-04-05T12:06:22.983Z","avatar_url":"https://github.com/gleicon.png","language":"Python","funding_links":[],"categories":["Python","Redis based Job Queues"],"sub_categories":[],"readme":"======\nRESTMQ\n======\n\nRedis based message queue.\n--------------------------\n\n:Info: See `RestMQ website \u003chttp://restmq.com\u003e`_\n:Info: See `my blog \u003chttp://blog.7co.cc\u003e` for more information.\n:Author: `Gleicon Moraes \u003chttp://github.com/gleicon\u003e`\n:Author: `Alexandre Fiori \u003chttp://github.com/fiorix/\u003e`_\n\n\nAbout\n=====\n\nRestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST\nresources. It stands on the shoulder of giants, built over Python, Twisted, `Cyclone \u003chttp://github.com/fiorix/cyclone\u003e`_ (a Tornado implementation over twisted) and Redis.\n\nRedis is more than just a key/value db, and its data types provided support for this project.\n\nQueues are created on the fly, when a message is sent to them. They are simple to use as a curl request can be. HTTP Comet, websockets and server sent events may be used for data streaming.\n\nThis release is a cleanup of the original code, that can also be found on github.\n\nInstall\n=======\n\n$ python setup.py install\nrun with bash start_scripts/restmq_server or taylor your own script. Note that currently restmq is presented as a twisted plugin.\n\nAlternatively you can utilize [Vagrant](https://www.vagrantup.com/) and our [Vagrantfile](Vagrantfile) which will handle installation and configuration of redis and RestMQ.\n\nAnother option is to run the project inside a docker container:\n\n```\ndocker build -t restmq .\ndocker run --rm  -p 6379:6379 -p 8888:8888 restmq\n```\n\nIt will run restmq, exposing both 8888 (for its http port) and 6379 (for its redis server) to the host environment.\n\nQueue Policy\n============\n\nEvery queue is created with a default policy, which is `broadcast`. It means that each message\npushed to a queue will be forwarded to all comet and websocket consumers.\n\nThe alternative policy is `roundrobin`, which will distribute these messages in a round robin\nfashion to all comet and websocket consumers.\n\nThe queue policy won't affect regular GET commands to pop a single message from a queue.\n\nSee `README.qp \u003chttp://github.com/gleicon/restmq/blob/master/README.qp\u003e`_ for details.\n\n\nQueue Flow Control\n==================\n\nYes, it does support start and stop. We just need to document it properly.\n\n\nExample\n========\nA http client (curl) post to /queue:\n\nPoint your browser to http://localhost:8888/c/test\n\nRun $ curl -d \"queue=test\u0026value=foobar\" http://localhost:8888/\n\nYour browser is acting as a consumer to the queue. Using json encoded data it's easy to fit the data into a js based app.\n\nAside from the COMET consumer, there are xmlrpc methods, rest routes and the JSON protocol to manipulate queue items.\n\n\nCOMET consumer\n==============\n\nThere is a COMET based consumer, which will bind even if the queue doesn't already exists.\n\nThe main route is thru /c/\u003cqueuename\u003e. It can be tested using curl:\n\n$ curl http://localhost:8888/c/test\n\nIn another terminal, run $ curl -d \"value=foobar\" http://localhost:8888/q/test\n\nThis is the basic usage pattern for map/reduce (see examples).\n\nSee below on how to purge and disconnect all consumers from a queue, using DELETE.\n\n\nWebSocket consumer\n==================\n\nNow that cyclone has websockets support, check README.websocket to test it.\n\nIf you are using a browser or library which already supports websockets, you may take advantage of this interface.\n\n\nREST services\n=============\n\nA queue can be accessed as /q/\u003cqueuename\u003e.\n\nGET requests will dequeue an object.\n\nPOST requests inserts an object in the queue\n\nDELETE requests will purgue the queue.\n\nThe usual pattern is listen in the COMET consumer (/c/\u003cqueuename\u003e) and insert new stuff at the REST route (POST /q/\u003cqueuename).\n\n\nJSON Protocol\n=============\n\nThe HTTP route /queue/\u003cqueuename\u003e uses the JSON protocol. Its the same protocol I've implemented for http://jsonqueue.appspot.com.\n\n::\n\n    {\n        \"cmd\": \"add\",\n        \"queue\": \"genesis\",\n        \"value\": \"abacab\"\n    }\n\nCreates the queue named *\"genesis\"* and inserts the string *\"abacab\"* as the message.\n\nIf we want to take that out of the queue, the payload would be like that:\n\n::\n\n    {\n        \"cmd\": \"take\",\n        \"queue\": \"genesis\"\n    }\n\n\nThe message can be formatted as a json object, so more complex data can be sent.\nIt really mimics some of `Amazon SQS \u003chttp://aws.amazon.com/sqs/\u003e`_ workings, because it's a simple queue.\n\nFor the first release it has:\n\n- Select, EPoll or KQueue concurrency (depends on twisted)\n- Persistent storage using Redis\n- Can work on pools, N daemons consuming from the same queues.\n- Small codebase\n- Lightweight\n- Cute ?\n\n\nDependencies\n============\n- `cyclone \u003chttp://github.com/fiorix/cyclone\u003e`_:\n  git clone git://github.com/fiorix/cyclone.git\n\n\nRunning\n=======\n\nThe `redis_server \u003chttp://github.com/gleicon/restmq/blob/master/restmq_server\u003e`_ script will start the service. It's a bash script used to both configure and run RestMQ. The default version of the wrapper script will run the server in foreground, and log messages will be written to the standard output.\n\nEditing the script is mandatory for configuring RestMQ for production.\n\n::\n\n    $ ./restmq_server --help\n    Usage: twistd [options] restmq [options]\n    Options:\n          --acl=         acl configuration file for endpoints [default: acl.conf]\n          --redis-host=  hostname or ip address of the redis server [default: 127.0.0.1]\n          --redis-port=  port number of the redis server [default: 6379]\n          --redis-pool=  connection pool size [default: 10]\n          --port=        port number to listen on [default: 8888]\n          --listen=      interface to listen on [default: 127.0.0.1]\n          --version\n          --help         Display this help and exit.\n\n\nTests\n=====\n\n::\n\n    examples/test_rest.sh\n    examples/test_xmlrpc.py\n    python examples/test_comet.py\n    python examples/twitter_trends.py\n    python examples/test_comet_curl.py\n    python restmq_engine.py -h\n\n\nFiles\n=====\n\nIf you're a developer looking for extending RestMQ's functionality, have a look at these files:\n\n- `restmq/web.py \u003chttp://github.com/gleicon/restmq/blob/master/restmq/web.py\u003e`_: the web service code\n- `restmq/core.py \u003chttp://github.com/gleicon/restmq/blob/master/restmq/core.py\u003e`_: redis/queue operations logic\n- `restmq/dispatch.py \u003chttp://github.com/gleicon/restmq/blob/master/restmq/dispatch.py\u003e`_: a simple command dispatcher\n- `restmq_engine.py \u003chttp://github.com/gleicon/restmq/blob/master/restmq_engine.py\u003e`_: the redis abstraction layer to the queue algorithm (command line tool)\n\n\nCredits\n=======\nThanks to (in no particular order):\n\n- Salvatore Sanfilippo for redis and for NoSQL patterns discussion.\n- Alexandre Fiori for the redis client enhancement and patches.\n- Roberto Gaiser for the collectd daemon\n- Vitor Pellegrino for dockerizing restmq\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgleicon%2Frestmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgleicon%2Frestmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgleicon%2Frestmq/lists"}