{"id":17294789,"url":"https://github.com/linsamtw/rabbitmqcelery","last_synced_at":"2025-03-26T19:43:58.472Z","repository":{"id":112933516,"uuid":"155801506","full_name":"linsamtw/RabbitmqCelery","owner":"linsamtw","description":"Rabbitmq \u0026 Celery install, example and tools.","archived":false,"fork":false,"pushed_at":"2019-12-03T07:35:46.000Z","size":78,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-01T01:51:22.638Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linsamtw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-02T02:09:56.000Z","updated_at":"2023-01-10T10:12:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"9ee14a9c-dcae-4d86-a519-c0fad4466b4c","html_url":"https://github.com/linsamtw/RabbitmqCelery","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsamtw%2FRabbitmqCelery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsamtw%2FRabbitmqCelery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsamtw%2FRabbitmqCelery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsamtw%2FRabbitmqCelery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linsamtw","download_url":"https://codeload.github.com/linsamtw/RabbitmqCelery/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245727635,"owners_count":20662552,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T11:08:05.995Z","updated_at":"2025-03-26T19:43:58.443Z","avatar_url":"https://github.com/linsamtw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rabbitmq \u0026 Celery\n\nIt is a Distributed queue system, you can send many jobs and many workers will do job for you. And you can monitor them.\u003cbr\u003e\n\n**note** : If you only have one computer, the **linode** is good support, the min calculator only need $5/month, it is enough for being worker.\n* [Rabbitmq \u0026 Celery](https://github.com/linsamtw/RabbitmqCelery#rabbitmq--celery-1)\n    *  [Install](https://github.com/linsamtw/RabbitmqCelery#install-rabbitmq)\n    *  [Create Account](https://github.com/linsamtw/RabbitmqCelery#create-web-account)\n    *  [Example](https://github.com/linsamtw/RabbitmqCelery#Example)\n    *  [Run](https://github.com/linsamtw/RabbitmqCelery#on-node)\n    *  [Return Result](https://github.com/linsamtw/RabbitmqCelery#return-result-by-apply_async)\n    *  [Set Queue Group](https://github.com/linsamtw/RabbitmqCelery#set-queue-group)\n* [Flower](https://github.com/linsamtw/RabbitmqCelery#flower)\n    *  [Install](https://github.com/linsamtw/RabbitmqCelery#install)\n* [Supervisor](https://github.com/linsamtw/RabbitmqCelery#supervisor)\n    *  [Install](https://github.com/linsamtw/RabbitmqCelery#install-1)\n* [Git](https://github.com/linsamtw/RabbitmqCelery#git)\n    *  [Git No Need Password](https://github.com/linsamtw/RabbitmqCelery#git-no-need-password)\n    *  [change github pull and push to no need password](https://github.com/linsamtw/RabbitmqCelery#change-github-pull-and-push-to-no-need-password)\n    *  [git command](https://github.com/linsamtw/RabbitmqCelery#git-command)\n*  [Other](https://github.com/linsamtw/RabbitmqCelery#other)\n    *  [Run Celery On Python3](https://github.com/linsamtw/RabbitmqCelery#run-celery-on-python3)\n    *  [Kill Process](https://github.com/linsamtw/RabbitmqCelery#kill-process)\n    *  [Set Watchdog](https://github.com/linsamtw/RabbitmqCelery#set-watchdog)\n    *  [MySQL-PARTITION](https://github.com/linsamtw/RabbitmqCelery#mysql-partition)\n\n----------------------------\n## Rabbitmq \u0026 Celery\n#### Install RabbitMQ\n    sudo apt-get update \n    sudo apt-get install erlang\n    sudo apt-get install rabbitmq-server\n    sudo rabbitmq-plugins enable rabbitmq_management# web running\n\nthen we can connect rabbitmq on\nhttp://IP:15672/  or http://localhost:15672/  \n\u003c!--if ERROR: node with name \"rabbit\" already running on \"localhost\"\nhttps://www.cnblogs.com/Sisiflying/p/6386988.html --\u003e\n\n#### Install Celery\n    apt-get install python3-pip\n    export LC_ALL=\"en_US.UTF-8\"\n    pip3 install celery\n\n------------------\n#### Create Web Account\n\n    cd /user/sbin\n    ./rabbitmqctl set_user user password # set user and password\n    ./rabbitmqctl set_user_tags user administrator # set user permission\n    ./rabbitmqctl set_permissions -p / user \".*\" \".*\" \".*\"     # set connect IP\nor\n\n    rabbitmqctl add_user user password # set user and password\n    rabbitmqctl set_user_tags user administrator # set user permission\n    rabbitmqctl set_permissions -p / user \".*\" \".*\" \".*\"     # set connect IP\n\n#### Create Worker Account\n\n    cd /user/sbin\n    ./rabbitmqctl add_user worker_user worker_password\n    ./rabbitmqctl set_user_tags worker_user policymaker\n    ./rabbitmqctl set_permissions -p / worker_user \".*\" \".*\" \".*\"   \nor\n\n    rabbitmqctl add_user worker_user worker_password\n    rabbitmqctl set_user_tags worker_user policymaker\n    rabbitmqctl set_permissions -p / worker_user \".*\" \".*\" \".*\"   \n    \nThe queues on rabbitmq web only can appear by worker user.\n\n------------------\n\n#### Example\nThe Distributed queue system has three roles.\n\n| Role | Job |\n|------|-----|\n|Worker|Get tasks from rabbitmq and do tasks, you maybe have more one workers.|\n|Broker|Rabbitmq server, transfer tasks.|\n|Producer|Push tasks to rabbitmq.|\n\nThe **worker** and **producer** should be different computers.\n\n------------------\nCreate Worker.py, \n\n    from celery import Celery\n    app = Celery(\"task\",\n                 include=[\"Tasks\"],# tasks file name\n                 broker='pyamqp://worker_user:worker_password@Rabbitmq_IP:5672/')\n\nCreate Tasks.py, \n\n    from Worker import app\n    @app.task()\n    def add(x,y):\n        return x+y\n        \nCreate Producer.py\n\n    from Tasks import add\n    add.apply_async(0,0)# \n    \n------------------\n#### On Node\nThe Worker role, run this command to get tasks.\n\n    celery -A Worker worker --loglevel=info\n#### On Server\nThe Producer and Broker role, run this command to push\n    \n    python3 Producer.py\n-------------------------------\n#### Return Result By apply_async\n    app = Celery(\"task\",\n                 include=[\"Tasks\"],# tasks file name\n                 backend='rpc://',\n                 broker='pyamqp://worker_user:worker_password@Rabbitmq_IP:5672/')\n-------------------------------\nServer need install : rabbitmq-server, celery\n\u003c!--crontab : git clone url, python3 job--\u003e\n\nNode need install : celery\n\u003c!--crontab : git clone url\nvim /etc/rc.local # it will run on boot--\u003e\n-------------------------------\n#### Set Queue Group\n* Send tasks : @app.task change to @app.task( queue = **queue_group_name** )\n* Login web : worker user, then queues will show tasks group by **queue_group_name**\n* Get tasks : celery -A task worker --loglevel=info -Q **queue_group_name** \n\n## Flower\n\nFlower is a monitor system, it can monitor workers.\n#### install\n    pip3 install flower\n#### Run\n\n    # web running\n    flower -A Worker --port=5555\n    #Node run this command\n    celery -A Worker worker --loglevel=info\n\n## Supervisor\nIt must be install by root.\n\n      sudo su\n      python # Running celery, it it run this python version.\nYou can install Anaconda(3) to control your version.\n\n      Do you wish the installer to prepend the Anaconda3 install location\n      to PATH in your /root/.bashrc ? [yes|no]\n      yes\n\n#### install\n    sudo apt-get install supervisor\n#### Running\n\n      sudo systemctl enable supervisor \n      sudo systemctl disable supervisor\n      sudo systemctl status supervisor \n\n      sudo service supervisor start\n      sudo service supervisor stop\n      sudo service supervisor restart\n#### Set\n    \n    sudo vim /etc/supervisor/conf.d/celery_*.conf\n\n    [program:celeryd_1]\n    directory=your directory path\n    command=celery -A tasks worker --loglevel=error --maxtasksperchild=1 --concurrency=8 --hostname=%%h_1 -Q seat_float\n    stdout_logfile=/var/log/celery/celeryd_1.log\n    stderr_logfile=/var/log/celery/celeryd_1.err\n    autostart=true\n    autorestart=true\n    startsecs=10\n    stopwaitsecs=600\n    stopasgroup=true\n    killasgroup=true\n-------------------------------\n## Git\n#### Git No Need Password\nPaste your SSH key from  '~/.ssh/id_rsa.pub'. If you have no id_rsa.pub, you can follow that to generate ssh key.\u003cbr\u003e\n\n    cd ~/.ssh/\n    ssh-keygen\nthen, paste to gitlab or github -\u003e option -\u003e ssh key\u003cbr\u003e\n\n#### change github pull and push to no need password\n\n    git remote -v #show git is follow https or ssh. push or pull by no password must be ssh.\n    #You can change the URL with:\n    git remote set-url origin git+ssh://git@github.com/username/reponame.git\n#### git command \n      # change loacal branch\n      git checkout master \n      \n      # check local branch version \n      git branch \n      \n      # create a new branch\n      git branch test \n      \n      # push branch to gitlab or github\n      git push origin test \n      \n      # merge \n      git checkout master \n      git merge test\n      \n      # delete\n      git branch -d \u003cbranch\u003e\n      \n      # back to last version\n      git reset --hard HEAD^\n      git push --force\n      \n      # add tag\n      git log --oneline\n      git tag tagname 975a240\n      # del local tag\n      git tag -d tagname\n      # del online tag\n      git push --delete origin tagname\n      # push tag\n      git push origin tagname\n      # push all tag\n      git push origin --tags\n      # checkout\n      git checkout tagname\n      \n-------------------------------\n## Other\n#### Run celery on Python3\n\n    pip3 install virtualenv \n    virtualenv celery_for_python3\n    source celery_for_python3/bin/activate\n#### Kill Process\n\n    ps aux | grep celery | awk '{print $2}' | xargs # print PID\n    ps aux | grep celery | awk '{print $2}' | xargs kill -9 # print PID and kill Process\n\n\n#### Set Watchdog\nIf your celery on running, but the code will be changed.\nThe celery process can't update, because the code has be compiled.\nThe Watchdog which provides watchmedo, it can re-loading code into Celery after a change.\n\n    pip3 install watchdog\n    watchmedo auto-restart -- celery -A task worker --loglevel=info\n\nIf temperature too high or memory too less, then reboot.\n\n#### Worker\nLinode or Raspberry Pi will install our python packages, we will have packages list.\n\n\n#### Set Linode System Date\nThe date from linode is follow US. If you are Taiwan, you can change the date from the command\n\n      sudo dpkg-reconfigure tzdata\n#### update package\n   \n      python3 setup.py sdist\n      twine upload dist/*\n      \n#### MySQL PARTITION\ncreate table and PARTITION\n\n      CREATE TABLE t (\n         id INT, \n         create_time DATETIME\n      )    \n      PARTITION BY RANGE(YEAR(create_time)) (\n         PARTITION p2017 VALUES LESS THAN (2018),\n         PARTITION p2018 VALUES LESS THAN (2019),\n         PARTITION p2019 VALUES LESS THAN (2020)\n      );\n\nadd PARTITION\n      \n      ALTER TABLE test ADD PARTITION (PARTITION p2020 VALUES LESS THAN (2021))\n      \n\u003c!--\n\n# git\n*/1 * * * *   pi  bash /home/pi/bin/autoGitForCrawlerTg.sh\n\nWorker\n\nsudo vim /etc/supervisor/conf.d/worker.conf\n\n[program:celeryd_1]\ndirectory=/worker/gitlab/FinancialMining/Tasks\ncommand=celery -A Worker worker --loglevel=info --maxtasksperchild=1 --concurrency=8 --hostname=%h_1\nstdout_logfile=/var/log/celery/worker.log\nstderr_logfile=/var/log/celery/worker.err\nautostart=true\nautorestart=true\nstartsecs=10\nstopwaitsecs=600\nstopasgroup=true\nkillasgroup=true\n\n\n### Rabbitmq\u0026Flower\n\nsudo vim /etc/supervisor/conf.d/flower.conf\n\n      [program:flowerd]\n      command=celery flower --broker=amqp://worker:worker@172.105.212.230:5672/\n      stdout_logfile=/var/log/celery/flowerd.log\n      stderr_logfile=/var/log/celery/flowerd.err\n      autostart=true\n      autorestart=false\n      startsecs=10\n      stopwaitsecs=600\n\nsudo vim /home/worker/gitlab/autogit.sh \n\n      cd your project path\n      git pull\n\n\n\n\nsudo vim /etc/crontab\n\n# git\n*/1 * * * *   root  bash /worker/gitlab/autogit.sh\n\n\n--\u003e\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsamtw%2Frabbitmqcelery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinsamtw%2Frabbitmqcelery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsamtw%2Frabbitmqcelery/lists"}