{"id":13880603,"url":"https://github.com/kvz/cronlock","last_synced_at":"2025-04-09T14:14:26.579Z","repository":{"id":3491164,"uuid":"4547247","full_name":"kvz/cronlock","owner":"kvz","description":"cronlock lets you deploy cronjobs cluster-wide without worrying about overlaps. It uses Redis to keep track of locks.","archived":false,"fork":false,"pushed_at":"2019-07-10T09:44:21.000Z","size":107,"stargazers_count":270,"open_issues_count":0,"forks_count":46,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-02T12:13:28.007Z","etag":null,"topics":["cron-jobs","locking","redis","shell"],"latest_commit_sha":null,"homepage":"http://kvz.io/blog/2012/12/31/lock-your-cronjobs/","language":"Shell","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kvz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-06-04T13:14:14.000Z","updated_at":"2025-02-11T21:31:28.000Z","dependencies_parsed_at":"2022-09-08T09:11:57.939Z","dependency_job_id":null,"html_url":"https://github.com/kvz/cronlock","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kvz%2Fcronlock","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kvz%2Fcronlock/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kvz%2Fcronlock/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kvz%2Fcronlock/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kvz","download_url":"https://codeload.github.com/kvz/cronlock/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248054193,"owners_count":21039952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cron-jobs","locking","redis","shell"],"created_at":"2024-08-06T08:03:16.313Z","updated_at":"2025-04-09T14:14:26.562Z","avatar_url":"https://github.com/kvz.png","language":"Shell","funding_links":["https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=kevin%40vanzonneveld%2enet\u0026lc=NL\u0026item_name=Open%20source%20donation%20to%20Kevin%20van%20Zonneveld\u0026currency_code=USD\u0026bn=PP-DonationsBF%3abtn_donate_SM%2egif%3aNonHosted"],"categories":["Shell"],"sub_categories":[],"readme":"# cronlock\n\n\u003c!-- badges/ --\u003e\n[![Build Status](https://secure.travis-ci.org/kvz/cronlock.svg?branch=master)](http://travis-ci.org/kvz/cronlock \"Check this project's build status on TravisCI\")\n[![Gittip donate button](http://img.shields.io/gittip/kvz.svg)](https://www.gittip.com/kvz/ \"Sponsor the development of cronlock via Gittip\")\n[![Flattr donate button](http://img.shields.io/flattr/donate.png?color=yellow)](https://flattr.com/submit/auto?user_id=kvz\u0026url=https://github.com/kvz/cronlock\u0026title=cronlock\u0026language=\u0026tags=github\u0026category=software \"Sponsor the development of cronlock via Flattr\")\n[![PayPayl donate button](http://img.shields.io/paypal/donate.png?color=yellow)](https://www.paypal.com/cgi-bin/webscr?cmd=_donations\u0026business=kevin%40vanzonneveld%2enet\u0026lc=NL\u0026item_name=Open%20source%20donation%20to%20Kevin%20van%20Zonneveld\u0026currency_code=USD\u0026bn=PP-DonationsBF%3abtn_donate_SM%2egif%3aNonHosted \"Sponsor the development of cronlock via Paypal\")\n[![BitCoin donate button](http://img.shields.io/bitcoin/donate.png?color=yellow)](https://coinbase.com/checkouts/19BtCjLCboRgTAXiaEvnvkdoRyjd843Dg2 \"Sponsor the development of cronlock via BitCoin\")\n\u003c!-- /badges --\u003e\n\n## Install\n\nOn most Linux \u0026 BSD machines, cronlock will install just by downloading it \u0026 making it executable.\nHere's the one-liner:\n\n```bash\nsudo curl -q -L https://raw.github.com/kvz/cronlock/master/cronlock -o /usr/bin/cronlock \u0026\u0026 sudo chmod +x $_\n```\n\nWith [Redis](http://redis.io/) present on `localhost`, cronlock should now already work in basic form.\nLet's test by letting it execute a simple `pwd`:\n\n```bash\nCRONLOCK_HOST=localhost cronlock pwd\n```\n\nIf this returns the current directory we're good to go. More examples below.\n\n## Introduction\n\nUses a central [Redis](http://redis.io/) server to globally lock cronjobs across a distributed system.\nThis can be usefull if you have 30 webservers that you deploy crontabs to (such as\nmailing your customers), but you don't want 30 cronjobs spawned.\n\nOf course you could also deploy your cronjobs to 1 box, but in volatile environments\nsuch as EC2 it can be helpful not to rely on 1 'throw away machine' for your scheduled tasks,\nand have 1 deploy-script for all your workers.\n\nAnother common problem that cronlock will solve is overlap by a single server/cronjob.\nIt happens a lot that developers underestimate how long a job will run.\nThis can happen because the job waits on something, acts different under high load/volume, or enters an endless loop.\n\nIn these cases you don't want the job to be fired again at the next cron-interval, making your problem twice as bad,\nsome intervals later, there's a huge `ps auxf` with overlapping cronjobs, high server load, and eventually a crash.\n\nBy settings locks, cronlock can also prevent the overlap in longer-than-expected-running cronjobs.\n\n## Design goals\n\n - Lightweight\n - As little dependencies as possible / No setup\n - Follows locking logic from [this Redis documentation](http://redis.io/commands/setnx)\n - Well tested \u0026 documented\n\n## Requirements\n\n - Bash (version 3.0 and above) with `/dev/tcp` enabled. Older Debian/Ubuntu systems disable `/dev/tcp`\n - `md5` or `md5sum`\n - A [Redis](http://redis.io/) server, or cluster of servers that is accessible by all cronlock machines\n\n## Options\n\n - `CRONLOCK_CONFIG` location of config file. this is optional since all config can also be\n passed as environment variables. default: `\u003cDIR\u003e/cronlock.conf`, `/etc/cronlock.conf`\n\nUsing the `CRONLOCK_CONFIG` file or by exporting in your environment, you can set these variables\nto change the behavior of cronlock:\n\n - `CRONLOCK_HOST` the Redis hostname. default: `localhost`\n - `CRONLOCK_PORT` the Redis port. default: `6379`\n - `CRONLOCK_AUTH` the Redis auth password. default: Not present\n - `CRONLOCK_DB` the Redis database. default: `0`\n - `CRONLOCK_REDIS_TIMEOUT` the length of time we wait for a response from redis before we consider it in an errored state.\n This ensures that if the redis connection goes away that we don't wait forever waiting for a response. default: `30`\n - `CRONLOCK_GRACE` determines how many seconds a lock should at least persist.\n This is to make sure that if you have a very small job, and clocks aren't in sync, the same job\n on server2/3/4/5/6/etc (maybe even slightly behind in time) will just fire right after server1 releases the lock. default: `40` (I recommend using a grace of at least 30s)\n - `CRONLOCK_RELEASE` determines how long a lock can persist at most.\n Acts as a failsafe so there can be no locks that persist forever in case of failure. default is a day: `86400`\n - `CRONLOCK_RECONNECT_ATTEMPTS` the number of times we try to reconnect before erroring.\n  If the redis connection is closed, we will attempt to reconnect to redis upto this amount of times. default: `5`\n - `CRONLOCK_RECONNECT_BACKOFF` the lenght of time to increase the wait between reconnects.\n  Acts as a failsafe to allow redis to be started before we try to reconnect. Set to 0 to retry the connection immediately. default: `5`\n - `CRONLOCK_KEY` a unique key for this command in the global Redis server. default: a hash of cronlock's arguments\n - `CRONLOCK_PREFIX` Redis key prefix used by all keys. default: `cronlock`\n - `CRONLOCK_VERBOSE` set to `yes` to print debug messages. default: `no`\n - `CRONLOCK_NTPDATE` set to `yes` update the server's clock againt `pool.ntp.org` before execution. default: `no`\n - `CRONLOCK_TIMEOUT` how long the command can run before it gets issues a `kill -9`. default: `0`; no timeout\n\n## Redis Cluster Support\n\nCronlock has support for Redis Cluster (http://redis.io/topics/cluster-spec) introduced in Redis 3.0. \n\nCronlock acts as a relatively \"dumb\" cluster client - it will react to MOVED and ASK commands and retry the request to the node given back in the response but it does not attempt to record the slot to node relationship for future use.\n\nCronlock supports the configuration of only one `CRONLOCK_HOST` and `CRONLOCK_PORT`. Cronlock will always connect to the configured host and port and issue the initial REDIS command with the calculated MD5 key. If the Redis node returns ASK or MOVED, Cronlock will disconnect and connect to the given host and port in the ASK or MOVED respoonse. The new Redis host is used for the remaining duration of the execution of Cronlock (or until another ASK or MOVED command is returned) - Cronlock will connect to the originally configured `CRONLOCK_HOST` and `CRONLOCK_PORT` for the next execution.\n\nGiven the support of only one `CRONLOCK_HOST` and `CRONLOCK_PORT` it is recommended that each server running cronlock is configured to initially connect a different master in the Redis Cluster. Thus if one Redis server goes down the instances of Cronlock configured to connect to the other master nodes will continue to operate. \n\nThis is easily done if the Redis Cluster is on the same servers as Cronlock - as `CRONLOCK_HOST` can be set to 127.0.0.1 - each copy of Cronlock will therefore initially connect to its local master node - before reconnecting to other alive Redis nodes.\n\nAside from configuring appropriate values for `CRONLOCK_HOST` and `CRONLOCK_PORT` for the systems running Cronlock - no additional configuration is required for Redis Cluster support.\n\n\n## Examples\n\n### Single box\n\n```bash\ncrontab -e\n* * * * * cronlock ls -al\n```\n\nIn this configuration, `ls -al` will be launched every minute. If the previous\n`ls -al` has not finished yet, another one is not started.\nThis works on 1 server, as the default `CRONLOCK_HOST` of `localhost` is used.\n\nIn this setup, cronlock works much like [Tim Kay](http://timkay.com/)'s [solo](https://github.com/timkay/solo),\nexcept cronlock requires [Redis](http://redis.io/), so I recommend using Tim Kay's solution here.\n\n### Distributed\n\n```bash\necho '0 8 * * * CRONLOCK_HOST=redis.mydomain.com cronlock /var/www/mail_customers.sh' | crontab\n```\n\nIn this configuration, a central Redis server is used to track the locking for\n`/var/www/mail_customers.sh`. So you see that throughout a cluster of 100 servers,\njust one instance of `/var/www/mail_customers.sh` is ran every morning. No less, no more.\n\nAs long as your Redis server and at least 1 volatile worker is alive, this happens.\n\n### Distributed using a config file\n\nTo avoid messy crontabs, you can use a config file for shared config instead.\nUnless `CRONLOCK_CONFIG` is set, cronlock will look in `./cronlock.conf`, then\nin `/etc/cronlock.conf`.\n\nExample:\n```bash\ncat \u003c\u003c EOF \u003e /etc/cronlock.conf\nCRONLOCK_HOST=\"redis.mydomain.com\"\nCRONLOCK_GRACE=50\nCRONLOCK_PREFIX=\"mycompany.cronlocks.\"\nCRONLOCK_NTPDATE=\"yes\"\nEOF\n\ncrontab -e\n* * * * * cronlock /var/www/mail_customers.sh # will use config from /etc/cronlock.conf\n```\n\n### Lock commands even though they have different arguments\n\nBy default cronlock uses your command and its arguments to make a unique identifier\nby which the global lock is acquired. However if you want to run: `ls -al` or `ls -a`,\nbut just 1 instance of either, you'll want to provide your own key:\n\n```bash\ncrontab -e\n# One of two will be executed because they share the same KEY\n* * * * * CRONLOCK_KEY=\"ls\" cronlock ls -al\n* * * * * CRONLOCK_KEY=\"ls\" cronlock ls -a\n```\n\n### Per application\n\nIf you use the same script and Redis server for multiple applications, an unwanted lock could deny app2 its script.\nYou could make up your own unique `CRONLOCK_KEY` to circumvent, but it's probably\nbetter to use the `CRONLOCK_PREFIX` for that:\n\n```bash\ncrontab -e\n* * * * * CRONLOCK_PREFIX=\"mylocks.app1.\" cronlock /var/www/mail_customers.sh\n```\n\n```bash\ncrontab -e\n* * * * * CRONLOCK_PREFIX=\"mylocks.app2.\" cronlock /var/www/mail_customers.sh\n```\n\nNow both /var/www/mail_customers.sh will run, because they have a different application in their prefixes.\n\n## Exit codes\n\n - = `200` Success (delete succeeded or lock not acquired, but normal execution)\n - = `201` Failure (cronlock error)\n - = `202` Failure (cronlock timeout)\n - \u003c `200` Success (acquired lock, executed your command), passes the exit code of your command\n\n## Versioning\n\nThis project implements the Semantic Versioning guidelines.\n\nReleases will be numbered with the following format:\n\n`\u003cmajor\u003e.\u003cminor\u003e.\u003cpatch\u003e`\n\nAnd constructed with the following guidelines:\n\n* Breaking backward compatibility bumps the major (and resets the minor and patch)\n* New additions without breaking backward compatibility bumps the minor (and resets the patch)\n* Bug fixes and misc changes bumps the patch\n\n\nFor more information on SemVer, please visit [http://semver.org](http://semver.org).\n\n## License\n\nCopyright (c) 2013 Kevin van Zonneveld, [http://kvz.io](http://kvz.io)  \nLicensed under MIT: [http://kvz.io/licenses/LICENSE-MIT](http://kvz.io/licenses/LICENSE-MIT)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkvz%2Fcronlock","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkvz%2Fcronlock","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkvz%2Fcronlock/lists"}