{"id":18284489,"url":"https://github.com/pplu/cloudcron","last_synced_at":"2025-12-12T02:42:26.140Z","repository":{"id":56834761,"uuid":"91695580","full_name":"pplu/CloudCron","owner":"pplu","description":"A simple distributed cloud friendly cron for the masses","archived":false,"fork":false,"pushed_at":"2019-01-14T19:06:13.000Z","size":134,"stargazers_count":25,"open_issues_count":2,"forks_count":5,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-21T00:26:07.367Z","etag":null,"topics":["aws","cloud","cloudformation","cloudwatch","cron","distributed","sqs"],"latest_commit_sha":null,"homepage":null,"language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pplu.png","metadata":{"files":{"readme":"README.md","changelog":"Changes","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-18T13:19:15.000Z","updated_at":"2023-01-23T22:08:51.000Z","dependencies_parsed_at":"2022-09-02T03:50:34.535Z","dependency_job_id":null,"html_url":"https://github.com/pplu/CloudCron","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2FCloudCron","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2FCloudCron/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2FCloudCron/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pplu%2FCloudCron/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pplu","download_url":"https://codeload.github.com/pplu/CloudCron/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247305872,"owners_count":20917197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","cloud","cloudformation","cloudwatch","cron","distributed","sqs"],"created_at":"2024-11-05T13:13:43.412Z","updated_at":"2025-12-12T02:42:26.103Z","avatar_url":"https://github.com/pplu.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"CloudCron\n=========\n\nA simple distributed, cost-effective, backwards-compatible, cloud-friendly cron for the masses.\n\nIntroduction\n============\n\nIf you google around looking for a \"cloud friendly cron\" that runs on AWS, you're out of luck. You're \nprobably just trying to solve one thing: to run a crontab on AWS.\n\nSolutions you find around tend to be in the following areas:\n\n - You need to build and maintain a complex, distributed, minimum of three nodes cluster.\n\n - Make it a Lambda function (since Lambdas can be scheduled).\n\n - Put the crons code in your app server and call it via HTTP.\n\nLots of times, users just put their cron on one \"cron instance\".\n\n\nThe CloudCron solution\n======================\n\n- Backwards-compatible\n\nGive it a cron file and it will run the same command you already have on the same server if you want. No need for reprogramming your jobs.\n\n- Cost-Effective\n\nWith minimal infrastructure (just a worker node).\n\n- Scalable/Parallelizable\n\nDo you have more crons than a machine can handle? You can scale your worker nodes at need.\n\n- Red/Black deployment friendly\n\nYou can deploy a new worker node with new code or patches without downtime.\n\n- No time limits for the task execution\n\nLambda-based solutions have a maximum running time of 5 minutes. Also, you have to convert your jobs to something Lambda-compatible.\n\n- CD Pipeline friendly\n\nYou get the opportunity to version control your crontab and deploy it as a part of your CD.\n\n- On-premise friendly\n\nYou can run your tasks on-premise. \n\nHow does it work?\n=================\n\nCloudCron is basically split into two halves. On one hand, you have a \"cron compiler\", that transforms a cronfile into scheduled CloudWatch events that queue to an SQS queue. The other half executes the events on one or more instances.\n\nYour crontab file gets transformed into a bunch of CloudWatch Events that get pushed to CloudFormation so they can be managed as a whole.\n\nThese CloudWatch events inject a message for each ocurrence of a cron event to an SQS queue, which is getting polled by a worker process. This worker process runs on the same machine (or machines) where your old cron could execute its jobs.\n\nInstallation\n============\n\nYou can install CloudCron with any perl package manager. We recommend using carton so you don't \ninstall dependencies in your system (carton is usually installable via your SOs package manager):\n\n```\nmkdir mycron\ncd mycron\necho 'requires \"CloudCron\";' \u003e\u003e cpanfile\necho 'requires \"CloudCron::Worker\";' \u003e\u003e cpanfile\ncarton install\ncarton exec $SHELL -l\n```\n\nCloudCron is split into two packages:\n - the management interface package: CloudCron enables you to create the necessary infrastructure\n - the worker package: CloudCron::Worker, the process that runs on your cloud nodes\n\nGet me started\n==============\n```\ncloudcron init --name MyCronQueue --region eu-west-1\n```\nThis command deploys the SQS queue and informs you how to start a worker. We can have as many queues as we want (for different groups of crons, for example)\n\nYou can start a worker with the output of the last command, but first create a log configuration file:\n```\nprintf \"log4perl.appender.Screen = Log::Log4perl::Appender::Screen\\nlog4perl.appender.Screen.layout = Log::Log4perl::Layout::PatternLayout\\nlog4perl.appender.Screen.layout.ConversionPattern = [%%d][CloudCron] %%p %%m%%n\\n# Catch all errors\\nlog4perl.logger = INFO, Screen\\n\" \u003e log.conf\n```\n\nAnd now, you can launch the worker in background, pointing to the log config you just created\n```\ncloudcron-worker --queue_url https://sqs.eu-west-1.amazonaws.com/012345678901/MyCronQueue-CloudCronQueue-LPNF3N07WF68 --region eu-west-1 --log_conf log.conf \u0026\n```\nThe worker is now idle, waiting for its first jobs, so we need to create them:\n\n```\ncloudcron deploy --name MyCronFile --destination_queue MyCronQueue --region eu-west-1 path_to_crontab_file\n```\nThe name of the destination queue is the name we gave to `cloudcron init`\n\nOnce the deploy command finishes, we wait for the worker to start receiving the messages and executing your jobs!\n\nIf you modify your crontab file, just redeploy. CloudCron will detect that you already deployed this cron and will update the events:\n```\ncloudcron deploy --name MyCronFile --destination_queue MyCronQueue --region eu-west-1 path_to_crontab_file\n```\n\nOnce you're ready, you can delete queues and crons with the `cloudcron remove` command.\n\nNitty gritty details\n====================\n - Each event launched by CloudWatch events is delivered to the crons' queue. The worker picks up the message ***only once***, even if the job\n   fails to execute (like in cron) it is not redelivered to the queue for retry. It's supposed that it will be executed the next time.\n\n - Each cron queue has a dead letter queue. Here you could find messages due to severe malfunction: messages that were delivered to a worker, but\n   never got acknowledged (deleted) in time. Notice that workers immediately delete a message after receiving it, even before attempting to execute\n   the job, so not deleting a message should not happen.\n\n - The worker executes with limited privileges. You control the user which the worker runs with. The worker shouldn't be run as root (unless you have good reasons to).\n\nKnown Limitations\n=================\n - The system depends on SQS, so it's possible for\n   - events to be processed out of order\n   - events to be processed more than one time by two concurrent workers and with, at least, once delivery. \n\n   SQS recently got FIFO and now it is exactly once delivery. It's on the TODO list to take advantage of that capability to provide a more cron-like behaviour.\n\n - There is no reentrancy control (like cron, you should take care of that in your job) (or you can contribute a patch to avoid reentrancy).\n\n - It's recommended to centralise the CloudCron logs with something like CloudWatch Logs if you're executing on autoscaling groups, for example.\n\n - If you don't have workers polling a queue, events will still get delivered to it and accumulate, so it's possible that you accumulate invocations of lots of events if you don't have a worker running.\n\n   SQS recently got a deduplication facility which we can use to try to avoid message accumulation.\n\n - If the system that hosts the worker fails before an event has been fully executed, the event will not be retried.\n\nTOPOLOGIES\n==========\n\nWith CloudCron you can generate a lot of different topologies to suit your needs:\n\nSingle crontab\n--------------\n\n```\ncrontab ---\u003e Queue \u003c--- Worker node\n```\n\n - create a queue with `cloudcron init`.\n - deploy your crontab file with `cloudcron deploy`.\n - start a `cloudcron-worker` on the machine that has to do the work (it can be on an EC2 instance or a machine in your datacenter).\n\nLots of jobs (multinode)\n------------------------\n\nIf one machine is not enough to handle the load of all your crons, you can just \nadd more CloudCron workers polling the same queue.\n\n```\n                                      |---- Worker node 1\ncrontab (lots of jobs)  ---\u003e Queue \u003c--|---- Worker node 2\n                                      |---- Worker node 3\n```\n\n - create a queue with `cloudcron init`\n - deploy your crontab file with `cloudcron deploy`\n - start one `cloudcron-worker` for each machine that has to do the work\n\nLots of jobs (autoscaling)\n--------------------------\n\nIf the load on your worker nodes varies enough, you might want to autoscale your\nworker node fleet by applying autoscaling to a worker node pool autoscaling group.\n\n```\n                                     |-A--   \ncrontab (lots of jobs) ---\u003e Queue \u003c--|-S-- Worker node N\n                      |              |-G--\n                      |                |\n                      +--- CloudWatch--+\n```\n\nCaution should be taken: when autoscaling shuts down a worker instance, it will kill the\nprocesses that are actually being executed. There is no facility in CloudCron to let your jobs finish\n -contributions welcome :).\n\nManage lots of jobs\n-------------------\n\nYou can deploy independent crontab files to the same queue, as long as the worker polling\nthe queue is able to execute the commands in your crontab.\n\n```\ncrontab for ETLs -----+\n                      |\ncrontab for cleanup --+---\u003e Queue \u003c--- Worker node\n                      |\ncrontab for X --------+\n```\n\n - create a queue with `cloudcron init`.\n - deploy each crontab file with `cloudcron deploy` to the same queue.\n - start a `cloudcron-worker` on the machine that has to do the work.\n\nCrons running with different users\n----------------------------------\n\nYou can deploy an independent worker for each user. The two workers can\nrun on the same instance\n\n```\ncrontab for user1 ----\u003e Queue1 \u003c---- Worker running with user1\n\ncrontab for user2 ----\u003e Queue2 \u003c---- Worker running with user2\n```\n\n - create two queues with `cloudcron init`.\n - deploy each crontab to its queue with `cloudcron deploy`.\n - start a worker under user1 pointing to the queue that its cron points to.\n - start the other worker under user2 pointing to the queue that its cron points to.\n\nFlexibility (custom topologies)\n-------------------------------\n\nYou can combine these scenarios as you want to adapt them to your needs. \nMix, match and report your topologies back so we can document them!\n\nDeploying cloudcron-worker\n==========================\n\nIn the examples directory, there is a sample of how to run a `cloudcron-worker` from an \nupstart job.\n\nBugs and Source\n===============\n\nThe source code is located here: https://github.com/capside/CloudCron.\n\nBugs and Issues can be reported here: https://github.com/capside/CloudCron/issues.\n\nAuthors\n=======\n\nPau Cervera, Eduard Badillo, Jose Luis Martinez\n\nLicense and Copyright\n=====================\n\nCopyright (c) 2017 by CAPSiDE SL\n\nThis code is distributed under the Apache 2 License. The full text of the license can be found in the LICENSE file included with this module.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplu%2Fcloudcron","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpplu%2Fcloudcron","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpplu%2Fcloudcron/lists"}