{"id":13815124,"url":"https://github.com/Nextdoor/ndscheduler","last_synced_at":"2025-05-15T07:32:01.547Z","repository":{"id":35687335,"uuid":"39963912","full_name":"Nextdoor/ndscheduler","owner":"Nextdoor","description":"A flexible python library for building your own cron-like system, with REST APIs and a Web UI.","archived":false,"fork":false,"pushed_at":"2024-11-15T15:07:56.000Z","size":411,"stargazers_count":1101,"open_issues_count":43,"forks_count":201,"subscribers_count":48,"default_branch":"master","last_synced_at":"2025-05-11T18:04:27.190Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Nextdoor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-07-30T17:41:48.000Z","updated_at":"2025-05-06T10:03:21.000Z","dependencies_parsed_at":"2022-06-27T14:33:00.830Z","dependency_job_id":"079b5dfd-cf0e-4d0f-9301-7ad0c3427ffb","html_url":"https://github.com/Nextdoor/ndscheduler","commit_stats":{"total_commits":86,"total_committers":16,"mean_commits":5.375,"dds":0.7906976744186046,"last_synced_commit":"d31016aaca480e38a69d75a66a9978a937c6a0b0"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndscheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndscheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndscheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Nextdoor%2Fndscheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Nextdoor","download_url":"https://codeload.github.com/Nextdoor/ndscheduler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254101618,"owners_count":22014909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T04:02:59.479Z","updated_at":"2025-05-15T07:31:58.365Z","avatar_url":"https://github.com/Nextdoor.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Nextdoor Scheduler\n\n![Apache](https://img.shields.io/hexpm/l/plug.svg) \n[![Build Status](https://travis-ci.org/Nextdoor/ndscheduler.svg)](https://travis-ci.org/Nextdoor/ndscheduler)\n\n``ndscheduler`` is a flexible python library for building your own cron-like system to schedule jobs, which is to run a tornado process to serve REST APIs and a web ui.\n\nCheck out our blog post - [We Don't Run Cron Jobs at Nextdoor](https://engblog.nextdoor.com/we-don-t-run-cron-jobs-at-nextdoor-6f7f9cc62040#.d2erw1pl6)\n\n**``ndscheduler`` currently supports Python 2 \u0026 3 on Mac OS X / Linux.**\n\n## Table of contents\n  \n  * [Key Abstractions](#key-abstractions)\n  * [Try it NOW](#try-it-now)\n  * [How to build Your own cron-replacement](#how-to-build-your-own-cron-replacement)\n    * [Install ndscheduler](#install-ndscheduler)\n    * [Three things](#three-things)\n    * [Reference Implementation](#reference-implementation)   \n  * [Contribute code to ndscheduler](#contribute-code-to-ndscheduler)\n  * [REST APIs](#rest-apis)\n  * [Web UI](#web-ui)\n\n## Key Abstractions\n\n* [CoreScheduler](https://github.com/Nextdoor/ndscheduler/tree/master/ndscheduler/corescheduler): encapsulates all core scheduling functionality, and consists of:\n  * [Datastore](https://github.com/Nextdoor/ndscheduler/tree/master/ndscheduler/corescheduler/datastore): manages database connections and makes queries; could support Postgres, MySQL, and sqlite.\n    * Job: represents a schedule job and decides how to run a paricular job.\n    * Execution: represents an instance of job execution.\n    * AuditLog: logs when and who runs what job.\n  * [ScheduleManager](https://github.com/Nextdoor/ndscheduler/blob/master/ndscheduler/corescheduler/scheduler_manager.py): access Datastore to manage jobs, i.e., schedule/modify/delete/pause/resume a job.\n* [Server](https://github.com/Nextdoor/ndscheduler/tree/master/ndscheduler/server): a tornado server that runs ScheduleManager and provides REST APIs and serves UI.\n* [Web UI](https://github.com/Nextdoor/ndscheduler/tree/master/ndscheduler/static): a single page HTML app; this is a default implementation.\n\nNote: ``corescheduler`` can also be used independently within your own service if you use a different Tornado server / Web UI.\n\n## Try it NOW\n\nFrom source code:\n\n    git clone https://github.com/Nextdoor/ndscheduler.git\n    cd ndscheduler\n    make simple\n\nOr use docker:\n\n    docker run -it -p 8888:8888 wenbinf/ndscheduler\n    \nOpen your browser and go to [localhost:8888](http://localhost:8888). \n\n**Demo**\n(Click for fullscreen play)\n![ndscheduler demo](https://giant.gfycat.com/NastyBossyBeaver.gif)\n\n## How to build Your own cron-replacement\n\n### Install ndscheduler\nUsing pip (from GitHub repo)\n\n    #\n    # Put this in requirements.txt, then run\n    #    pip install -r requirements.txt\n    #\n\n    # If you want the latest build\n    git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler\n\n    # Or put this if you want a specific commit\n    git+https://github.com/Nextdoor/ndscheduler.git@5843322ebb440d324ca5a66ba55fea1fd00dabe8\n\n    # Or put this if you want a specific tag version\n    git+https://github.com/Nextdoor/ndscheduler.git@v0.1.0#egg=ndscheduler\n    \n    #\n    # Run from command line\n    #\n\n    pip install -e git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler\n\n(We'll upload the package to PyPI soon.)\n\n### Three things\n\nYou have to implement three things for your scheduler, i.e., ``Settings``, ``Server``, and ``Jobs``.\n\n**Settings**\n\nIn your implementation, you need to provide a settings file to override default settings (e.g., [settings in simple_scheduler](https://github.com/Nextdoor/ndscheduler/blob/master/simple_scheduler/settings.py)). You need to specify the python import path in the environment variable ``NDSCHEDULER_SETTINGS_MODULE`` before running the server.\n\nAll available settings can be found in [default_settings.py](https://github.com/Nextdoor/ndscheduler/blob/master/ndscheduler/default_settings.py) file.\n\n**Server**\n\nYou need to have a server file to import and run ``ndscheduler.server.server.SchedulerServer``.\n\n**Jobs**\n\nEach job should be a standalone class that is a subclass of ``ndscheduler.job.JobBase`` and put the main logic of the job in ``run()`` function.\n\nAfter you set up ``Settings``, ``Server`` and ``Jobs``, you can run the whole thing like this:\n\n    NDSCHEDULER_SETTINGS_MODULE=simple_scheduler.settings \\\n    PYTHONPATH=.:$(PYTHONPATH) \\\n\t\t    python simple_scheduler/scheduler.py\n\t\t  \n### Upgrading\n\nIt is best practice to backup your database before doing any upgrade. ndscheduler relies on [apscheduler](https://apscheduler.readthedocs.io/en/latest/) to serialize jobs to the database, and while it is usually backwards-compatible (i.e. jobs created with an older version of apscheduler will continue to work after upgrading apscheduler) this is not guaranteed, and it is known that downgrading apscheduler can cause issues. See [this PR comment](https://github.com/Nextdoor/ndscheduler/pull/54#issue-262152050) for more details.\n\n### Reference Implementation\n\nSee code in the [simple_scheduler/](https://github.com/Nextdoor/ndscheduler/tree/master/simple_scheduler) directory for inspiration :)\n\nRun it\n\n    make simple\n    \nAccess the web ui via [localhost:8888](http://localhost:8888)\n\nThe reference implementation also comes with [several sample jobs](https://github.com/Nextdoor/ndscheduler/tree/master/simple_scheduler/jobs).\n* AwesomeJob: it just prints out 2 arguments you pass in.\n* SlackJob: it sends a slack message periodically, for example, team standup reminder.\n* ShellJob: it runs an executable command, for example, run curl to crawl web pages.\n* CurlJob: it's like running [curl](http://curl.haxx.se/) periodically.\n\nAnd it's [dockerized](https://github.com/Nextdoor/ndscheduler/tree/master/simple_scheduler/docker).\n\n## Contribute code to ndscheduler\n\n**Install dependencies**\n\n    # Each time we introduce a new dependency in setup.py, you have to run this\n    make install\n\n**Run unit tests**\n\n    make test\n    \n**Clean everything and start from scratch**\n    \n    make clean\n\nFinally, send pull request. Please make sure the [CI](https://travis-ci.org/Nextdoor/ndscheduler) passes for your PR.\n\n## REST APIs\n\nPlease see [README.md in ndscheduler/server/handlers](https://github.com/Nextdoor/ndscheduler/blob/master/ndscheduler/server/handlers/README.md).\n\n## Web UI\n\nWe provide a default implementation of web ui. You can replace the default web ui by overwriting these settings\n\n    STATIC_DIR_PATH = :static asset directory paths:\n    TEMPLATE_DIR_PATH = :template directory path:\n    APP_INDEX_PAGE = :the file name of the single page app's html:\n    \n### The default web ui\n\n**List of jobs**\n\n![List of jobs](http://i.imgur.com/dGILbkZ.png)\n\n**List of executions**\n\n![List of executions](http://i.imgur.com/JpjzrlU.png)\n\n**Audit Logs**\n\n![Audit logs](http://i.imgur.com/eHLzHhw.png)\n\n**Modify a job**\n\n![Modify a job](http://i.imgur.com/aWv6xOR.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNextdoor%2Fndscheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNextdoor%2Fndscheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNextdoor%2Fndscheduler/lists"}