Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ewolters/tempora

Tempora scheduling engine
https://github.com/ewolters/tempora

background-jobs consensus distributed-cron distributed-systems django fault-tolerance high-availability infrastructure job-scheduler leader-election leader-election-algorithm python raft scheduler task-scheduler

Last synced: 5 months ago
JSON representation

Tempora scheduling engine

Awesome Lists containing this project

README 

          
# Tempora



**Native High Availability Distributed Task Scheduler for Python and Django**



[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)

[![Django 4.2+](https://img.shields.io/badge/django-4.2+-green.svg)](https://www.djangoproject.com/)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

[![PostgreSQL](https://img.shields.io/badge/database-PostgreSQL-336791.svg)](https://www.postgresql.org/)



Tempora is a production-ready **distributed task scheduler** with built-in **high availability**, **Raft consensus**, and **automatic failover**. Unlike Celery or other task queues that require external coordination (Redis, RabbitMQ, ZooKeeper), Tempora provides **native clustering** with no additional infrastructure.



## Key Features



- **Raft Consensus Protocol** - Battle-tested distributed consensus for leader election and log replication

- **Automatic Failover** - Sub-second leader election with pre-vote optimization

- **Split-Brain Prevention** - Fencing tokens and distributed locking prevent data corruption

- **Zero External Dependencies** - No Redis, no ZooKeeper, no RabbitMQ required for coordination

- **Django Integration** - Native Django ORM support with PostgreSQL persistence

- **Production Hardened** - TLS 1.3, HMAC-SHA256 authentication, rate limiting, health monitoring

- **Flexible Work Distribution** - Round-robin, least-loaded, and affinity-based strategies



## Why Tempora?



| Feature | Tempora | Celery | APScheduler | Django-Q |

|---------|---------|--------|-------------|----------|

| Native HA | Yes | No (requires Redis Sentinel) | No | No |

| Raft Consensus | Yes | No | No | No |

| Zero External Deps | Yes | No | Yes | No |

| Automatic Failover | Yes | Partial | No | No |

| Split-Brain Safe | Yes | No | No | No |

| Django Native | Yes | Partial | Partial | Yes |



## Installation



```bash

pip install tempora-scheduler

```



### Requirements



- Python 3.11+

- Django 4.2+

- PostgreSQL 14+ (for persistence)



## Quick Start



### 1. Add to Django Settings



```python

INSTALLED_APPS = [

    # ...

    'tempora',

]



# Tempora Configuration

TEMPORA_CLUSTER_SECRET = "your-32-byte-cluster-secret-key"

TEMPORA_NODE_ID = "node-1"

TEMPORA_CLUSTER_NODES = [

    {"id": "node-1", "host": "10.0.0.1", "port": 7000},

    {"id": "node-2", "host": "10.0.0.2", "port": 7000},

    {"id": "node-3", "host": "10.0.0.3", "port": 7000},

]

```



### 2. Run Migrations



```bash

python manage.py migrate tempora

```



### 3. Start the Scheduler



```python

from tempora import TemporaCluster



cluster = TemporaCluster(

    node_id="node-1",

    cluster_secret="your-32-byte-cluster-secret-key",

    peers=[

        ("node-2", "10.0.0.2", 7000),

        ("node-3", "10.0.0.3", 7000),

    ]

)



await cluster.start()

```



### 4. Schedule Tasks



```python

from tempora import schedule_task

from datetime import timedelta



# One-time task

schedule_task(

    name="send_welcome_email",

    func="myapp.tasks.send_email",

    args={"user_id": 123},

    run_at=timezone.now() + timedelta(hours=1)

)



# Recurring task

schedule_task(

    name="daily_cleanup",

    func="myapp.tasks.cleanup",

    cron="0 2 * * *"  # Every day at 2 AM

)

```



## Architecture



Tempora uses a **3-tier architecture** for distributed scheduling:



```

┌─────────────────────────────────────────────────────────────┐

│                     Coordination Layer                       │

│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │

│  │   Node 1    │  │   Node 2    │  │   Node 3    │          │

│  │  (Leader)   │◄─┤ (Follower)  │──┤ (Follower)  │          │

│  └─────────────┘  └─────────────┘  └─────────────┘          │

│         │              Raft Consensus Protocol               │

└─────────┼───────────────────────────────────────────────────┘

          │

┌─────────▼───────────────────────────────────────────────────┐

│                    Replication Layer                         │

│         Log Replication • Commit Tracking • Snapshots        │

└─────────────────────────────────────────────────────────────┘

          │

┌─────────▼───────────────────────────────────────────────────┐

│                    Persistence Layer                         │

│              PostgreSQL • Django ORM • Migrations            │

└─────────────────────────────────────────────────────────────┘

```



## Configuration Reference



### Environment Variables



| Variable | Description | Default |

|----------|-------------|---------|

| `TEMPORA_SECRET_KEY` | Django secret key | Required |

| `TEMPORA_CLUSTER_SECRET` | Cluster authentication secret (32+ bytes) | Required |

| `TEMPORA_NODE_ID` | Unique node identifier | Required |

| `TEMPORA_DB_HOST` | PostgreSQL host | `localhost` |

| `TEMPORA_DB_PORT` | PostgreSQL port | `5432` |

| `TEMPORA_DB_NAME` | Database name | `tempora` |

| `TEMPORA_DB_USER` | Database user | `tempora` |

| `TEMPORA_DB_PASSWORD` | Database password | Required |

| `TEMPORA_LOG_LEVEL` | Logging level | `INFO` |



### Cluster Settings



```python

TEMPORA_SETTINGS = {

    # Raft timing

    "election_timeout_min": 150,  # ms

    "election_timeout_max": 300,  # ms

    "heartbeat_interval": 50,     # ms



    # Replication

    "max_entries_per_append": 100,

    "snapshot_threshold": 10000,



    # Security

    "tls_enabled": True,

    "tls_cert_file": "/path/to/cert.pem",

    "tls_key_file": "/path/to/key.pem",



    # Rate limiting

    "max_requests_per_second": 1000,

    "connection_pool_size": 100,

}

```



## Production Deployment



### Minimum Cluster Size



For high availability, deploy **3 or 5 nodes** (odd numbers prevent split-brain):



- **3 nodes**: Tolerates 1 failure

- **5 nodes**: Tolerates 2 failures



### Health Monitoring



Tempora exposes health endpoints:



```python

from tempora import get_cluster_health



health = await get_cluster_health()

# {

#     "status": "healthy",

#     "leader": "node-1",

#     "term": 42,

#     "commit_index": 15678,

#     "nodes": {

#         "node-1": {"status": "leader", "last_heartbeat": "2024-01-15T10:30:00Z"},

#         "node-2": {"status": "follower", "last_heartbeat": "2024-01-15T10:30:00Z"},

#         "node-3": {"status": "follower", "last_heartbeat": "2024-01-15T10:30:00Z"},

#     }

# }

```



### Graceful Shutdown



```python

# Trigger leadership transfer before shutdown

await cluster.transfer_leadership()

await cluster.stop()

```



## Comparison with Alternatives



### vs Celery



Celery requires Redis or RabbitMQ for coordination and doesn't provide native high availability. Tempora eliminates external dependencies and provides built-in Raft consensus for automatic failover.



### vs APScheduler



APScheduler is single-node only. Tempora provides distributed scheduling across multiple nodes with automatic leader election.



### vs Django-Q



Django-Q requires Redis or Django ORM polling. Tempora uses native TCP coordination with Raft consensus for true distributed scheduling.



## Use Cases



- **Distributed Cron Jobs** - Run scheduled tasks across a cluster with guaranteed execution

- **Task Queues** - Process background jobs with automatic failover

- **Workflow Orchestration** - Coordinate multi-step workflows across services

- **Event Processing** - Handle events with at-least-once delivery guarantees

- **Batch Processing** - Distribute large batch jobs across worker nodes



## API Reference



### Core Classes



- `TemporaCluster` - Main cluster coordinator

- `RaftNode` - Raft consensus node

- `LogReplicator` - Log replication manager

- `WorkDistributor` - Task distribution strategies



### Management Commands



```bash

# Check cluster status

python manage.py tempora_status



# Trigger leader election

python manage.py tempora_election



# View replication lag

python manage.py tempora_replication

```



## Performance



Benchmarks on 3-node cluster (AWS c5.xlarge):



| Metric | Value |

|--------|-------|

| Leader election | < 300ms |

| Task scheduling | 10,000+ tasks/sec |

| Replication lag | < 10ms |

| Failover time | < 500ms |



## License



Tempora is dual-licensed:



- **MIT License** - Free for open source projects

- **Commercial License** - For proprietary applications



See [LICENSE](LICENSE) for details.



## Contributing



We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.



## Support



- **Documentation**: [https://tempora.readthedocs.io](https://tempora.readthedocs.io)

- **Issues**: [GitHub Issues](https://github.com/yourusername/tempora/issues)

- **Discussions**: [GitHub Discussions](https://github.com/yourusername/tempora/discussions)



---



**Keywords**: distributed task scheduler, python task queue, django scheduler, high availability scheduler, raft consensus python, distributed cron, celery alternative, background job processing, task orchestration, distributed systems python, fault-tolerant scheduler, leader election python, distributed consensus, job scheduler django, async task queue