https://github.com/getsentry/hackweek-coda
A queue thing with a sprinkle of Italian
https://github.com/getsentry/hackweek-coda
tag-non-production
Last synced: 3 months ago
JSON representation
A queue thing with a sprinkle of Italian
- Host: GitHub
- URL: https://github.com/getsentry/hackweek-coda
- Owner: getsentry
- License: apache-2.0
- Created: 2023-08-21T11:34:44.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-10-14T08:09:34.000Z (over 1 year ago)
- Last Synced: 2024-05-09T23:58:40.360Z (about 1 year ago)
- Topics: tag-non-production
- Language: Python
- Homepage:
- Size: 1.36 MB
- Stars: 27
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Meet **Coda**, a hackweek project on a mission to revolutionize queue systems and simplify complex task workflows at [Sentry](https://sentry.io).
# What's Coda?
Coda is a hackweek project aimed at exploring alternative ideas for how queue systems can function. We extensively employ
Celery at Sentry, and although Celery is not a bad project, it does not offer an ideal abstraction for many of the
functionalities we intend to develop.
Within Sentry, some async jobs exhibit workflow-like behavior, relying on multiple tasks for their execution.
These tasks mainly rely on Redis as auxiliary storage to exchange data during the execution, resulting in many
rather complex and challenging implementations, which can differ significantly.
We quite appreciate the design of [Temporal](https://temporal.io/), but unfortunately it is quite complex for many of our
use cases. Coda is our attempt at building a workflow engine that mixes simplicity with some of Temporal's ideas.
# Example
This workflow starts two tasks. When the workflow reruns because for instance it
failed, it will not invoke tasks which have already yielded a result (as keyed by
`cache_key`).
```python
import coda
from .tasks import normalize_event, store_event
@coda.workflow()
async def process_event(project_id, event_data, **kwargs):
event_data = await coda.context.spawn_task(
normalize_event,
args={"project_id": project_id, "event_data": event_data},
cache_key=[event_data["event_id"]]
)
result = await coda.context.spawn_task(
store_event,
args={"project_id": project_id, "event_data": event_data},
cache_key=[event_data["event_id"]]
)
print(f"Workflow finished with result {result}")
```
# High-Level Design
Coda consists of a **supervisor** responsible for spawning workers. Currently, there is only a Python client available.
The workers relay all their communication through the supervisor, connecting to neither a queue nor a state management
system. All data is managed by the supervisor.
The supervisor's primary responsibility is to communicate with the actual workflow engine, which has not been
implemented yet. Consequently, all state data is stored directly in memory within the supervisor. This state is
associated with the workflow.