Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mypebble/django-queue-fetcher
A Queue Fetching system for Django integrated into SQS
https://github.com/mypebble/django-queue-fetcher
django python sqs
Last synced: about 1 month ago
JSON representation
A Queue Fetching system for Django integrated into SQS
- Host: GitHub
- URL: https://github.com/mypebble/django-queue-fetcher
- Owner: mypebble
- Created: 2016-05-16T14:17:17.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2021-06-10T19:06:21.000Z (over 3 years ago)
- Last Synced: 2024-06-21T22:30:06.916Z (3 months ago)
- Topics: django, python, sqs
- Language: Python
- Size: 47.9 KB
- Stars: 3
- Watchers: 4
- Forks: 2
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- starred-awesome - django-queue-fetcher - A Queue Fetching system for Django integrated into SQS (Python)
README
# QueueFetcher for Django and SQS
[](https://circleci.com/gh/mypebble/django-queue-fetcher)
[](https://badge.fury.io/py/queue-fetcher)
QueueFetcher allows you to deal with Amazon SQS queues
in an easier manner in Django.
It provides:
* `run_queue` management task to start the task from cli
* `QueueFetcher` class to do the heavy lifting with the pieces
seperated out and testable
## Getting started
Install `queue-fetcher` from pip
Add `queue_fetcher` to `INSTALLED_APPLICATIONS`
Add to your settings.py:
```python
TEST_SQS = False
QUEUES = {
'Internal Name': 'Name On Amazon'
}
```
Now build your tasks in your tasks package:
```
from queue_fetcher.tasks import QueueFetcher
class SampleQueueTask(QueueFetcher):
queue = 'test'
def process_sample(self, msg):
raise NotImplementedError('This does nothing.. yet')
```
QueueFetcher expects messages from SQS to contain
a list of events, with each event containing a `message_type`
attribute of something like `update_transaction`.
This is then dispatched to a function prefixed with `process_`.
### Visibility Timeout
Tasks run from Django Queue Fetcher can, when they hit an error, keep thrashing
your SQS queues. We recommend you set a `visibility_timeout` on each task to
minimise your costs and any errors you send to your logs:
```python
from queue_fetcher.tasks import QueueFetcher
class MyQueueFetcher(QueueFetcher):
"""Same Queue Fetcher.
"""
queue = 'test'
visibility_timeout = 60 # Tell SQS to give us 60 seconds to process
def process_my_message(self, msg):
"""Process a message.
"""
return True
```
### Testing your Code
The `queue-fetcher` app includes a `QueueTestCase` class that removes the need
to handle SQS in your test code. To use it, simple extend the class and use
`get_yaml` or `get_json` to get your fixtures, located in the same app as your
test.
```python
from queue_fetcher.test import QueueTestCase
from .tasks import ExampleTaskClass
class ExampleTestCase(QueueTestCase):
"""
"""
def test_my_app(self):
"""
"""
fixture = self.get_json('exampleapp/test.sqs.json')
task = ExampleTaskClass()
task.read(fixture)
# Insert your assertions here
```