https://github.com/nvdv/atq

distributed task queue for asyncio
https://github.com/nvdv/atq

async-await asyncio python python3 task-queue

Last synced: about 1 month ago
JSON representation

distributed task queue for asyncio

• Host: GitHub
• URL: https://github.com/nvdv/atq
• Owner: nvdv
• Created: 2017-09-01T12:54:27.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-10-26T09:43:51.000Z (about 7 years ago)
• Last Synced: 2024-03-14T09:32:01.270Z (9 months ago)
• Topics: async-await, asyncio, python, python3, task-queue
• Language: Python
• Size: 18.6 KB
• Stars: 89
• Watchers: 6
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.rst

Awesome Lists containing this project

• starred-awesome - atq - distributed task queue for asyncio (Python)

README

        
atq

===

``atq`` is a pure-Python asynchronous task queue built to work with ``asyncio``.

It is designed to run costly functions outside main event loop using

distributed workers.

``atq`` requires Python 3.5+ and is distributed under BSD license.

Usage

-----

First, you should start workers on the servers you plan to use for task execution

.. code-block ::

   atqserver --host  --port  --worker 

where

- ```` is a hostname of the server

- ```` is a port that server will listen on

- ```` is a number of worker processes

Please note that code of tasks should be accessible by ``atq``, so it's advised

to run ``atq`` server from your project root directory in more complex

situations.

Then you will need to create a client using hostnames and ports of initialized

servers:

.. code-block:: python

   import atq

   q = atq.Q([

        ("localhost", 12345),

    ])

Finally you can use ``atq`` in your code:

.. code-block:: python

    import atq

    import asyncio

    import requests

    from collections import ChainMap

    from collections import Counter

    URLS = [

        ...

    ]

    q = atq.Q([

        ("localhost", 12345),

    ])

    def top_words(url, n):

        """Returns top n words from text specified by url."""

        text = requests.get(url).text.split()

        return {url: Counter(text).most_common(n)}

    async def get_top_words(urls, n):

        """Returns top n words in documents specified by URLs."""

        tops_in_url = await asyncio.gather(

            *[q.q(top_words, url, n) for url in urls])

        return ChainMap(*tops_in_url)

    top = asyncio.get_event_loop().run_until_complete(get_top_words(URLS, 10))

You can find more examples in ``examples`` subdirectory.

Installation

------------

.. code-block ::

    pip3 install atq

Testing

-------

.. code-block ::

    python3 setup.py test

License

-------

BSD