{"id":20495501,"url":"https://github.com/vyahello/python-optimization-cheetsheet","last_synced_at":"2025-09-25T06:30:52.269Z","repository":{"id":53792320,"uuid":"178895877","full_name":"vyahello/python-optimization-cheetsheet","owner":"vyahello","description":"📚 Contains a set of tips and tricks to optimize python code with generators, coroutines and asyncIO","archived":false,"fork":false,"pushed_at":"2023-07-20T15:09:36.000Z","size":38,"stargazers_count":4,"open_issues_count":3,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-15T17:46:07.584Z","etag":null,"topics":["asyncio","cheetsheet","coroutines","python-generators","yield"],"latest_commit_sha":null,"homepage":"https://vyahello.github.io/python-optimization-cheetsheet","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vyahello.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-01T15:46:13.000Z","updated_at":"2024-06-12T03:01:20.000Z","dependencies_parsed_at":"2022-08-25T15:10:16.302Z","dependency_job_id":null,"html_url":"https://github.com/vyahello/python-optimization-cheetsheet","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyahello%2Fpython-optimization-cheetsheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyahello%2Fpython-optimization-cheetsheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyahello%2Fpython-optimization-cheetsheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vyahello%2Fpython-optimization-cheetsheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vyahello","download_url":"https://codeload.github.com/vyahello/python-optimization-cheetsheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234157963,"owners_count":18788501,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asyncio","cheetsheet","coroutines","python-generators","yield"],"created_at":"2024-11-15T17:46:14.318Z","updated_at":"2025-09-25T06:30:46.976Z","avatar_url":"https://github.com/vyahello.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)\n\n# Python optimization cheetsheet\nImplementation of python optimization cheetsheet (`yield`, `generators`, `coroutines` and `asyncio`). The source code is located [here](materials).\n\n**Tools**\n- python 3.6, 3.7, 3.8\n- [asyncio](https://docs.python.org/3/library/asyncio.html)\n- [black](https://black.readthedocs.io/en/stable/)\n- [pylint](https://www.pylint.org/)\n\n## Content\n- [generators](#generators)\n  - [functions](#functions)\n  - [expressions](#expressions)\n  - [new iteration patterns with generators](#new-iteration-patterns-with-generators)\n  - [advanced operations with generators](#advanced-operations-with-generators)\n  - [memory efficient objects](#memory-efficient-objects)\n- [coroutines](#coroutines)\n- [asyncio](#asyncio)\n\n## Generators\n\u003e Basic functions calculate values and returns them, otherwise generators return a lazy iterator that returns a stream of values.\n\u003e \n\u003e A common use case of generators is to work with data streams or large files like `.csv` files\n### functions\n**Basic generator sample**\n```python\n# materials/generator_sample.py\n\ndef generator_sample():\n    yield 100\n\n\ngenerator = generator_sample()\nprint(generator)\nprint(type(generator))\nprint(dir(generator))\nprint(hasattr(generator, '__next__'))\nprint(next(generator))\nprint(next(generator))\n\ngenerator_list = list(generator_sample())\nprint(generator_list)\nprint(len(generator_list))\n\nprint(sum(generator_sample()))\n```\n**Generator with multiple yield statements**\n```python\n# materials/multiple_yields.py\n\ndef multiple_yields():\n    yield 'This'\n    yield 'is'\n    yield 'my'\n    yield 'generator'\n    yield 'function'\n    yield '!'\n\n\nvalues = multiple_yields()\nprint(next(values))\nprint(next(values))\nprint(next(values))\nprint(next(values))\nprint(next(values))\n\nother_values = multiple_yields()\nfor value in other_values:\n    print(value)\n```\n**Yielding iterable with generator**\n\u003e Any function that has `yield` operator is a generator.\n\u003e\n\u003e Generation an infinite sequence, however, will require the use of a generator, since your computer memory is finite.\n\u003e Yield is an expression rather than statement.\n```python\n# materials/yielding.py\n\ndef is_palindrome_number(number):\n    return number == int(str(number)[::-1])\n\n\ndef infinite_sequence():\n    num = 0\n    while True:\n        yield num\n        num += 1\n\n\nfor number in infinite_sequence():\n    if is_palindrome_number(number):\n        print(number)\n\n\ndef countdown_from(number):\n    print(f'Starting to count from {number}!')\n    while number \u003e 0:\n        yield number\n        number -= 1\n    print('Done!')\n\n\ndef increment(start, stop):\n    yield from range(start, stop)\n\n\ncountdown = countdown_from(number=10)\nfor count in countdown:\n    print(count)\n\n\nincremental = increment(start=1, stop=10)\nfor inc in incremental:\n    print(inc)\n```\n\n### expressions\n```python\n# materials/generator_expressions.py\n\neven_numbers = (num for num in range(15) if num % 2 == 0)\nprint(even_numbers)\nfor num in even_numbers:\n    print(num)\n\n\ndef multiply_each_by(multiplier):\n    return (element * multiplier for element in range(5))\n\n\nmultiplied_container = multiply_each_by(multiplier=3)\nprint(multiplied_container)\nfor obj in multiplied_container:\n    print(obj)\n```\n\n### new iteration patterns with generators\n```python\n# materials/float_range.py\n\ndef float_range(start, stop, increment):\n    initial_point = start\n    while initial_point \u003c stop:\n        yield initial_point\n        initial_point += increment\n\n\nfor number in float_range(0, 4, 0.5):\n    print(number)\n```\n```python\n# materials/countdown.py\n\nclass Countdown:\n    def __init__(self, start):\n        self._start = start\n\n    def __iter__(self):\n        number = self._start\n        while number \u003e 0:\n            yield number\n            number -= 1\n\n    def __reversed__(self):\n        number = 1\n        while number \u003c= self._start:\n            yield number\n            number += 1\n\n\nforward_countdown = Countdown(10)\nfor f_count in forward_countdown:\n    print(f_count)\n\nreversed_countdown = reversed(Countdown(10))\nfor r_count in reversed_countdown:\n    print(r_count)\n```\n\n### advanced operations with generators\n**Slice generator elements**\n```python\n# materials/slice_generators.py\n\nimport itertools\n\n\ndef doubles_of(number):\n    for num in range(number):\n        yield 2 * num\n\n\nprint(help(itertools.islice))\n\nfor element in itertools.islice(doubles_of(50), 10, 15):\n    print(element)\n```\n**Concatenate generators sequence**\n```python\n# materials/concatenate_generators.py\n\nimport itertools\n\n\ndef fruits():\n    for fruit in ('apple', 'orange', 'banana'):\n        yield fruit\n\n\ndef vegetables():\n    for vegetable in ('potato', 'tomato', 'cucumber'):\n        yield vegetable\n\n\nprint(help(itertools.chain))\n\nbucket = itertools.chain(fruits(), vegetables())\nfor item in bucket:\n    print(item)\n```\n**Zip generators elements**\n```python\n# materials/zip_generators.py\n\nimport itertools\n\n\ndef ascending():\n    yield from (1, 2, 3, 4, 5)\n\n\ndef descending():\n    yield from (5, 4, 3, 2, 1)\n\n\nfor pair in itertools.zip_longest(ascending(), descending()):\n    print(pair)\n```\n\n### memory efficient objects\n```python\n# materials/memory_efficacy.py\n\nimport sys\nimport cProfile\n\ngenerator_container = (num * 3 for num in range(10000000) if num % 6 == 0 or num % 7 == 0)\nprint(sys.getsizeof(generator_container))\n\nlist_container = [num * 3 for num in range(10000000) if num % 6 == 0 or num % 7 == 0]\nprint(sys.getsizeof(list_container))\n\nprint(cProfile.run('sum(generator_container)'))\nprint(cProfile.run('sum(list_container)'))\n```\n\n## Coroutines\n\u003e Coroutines can consume and produce data. They can pause stream execution till next message is sent.\n\u003e\n\u003e Generators produce data for iteration while coroutines can also consume data.\n\n**Sending values with `.send` method**\n```python\n# materials/send_coroutines.py\n\ndef coroutine():\n    while True:\n        value = yield  # allows to manipulate yielded value\n        print(value)\n\n\ni = coroutine()\ni.send(None)  # initial value should be 'None'\ni.send(1)\ni.send(10)\n\n\ndef counter(maximum):\n    initial = 0\n    while initial \u003c maximum:\n        value = (yield initial)  # equals to None till .send(number) is called\n        # If value is given (remember default is None) then change the counter\n        if value is not None:\n            initial = value\n        else:\n            initial += 1\n\n\nc = counter(10)\nprint(next(c))  # 0\nprint(next(c))  # 1\nprint(c.send(5))  # 5\nprint(next(c))  # 6\n\ndef is_palindrome_number(number):\n    return number == int(str(number)[::-1])\n\n\ndef infinite_palindromes():\n    number = 0\n    while True:\n        if is_palindrome_number(number):\n            i = (yield number)\n            if i is not None:\n                number = i\n        number += 1\n\n\nc = infinite_palindromes()\nprint(next(c))  # 0\nprint(next(c))  # 1\nprint(c.send(100))  # 101\nprint(next(c))  # 111\n\n\ndef print_name(prefix):\n    print(\"Search for \", prefix, \" prefix\")\n    while True:\n        name = yield\n        if prefix in name:\n            print(name)\n\n\npn = print_name(\"Dear\")\nnext(pn)  # calls first yield expression\npn.send(\"Alex\")\npn.send(\"Dear Alex\")  # matches with prefix\n\n\ndef grep(pattern):\n    print(f\"Search for '{pattern}' pattern\")\n    while True:\n        value = yield\n        if pattern in value:\n            print(f\"Matched: '{value}'\")\n\n\ng = grep(\"hey\")\nnext(g)  # to start coroutine\ng.send(\"hello\")\ng.send(\"hey\")\ng.send(\"hey Mike\")\n```\n\n**Raise an exception with `.throw` method**\n\u003e `.throw()` allows you to throw exceptions through the generator.\n\n```python\n# materials/throw_coroutines.py\n\ndef counter(maximum):\n    initial = 0\n    while initial \u003c maximum:\n        value = (yield initial)  # equals to None till .send(number) is called\n        # If value is given (remember default is None) then change the counter\n        if value is not None:\n            initial = value\n        else:\n            initial += 1\n\n\nc = counter(10)\nfor i in c:\n    print(i)\n    if i == 5:\n        c.throw(ValueError(\"It is too large\"))\n```\n\n**Stop generator with `.close` method**\n\u003e `.close()` allows you to stop a generator. Instead of calling `.throw()`, you use `.close()` (it calls StopIteration error). \n\n```python\n# materials/close_coroutines.py\n\ndef counter(maximum):\n    initial = 0\n    while initial \u003c maximum:\n        value = (yield initial)  # equals to None till .send(number) is called\n        # If value is given (remember default is None) then change the counter\n        if value is not None:\n            initial = value\n        else:\n            initial += 1\n\n\nc = counter(10)\nfor i in c:\n    print(i)\n    if i == 5:\n        c.close()  # stops as here is raises 'StopIteration' exception\n\n\ndef print_name(prefix):\n    print(\"Search for\", prefix, \"prefix\")\n    try:\n        while True:\n            name = yield\n            if prefix in name:\n                print(name)\n    except GeneratorExit:\n        print(\"Closing generator!\")\n\n\npn = print_name(\"Dear\")\nnext(pn)  # calls first yield expression\npn.send(\"Alex\")\npn.send(\"Dear Alex\")  # matches with prefix\n```\n\n**Create pipelines**\n\u003e Coroutines can be used to set pipes\n\n```python\n# materials/coroutine_chaining.py\n\ndef producer(sentence: str, next_coroutine):\n    \"\"\"Split strings and feed it to pattern_filter coroutine.\"\"\"\n    tokens = sentence.split(\" \")\n    for token in tokens:\n        next_coroutine.send(token)\n    next_coroutine.close()\n\n\ndef pattern_filter(pattern=\"ing\", next_coroutine=None):\n    \"\"\"Search for pattern and if pattern got matched, send it to print_token coroutine.\"\"\"\n    print(f\"Search for {pattern} pattern\")\n    try:\n        while True:\n            token = yield\n            if pattern in token:\n                next_coroutine.send(token)\n    except GeneratorExit:\n        print(\"Done with filtering\")\n\n\ndef print_token():\n    \"\"\"Act as a sink, simply print the token.\"\"\"\n    print(\"I'm sink, I'll print tokens\")\n    try:\n        while True:\n            token = yield\n            print(token)\n    except GeneratorExit:\n        print(\"Done with printing\")\n\n\npt = print_token()\nnext(pt)\npf = pattern_filter(next_coroutine=pt)\nnext(pf)\n\nsentence = \"Bob is running behind a fast moving car\"\nproducer(sentence, pf)\n```\n\n**Tricks**\n```python\n# materials/decorator.py\n\ndef coroutine(func):\n    \"\"\"A decorator function that eliminates the need to call .next() when starting a coroutine.\"\"\"\n    def start(*args, **kwargs):\n        cr = func(*args, **kwargs)\n        next(cr)\n        return cr\n    return start\n\n\nif __name__ == \"__main__\":\n    @coroutine\n    def grep(pattern):\n        print(f\"Search for '{pattern}' pattern\")\n        while True:\n            value = yield\n            if pattern in value:\n                print(value)\n\n\n    g = grep(\"python\")\n    # Notice now you don't need a next() call here\n    g.send(\"Yeah, but no, but yeah, but no\")\n    g.send(\"A series of tubes\")\n    g.send(\"python generators rock!\")\n```\n```python\n# materials/benchmark.py\n\nfrom timeit import timeit\nfrom materials.decorator import coroutine\n\n\n# An object\nclass GrepHandler:\n    def __init__(self, pattern, target):\n        self._pattern = pattern\n        self._target = target\n\n    def send(self, line):\n        if self._pattern in line:\n            self._target.send(line)\n\n\n# a coroutine\n@coroutine\ndef grep(pattern, target):\n    while True:\n        line = yield\n        if pattern in line:\n            target.send(line)\n\n\n# A null-sink to send data\n@coroutine\ndef null():\n    while True:\n        item = yield\n\n\nif __name__ == \"__main__\":\n    # A benchmark\n    line = \"python is nice\"\n    p1 = grep(\"python\", null())  # coroutine\n    p2 = GrepHandler(\"python\", null())  # an object\n\n    print(\"Coroutine: \", timeit(\"p1.send(line)\", \"from __main__ import line, p1\"))\n    print(\"Object: \", timeit(\"p2.send(line)\", \"from __main__ import line, p2\"))\n```\n```python\n# materials/broadcast.py\n\n\"\"\"\nAn example of broadcasting a data stream onto multiple coroutine targets.\n\"\"\"\n\nimport time\nfrom materials.decorator import coroutine\n\n\n# A data source. This is not a coroutine, but it sends data into one target\ndef follow(thefile, target):\n    thefile.seek(0, 2)  # Go to end of a file\n    while True:\n        line = thefile.readline()\n        if not line:\n            time.sleep(0.1)\n            continue\n        target.send(line)\n\n\n# A filter\n@coroutine\ndef grep(pattern, target):\n    while True:\n        line = yield  # Receive a line\n        if pattern in line:\n            target.send(line)  # Send to next stage\n\n\n# A sink. A coroutine that receives data\n@coroutine\ndef printer():\n    while True:\n        line = yield\n        print(line)\n\n\n# Broadcast a stream onto multiple targets\n@coroutine\ndef broadcast(targets):\n    while True:\n        item = yield\n        for target in targets:\n            target.send(item)\n\n\nif __name__ == \"__main__\":\n    f = open(\"access.log\", \"+a\")\n    follow(f, broadcast((grep(\"python\", printer()), grep(\"ply\", printer()), grep(\"swig\", printer()))))\n```\n## AsyncIO\n\u003e `Asynchronous IO` is a concurrent programming design (paradigm).\n\u003e `Coroutines` (specialized generator functions) are the heart of async IO in Python.\n\u003e \n\u003e `Parallelism` consists of performing multiple operations at the same time. \n\u003e Multiprocessing is a means to effect parallelism, and it entails spreading tasks over a computer’s central processing units (CPUs, or cores).\n\u003e \n\u003e `Concurrency` is a slightly broader term than parallelism. Multiple tasks have the ability to run in an overlapping manner.\n\u003e Concurrency (`concurrent.futures` package) include both multiprocessing and threading.\n\u003e\n\u003e `Threading` is a concurrent execution model whereby multiple threads take turns executing tasks. One process can contain multiple threads.\n\u003e\n\u003e `asyncio` is a library to write concurrent code. It is not threading, nor is it multiprocessing.\n\u003e In fact, async IO is a single-threaded, single-process design: it uses cooperative multitasking. \n\u003e Coroutines (a central feature of async IO) can be scheduled concurrently, but they are not inherently concurrent.\n\u003e\n\u003e async IO is a style of concurrent programming, but it is not parallelism. \n\u003e It’s more closely aligned with threading than with multiprocessing \n\u003e but is very much distinct from both of these and is a standalone member in concurrency’s bag of tricks\n\u003e \n\u003e What is `asynchronous` ?\n\u003e - Asynchronous routines are able to “pause” while waiting on their ultimate result and let other routines run in the meantime\n\u003e - Asynchronous code, facilitates concurrent execution\n\u003e\n\u003e Async IO takes long waiting periods in which functions would otherwise be blocking and allows other functions to run during that downtime\n\u003e\n\u003e `async` built on non-blocking sockets, callbacks and event loops.\n\u003e `async def` syntax stand for native coroutine or asynchronous generator.\n\u003e `await` keyword passes function control back to event loop. It suspends the execution of coroutine.\n\u003e \n```python\n# materials/async_.py\n\nimport asyncio\n\n\nasync def count():  # single event loop\n    print(\"One\")\n    await asyncio.sleep(1)  # when task reaches here it will sleep to 1 seconds ands says to do other job meantime\n    print(\"Two\")\n\n\nasync def main():\n    await asyncio.gather(count(), count(), count())\n\n\nif __name__ == \"__main__\":\n    import time\n    s = time.perf_counter()\n    asyncio.run(main())\n    elapsed = time.perf_counter() - s\n    print(f\"{__file__} executed in {elapsed:0.2f} seconds.\")\n```\n```python\n# materials/sync.py\n\nimport time\n\n\ndef count():\n    print(\"One\")\n    time.sleep(1)\n    print(\"Two\")\n\n\ndef main():\n    for _ in range(3):\n        count()\n\n\nif __name__ == \"__main__\":\n    s = time.perf_counter()\n    main()\n    elapsed = time.perf_counter() - s\n    print(f\"{__file__} executed in {elapsed:0.2f} seconds.\")  # 3.01 seconds\n```\n\u003e If Python encounters an await f() expression in the scope of g(), this is how await tells the event loop, \n\u003e “Suspend execution of g() until whatever I’m waiting on—the result of f() — is returned. In the meantime, go let something else run.”\n\u003e `async def` is a coroutine. It may use await, return, or yield, but all of these are optional.\n```python\nasync def g():\n    # Pause here and come back to g() when f() is ready\n    r = await f()\n    return r\n```\n\u003e Using `await` and/or `return` creates a `coroutine` function. \n\u003e To call a coroutine function, you must `await` it to get its results.\n\u003e\n\u003e Using `yield` in an `async def` block creates an asynchronous generator, which you iterate over with `async` for.\n\u003e `yield from` in an `async def` will raise SyntaxError.\n```python\n# materials/async_gen.py\n\nasync def genfunc():\n    yield 1\n    yield 2\n\ngen = genfunc()\nassert gen.__aiter__() is gen\nassert await gen.__anext__() == 1\nassert await gen.__anext__() == 2\nawait gen.__anext__()  # This line will raise StopAsyncIteration.\n```\n```python\nasync def f(x):\n    y = await z(x)  # OK - `await` and `return` allowed in coroutines\n    return y\n\nasync def g(x):\n    yield x  # OK - this is an async generator\n\nasync def m(x):\n    yield from gen(x)  # No - SyntaxError\n\ndef m(x):\n    y = await z(x)  # Still no - SyntaxError (no `async def` here)\n    return y\n```\n\n\n### Materials\n- https://docs.python.org/3/howto/functional.html#generator-expressions-and-list-comprehensions\n- https://www.python.org/dev/peps/pep-0289\n- https://www.python.org/dev/peps/pep-0342\n- https://www.python.org/dev/peps/pep-0525\n- https://docs.python.org/3/library/asyncio.html\n- https://docs.python.org/3.6/glossary.html#term-generator\n- https://realpython.com/introduction-to-python-generators\n- https://www.geeksforgeeks.org/coroutine-in-python\n- http://www.dabeaz.com/coroutines\n- https://realpython.com/async-io-python\n\n### Meta\nAuthor – Volodymyr Yahello vyahello@gmail.com\n\nDistributed under the `Apache (2.0)` license. See [LICENSE](LICENSE.md) for more information.\n\nYou can reach out me at:\n* [https://github.com/vyahello](https://github.com/vyahello)\n* [https://www.linkedin.com/in/volodymyr-yahello-821746127](https://www.linkedin.com/in/volodymyr-yahello-821746127)\n\n### Contributing\n1. clone the repository\n2. configure **git** for the first time after cloning with your `name` and `email`\n3. `pip install -r requirements.txt` to install all project dependencies\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvyahello%2Fpython-optimization-cheetsheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvyahello%2Fpython-optimization-cheetsheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvyahello%2Fpython-optimization-cheetsheet/lists"}