{"id":13478427,"url":"https://github.com/mosquito/aiofile","last_synced_at":"2025-10-07T10:33:50.006Z","repository":{"id":40447567,"uuid":"77929483","full_name":"mosquito/aiofile","owner":"mosquito","description":"Real asynchronous file operations with asyncio support.","archived":false,"fork":false,"pushed_at":"2025-04-23T06:03:09.000Z","size":289,"stargazers_count":538,"open_issues_count":16,"forks_count":30,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-23T07:20:07.987Z","etag":null,"topics":["aio","asyncio","file","filesystem","io","nfs","posix-systems","windows"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mosquito.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-03T15:32:01.000Z","updated_at":"2025-04-23T06:03:13.000Z","dependencies_parsed_at":"2024-01-13T19:21:50.735Z","dependency_job_id":"a156a724-0d39-4b26-8567-0104ea8a7964","html_url":"https://github.com/mosquito/aiofile","commit_stats":{"total_commits":206,"total_committers":14,"mean_commits":"14.714285714285714","dds":"0.11650485436893199","last_synced_commit":"0ff87df338bf6eee51fb6d04cdc7d320144e1c64"},"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosquito%2Faiofile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosquito%2Faiofile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosquito%2Faiofile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mosquito%2Faiofile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mosquito","download_url":"https://codeload.github.com/mosquito/aiofile/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270646,"owners_count":22042859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aio","asyncio","file","filesystem","io","nfs","posix-systems","windows"],"created_at":"2024-07-31T16:01:56.844Z","updated_at":"2025-10-07T10:33:49.900Z","avatar_url":"https://github.com/mosquito.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# AIOFile\n\n[![Github Actions](https://github.com/mosquito/aiofile/workflows/tox/badge.svg)](https://github.com/mosquito/aiofile/actions?query=branch%3Amaster) [![Latest Version](https://img.shields.io/pypi/v/aiofile.svg)](https://pypi.python.org/pypi/aiofile/) [![Wheel](https://img.shields.io/pypi/wheel/aiofile.svg)](https://pypi.python.org/pypi/aiofile/) [![Python Versions](https://img.shields.io/pypi/pyversions/aiofile.svg)](https://pypi.python.org/pypi/aiofile/) [![License](https://img.shields.io/pypi/l/aiofile.svg)](https://pypi.python.org/pypi/aiofile/) [![Coverage Status](https://coveralls.io/repos/github/mosquito/aiofile/badge.svg?branch=master)](https://coveralls.io/github/mosquito/aiofile?branch=master)\n\nReal asynchronous file operations with asyncio support.\n\n## Status\n\nDevelopment - Stable\n\n## Features\n\n* Since version 2.0.0 using [caio](https://pypi.org/project/caio), which contains linux `libaio` and two\n  thread-based implementations (c-based and pure-python).\n* AIOFile has no internal pointer. You should pass `offset` and\n  `chunk_size` for each operation or use helpers (Reader or Writer).\n  The simplest way is to use `async_open` for creating object with\n  file-like interface.\n* For Linux using implementation based on [libaio](https://pagure.io/libaio).\n* For POSIX (MacOS X and optional Linux) using implementation\n  based on [threadpool](https://github.com/mbrossard/threadpool/).\n* Otherwise using pure-python thread-based implementation.\n* Implementation chooses automatically depending on system compatibility.\n\n## Limitations\n\n* Linux native AIO implementation is not able to open special files.\n  Asynchronous operations against special fs like `/proc/` `/sys/` are not\n  supported by the kernel. It's not a aiofile's or caio issue.\n  In these cases, you might switch to thread-based implementations\n  (see [Troubleshooting](#troubleshooting) section).\n  However, when used on supported file systems, the linux implementation has a\n  smaller overhead and is preferred but it's not a silver bullet.\n\n## Code examples\n\nAll code examples require Python 3.6+.\n\n### High-level API\n\n#### `async_open` helper\n\nHelper mimics python file-like objects, it returns file-like\nobjects with similar but async methods.\n\nSupported methods:\n\n* `async def read(length = -1)` - reading chunk from file, when length is\n  `-1`, will be reading file to the end.\n* `async def write(data)` - writing chunk to file\n* `def seek(offset)` - setting file pointer position\n* `def tell()` - returns current file pointer position\n* `async def readline(size=-1, newline=\"\\n\")` - read chunks until\n  newline or EOF. Since version 3.7.0 `__aiter__` returns `LineReader`.\n\n  This method is suboptimal for small lines because it doesn't reuse read buffer.\n  When you want to read file by lines please avoid using `async_open`\n  use `LineReader` instead.\n* `def __aiter__() -\u003e LineReader` - iterator over lines.\n* `def iter_chunked(chunk_size: int = 32768) -\u003e Reader` - iterator over\n  chunks.\n* `.file` property contains AIOFile object\n\nBasic example:\n\n\u003c!-- name: test_basic --\u003e\n```python\nimport asyncio\nfrom pathlib import Path\nfrom tempfile import gettempdir\n\nfrom aiofile import async_open\n\ntmp_filename = Path(gettempdir()) / \"hello.txt\"\n\nasync def main():\n    async with async_open(tmp_filename, 'w+') as afp:\n        await afp.write(\"Hello \")\n        await afp.write(\"world\")\n        afp.seek(0)\n\n        print(await afp.read())\n\n        await afp.write(\"Hello from\\nasync world\")\n        print(await afp.readline())\n        print(await afp.readline())\n\nasyncio.run(main())\n```\n\nExample without context manager:\n\n\u003c!-- name: test_basic_without_context_manager --\u003e\n```python\nimport asyncio\nimport atexit\nimport os\nfrom tempfile import mktemp\n\nfrom aiofile import async_open\n\n\nTMP_NAME = mktemp()\natexit.register(os.unlink, TMP_NAME)\n\n\nasync def main():\n    afp = await async_open(TMP_NAME, \"w\")\n    await afp.write(\"Hello\")\n    await afp.close()\n\n\nasyncio.run(main())\nassert open(TMP_NAME, \"r\").read() == \"Hello\"\n```\n\nConcatenate example program (`cat`):\n\n```python\nimport asyncio\nimport sys\nfrom argparse import ArgumentParser\nfrom pathlib import Path\n\nfrom aiofile import async_open\n\nparser = ArgumentParser(\n    description=\"Read files line by line using asynchronous io API\"\n)\nparser.add_argument(\"file_name\", nargs=\"+\", type=Path)\n\nasync def main(arguments):\n    for src in arguments.file_name:\n        async with async_open(src, \"r\") as afp:\n            async for line in afp:\n                sys.stdout.write(line)\n\n\nasyncio.run(main(parser.parse_args()))\n```\n\nCopy file example program (`cp`):\n\n```python\nimport asyncio\nfrom argparse import ArgumentParser\nfrom pathlib import Path\n\nfrom aiofile import async_open\n\nparser = ArgumentParser(\n    description=\"Copying files using asynchronous io API\"\n)\nparser.add_argument(\"source\", type=Path)\nparser.add_argument(\"dest\", type=Path)\nparser.add_argument(\"--chunk-size\", type=int, default=65535)\n\n\nasync def main(arguments):\n    async with async_open(arguments.source, \"rb\") as src, \\\n               async_open(arguments.dest, \"wb\") as dest:\n        async for chunk in src.iter_chunked(arguments.chunk_size):\n            await dest.write(chunk)\n\n\nasyncio.run(main(parser.parse_args()))\n```\n\nExample with opening already open file pointer:\n\n```python\nimport asyncio\nfrom typing import IO, Any\nfrom aiofile import async_open\n\n\nasync def main(fp: IO[Any]):\n    async with async_open(fp) as afp:\n        await afp.write(\"Hello from\\nasync world\")\n        print(await afp.readline())\n\n\nwith open(\"test.txt\", \"w+\") as fp:\n    asyncio.run(main(fp))\n```\n\nLinux native aio doesn't support reading and writing special files\n(e.g. procfs/sysfs/unix pipes/etc.), so you can perform operations with\nthese files using compatible context objects.\n\n```python\nimport asyncio\nfrom aiofile import async_open\nfrom caio import thread_aio_asyncio\nfrom contextlib import AsyncExitStack\n\n\nasync def main():\n    async with AsyncExitStack() as stack:\n\n        # Custom context should be reused\n        ctx = await stack.enter_async_context(\n            thread_aio_asyncio.AsyncioContext()\n        )\n\n        # Open special file with custom context\n        src = await stack.enter_async_context(\n            async_open(\"/proc/cpuinfo\", \"r\", context=ctx)\n        )\n\n        # Open regular file with default context\n        dest = await stack.enter_async_context(\n            async_open(\"/tmp/cpuinfo\", \"w\")\n        )\n\n        # Copying file content line by line\n        async for line in src:\n            await dest.write(line)\n\n\nasyncio.run(main())\n```\n\n### Low-level API\n\nThe `AIOFile` class is a low-level interface for asynchronous file operations, and the read and write methods accept\nan `offset=0` in bytes at which the operation will be performed.\n\nThis allows you to do many independent IO operations on a once open file without moving the virtual carriage.\n\nFor example, you may make 10 concurrent HTTP requests by specifying the `Range` header, and asynchronously write\none opened file, while the offsets must either be calculated manually, or use 10 instances of `Writer` with\nspecified initial offsets.\n\nIn order to provide sequential reading and writing, there is `Writer`, `Reader` and `LineReader`. Keep in mind\n`async_open` is not the same as AIOFile, it provides a similar interface for file operations, it simulates methods\nlike read or write as it is implemented in the built-in open.\n\n```python\nimport asyncio\nfrom aiofile import AIOFile\n\n\nasync def main():\n    async with AIOFile(\"hello.txt\", 'w+') as afp:\n        payload = \"Hello world\\n\"\n\n        await asyncio.gather(\n            *[afp.write(payload, offset=i * len(payload)) for i in range(10)]\n        )\n\n        await afp.fsync()\n\n        assert await afp.read(len(payload) * 10) == payload * 10\n\nasyncio.run(main())\n```\n\nThe Low-level API in fact is just little bit sugared `caio` API.\n\n```python\nimport asyncio\nfrom aiofile import AIOFile\n\n\nasync def main():\n    async with AIOFile(\"/tmp/hello.txt\", 'w+') as afp:\n        await afp.write(\"Hello \")\n        await afp.write(\"world\", offset=7)\n        await afp.fsync()\n\n        print(await afp.read())\n\n\nasyncio.run(main())\n```\n\n#### `Reader` and `Writer`\n\nWhen you want to read or write file linearly following example\nmight be helpful.\n\n```python\nimport asyncio\nfrom aiofile import AIOFile, Reader, Writer\n\n\nasync def main():\n    async with AIOFile(\"/tmp/hello.txt\", 'w+') as afp:\n        writer = Writer(afp)\n        reader = Reader(afp, chunk_size=8)\n\n        await writer(\"Hello\")\n        await writer(\" \")\n        await writer(\"World\")\n        await afp.fsync()\n\n        async for chunk in reader:\n            print(chunk)\n\n\nasyncio.run(main())\n```\n\n#### `LineReader` - read file line by line\n\nLineReader is a helper that is very effective when you want to read a file\nlinearly and line by line.\n\nIt contains a buffer and will read the fragments of the file chunk by\nchunk into the buffer, where it will try to find lines.\n\nThe default chunk size is 4KB.\n\n```python\nimport asyncio\nfrom aiofile import AIOFile, LineReader, Writer\n\n\nasync def main():\n    async with AIOFile(\"/tmp/hello.txt\", 'w+') as afp:\n        writer = Writer(afp)\n\n        await writer(\"Hello\")\n        await writer(\" \")\n        await writer(\"World\")\n        await writer(\"\\n\")\n        await writer(\"\\n\")\n        await writer(\"From async world\")\n        await afp.fsync()\n\n        async for line in LineReader(afp):\n            print(line)\n\n\nasyncio.run(main())\n```\n\nWhen you want to read file by lines please avoid to use `async_open`\nuse `LineReader` instead.\n\n## More examples\n\nUseful examples with `aiofile`\n\n### Async CSV Dict Reader\n\n```python\nimport asyncio\nimport io\nfrom csv import DictReader\n\nfrom aiofile import AIOFile, LineReader\n\n\nclass AsyncDictReader:\n    def __init__(self, afp, **kwargs):\n        self.buffer = io.BytesIO()\n        self.file_reader = LineReader(\n            afp, line_sep=kwargs.pop('line_sep', '\\n'),\n            chunk_size=kwargs.pop('chunk_size', 4096),\n            offset=kwargs.pop('offset', 0),\n        )\n        self.reader = DictReader(\n            io.TextIOWrapper(\n                self.buffer,\n                encoding=kwargs.pop('encoding', 'utf-8'),\n                errors=kwargs.pop('errors', 'replace'),\n            ), **kwargs,\n        )\n        self.line_num = 0\n\n    def __aiter__(self):\n        return self\n\n    async def __anext__(self):\n        if self.line_num == 0:\n            header = await self.file_reader.readline()\n            self.buffer.write(header)\n\n        line = await self.file_reader.readline()\n\n        if not line:\n            raise StopAsyncIteration\n\n        self.buffer.write(line)\n        self.buffer.seek(0)\n\n        try:\n            result = next(self.reader)\n        except StopIteration as e:\n            raise StopAsyncIteration from e\n\n        self.buffer.seek(0)\n        self.buffer.truncate(0)\n        self.line_num = self.reader.line_num\n\n        return result\n\n\nasync def main():\n    async with AIOFile('sample.csv', 'rb') as afp:\n        async for item in AsyncDictReader(afp):\n            print(item)\n\n\nasyncio.run(main())\n```\n\n## Troubleshooting\n\nThe caio `linux` implementation works normal for modern linux kernel versions\nand file systems. So you may have problems specific for your environment.\nIt's not a bug and might be resolved some ways:\n\n1. Upgrade the kernel\n2. Use compatible file systems\n3. Use threads based or pure python implementation.\n\nThe caio since version 0.7.0 contains some ways to do this.\n\n1. In runtime use the environment variable `CAIO_IMPL` with\npossible values:\n\n    * `linux` - use native linux kernels aio mechanism\n    * `thread` - use thread based implementation written in C\n    * `python` - use pure python implementation\n\n2. File `default_implementation` located near `__init__.py` in caio\ninstallation path. It's useful for distros package maintainers. This file\nmight contains comments (lines starts with `#` symbol) and the first line\nshould be one of `linux` `thread` or `python`.\n\n3. You might manually manage contexts:\n\n```python\nimport asyncio\n\nfrom aiofile import async_open\nfrom caio import linux_aio_asyncio, thread_aio_asyncio\n\n\nasync def main():\n    linux_ctx = linux_aio_asyncio.AsyncioContext()\n    threads_ctx = thread_aio_asyncio.AsyncioContext()\n\n    async with async_open(\"/tmp/test.txt\", \"w\", context=linux_ctx) as afp:\n        await afp.write(\"Hello\")\n\n    async with async_open(\"/tmp/test.txt\", \"r\", context=threads_ctx) as afp:\n        print(await afp.read())\n\n\nasyncio.run(main())\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmosquito%2Faiofile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmosquito%2Faiofile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmosquito%2Faiofile/lists"}