{"id":13502105,"url":"https://github.com/benhoyt/scandir","last_synced_at":"2025-04-13T18:33:53.436Z","repository":{"id":8437102,"uuid":"10027254","full_name":"benhoyt/scandir","owner":"benhoyt","description":"Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib","archived":false,"fork":false,"pushed_at":"2023-08-29T09:32:19.000Z","size":680,"stargazers_count":533,"open_issues_count":6,"forks_count":69,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-04-06T15:07:02.541Z","etag":null,"topics":["directory","iterator","performance","python","scandir"],"latest_commit_sha":null,"homepage":"https://benhoyt.com/writings/scandir/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benhoyt.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-05-13T08:14:57.000Z","updated_at":"2025-03-25T07:13:58.000Z","dependencies_parsed_at":"2024-06-18T12:25:41.430Z","dependency_job_id":"411d26fc-e42e-4004-8d43-c8d2b4918022","html_url":"https://github.com/benhoyt/scandir","commit_stats":{"total_commits":413,"total_committers":22,"mean_commits":"18.772727272727273","dds":"0.21307506053268765","last_synced_commit":"34a0cc1dd2b8f31d6f8a859db7b287c491f50fd9"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benhoyt%2Fscandir","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benhoyt%2Fscandir/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benhoyt%2Fscandir/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benhoyt%2Fscandir/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benhoyt","download_url":"https://codeload.github.com/benhoyt/scandir/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248760440,"owners_count":21157359,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["directory","iterator","performance","python","scandir"],"created_at":"2024-07-31T22:02:02.014Z","updated_at":"2025-04-13T18:33:53.414Z","avatar_url":"https://github.com/benhoyt.png","language":"Python","funding_links":[],"categories":["Python","C","File \u0026 Path Utilities"],"sub_categories":[],"readme":"\nscandir, a better directory iterator and faster os.walk()\n=========================================================\n\n.. image:: https://img.shields.io/pypi/v/scandir.svg\n   :target: https://pypi.org/project/scandir/\n   :alt: scandir on PyPI (Python Package Index)\n\n.. image:: https://github.com/benhoyt/scandir/actions/workflows/tests.yml/badge.svg\n   :target: https://github.com/benhoyt/scandir/actions/workflows/tests.yml\n   :alt: GitHub Actions Tests\n\n\n``scandir()`` is a directory iteration function like ``os.listdir()``,\nexcept that instead of returning a list of bare filenames, it yields\n``DirEntry`` objects that include file type and stat information along\nwith the name. Using ``scandir()`` increases the speed of ``os.walk()``\nby 2-20 times (depending on the platform and file system) by avoiding\nunnecessary calls to ``os.stat()`` in most cases.\n\n\nNow included in a Python near you!\n----------------------------------\n\n``scandir`` has been included in the Python 3.5 standard library as\n``os.scandir()``, and the related performance improvements to\n``os.walk()`` have also been included. So if you're lucky enough to be\nusing Python 3.5 (release date September 13, 2015) you get the benefit\nimmediately, otherwise just\n`download this module from PyPI \u003chttps://pypi.python.org/pypi/scandir\u003e`_,\ninstall it with ``pip install scandir``, and then do something like\nthis in your code:\n\n.. code-block:: python\n\n    # Use the built-in version of scandir/walk if possible, otherwise\n    # use the scandir module version\n    try:\n        from os import scandir, walk\n    except ImportError:\n        from scandir import scandir, walk\n\n`PEP 471 \u003chttps://www.python.org/dev/peps/pep-0471/\u003e`_, which is the\nPEP that proposes including ``scandir`` in the Python standard library,\nwas `accepted \u003chttps://mail.python.org/pipermail/python-dev/2014-July/135561.html\u003e`_\nin July 2014 by Victor Stinner, the BDFL-delegate for the PEP.\n\nThis ``scandir`` module is intended to work on Python 2.7+ and Python\n3.4+ (and it has been tested on those versions).\n\n\nBackground\n----------\n\nPython's built-in ``os.walk()`` is significantly slower than it needs to be,\nbecause -- in addition to calling ``listdir()`` on each directory -- it calls\n``stat()`` on each file to determine whether the filename is a directory or not.\nBut both ``FindFirstFile`` / ``FindNextFile`` on Windows and ``readdir`` on Linux/OS\nX already tell you whether the files returned are directories or not, so\nno further ``stat`` system calls are needed. In short, you can reduce the number\nof system calls from about 2N to N, where N is the total number of files and\ndirectories in the tree.\n\nIn practice, removing all those extra system calls makes ``os.walk()`` about\n**7-50 times as fast on Windows, and about 3-10 times as fast on Linux and Mac OS\nX.** So we're not talking about micro-optimizations. See more benchmarks\nin the \"Benchmarks\" section below.\n\nSomewhat relatedly, many people have also asked for a version of\n``os.listdir()`` that yields filenames as it iterates instead of returning them\nas one big list. This improves memory efficiency for iterating very large\ndirectories.\n\nSo as well as a faster ``walk()``, scandir adds a new ``scandir()`` function.\nThey're pretty easy to use, but see \"The API\" below for the full docs.\n\n\nBenchmarks\n----------\n\nBelow are results showing how many times as fast ``scandir.walk()`` is than\n``os.walk()`` on various systems, found by running ``benchmark.py`` with no\narguments:\n\n====================   ==============   =============\nSystem version         Python version   Times as fast\n====================   ==============   =============\nWindows 7 64-bit       2.7.7 64-bit     10.4\nWindows 7 64-bit SSD   2.7.7 64-bit     10.3\nWindows 7 64-bit NFS   2.7.6 64-bit     36.8\nWindows 7 64-bit SSD   3.4.1 64-bit     9.9\nWindows 7 64-bit SSD   3.5.0 64-bit     9.5\nUbuntu 14.04 64-bit    2.7.6 64-bit     5.8\nMac OS X 10.9.3        2.7.5 64-bit     3.8\n====================   ==============   =============\n\nAll of the above tests were done using the fast C version of scandir\n(source code in ``_scandir.c``).\n\nNote that the gains are less than the above on smaller directories and greater\non larger directories. This is why ``benchmark.py`` creates a test directory\ntree with a standardized size.\n\n\nThe API\n-------\n\nwalk()\n~~~~~~\n\nThe API for ``scandir.walk()`` is exactly the same as ``os.walk()``, so just\n`read the Python docs \u003chttps://docs.python.org/3.5/library/os.html#os.walk\u003e`_.\n\nscandir()\n~~~~~~~~~\n\nThe full docs for ``scandir()`` and the ``DirEntry`` objects it yields are\navailable in the `Python documentation here \u003chttps://docs.python.org/3.5/library/os.html#os.scandir\u003e`_. \nBut below is a brief summary as well.\n\n    scandir(path='.') -\u003e iterator of DirEntry objects for given path\n\nLike ``listdir``, ``scandir`` calls the operating system's directory\niteration system calls to get the names of the files in the given\n``path``, but it's different from ``listdir`` in two ways:\n\n* Instead of returning bare filename strings, it returns lightweight\n  ``DirEntry`` objects that hold the filename string and provide\n  simple methods that allow access to the additional data the\n  operating system may have returned.\n\n* It returns a generator instead of a list, so that ``scandir`` acts\n  as a true iterator instead of returning the full list immediately.\n\n``scandir()`` yields a ``DirEntry`` object for each file and\nsub-directory in ``path``. Just like ``listdir``, the ``'.'``\nand ``'..'`` pseudo-directories are skipped, and the entries are\nyielded in system-dependent order. Each ``DirEntry`` object has the\nfollowing attributes and methods:\n\n* ``name``: the entry's filename, relative to the scandir ``path``\n  argument (corresponds to the return values of ``os.listdir``)\n\n* ``path``: the entry's full path name (not necessarily an absolute\n  path) -- the equivalent of ``os.path.join(scandir_path, entry.name)``\n\n* ``is_dir(*, follow_symlinks=True)``: similar to\n  ``pathlib.Path.is_dir()``, but the return value is cached on the\n  ``DirEntry`` object; doesn't require a system call in most cases;\n  don't follow symbolic links if ``follow_symlinks`` is False\n\n* ``is_file(*, follow_symlinks=True)``: similar to\n  ``pathlib.Path.is_file()``, but the return value is cached on the\n  ``DirEntry`` object; doesn't require a system call in most cases; \n  don't follow symbolic links if ``follow_symlinks`` is False\n\n* ``is_symlink()``: similar to ``pathlib.Path.is_symlink()``, but the\n  return value is cached on the ``DirEntry`` object; doesn't require a\n  system call in most cases\n\n* ``stat(*, follow_symlinks=True)``: like ``os.stat()``, but the\n  return value is cached on the ``DirEntry`` object; does not require a\n  system call on Windows (except for symlinks); don't follow symbolic links\n  (like ``os.lstat()``) if ``follow_symlinks`` is False\n\n* ``inode()``: return the inode number of the entry; the return value\n  is cached on the ``DirEntry`` object\n\nHere's a very simple example of ``scandir()`` showing use of the\n``DirEntry.name`` attribute and the ``DirEntry.is_dir()`` method:\n\n.. code-block:: python\n\n    def subdirs(path):\n        \"\"\"Yield directory names not starting with '.' under given path.\"\"\"\n        for entry in os.scandir(path):\n            if not entry.name.startswith('.') and entry.is_dir():\n                yield entry.name\n\nThis ``subdirs()`` function will be significantly faster with scandir\nthan ``os.listdir()`` and ``os.path.isdir()`` on both Windows and POSIX\nsystems, especially on medium-sized or large directories.\n\n\nFurther reading\n---------------\n\n* `The Python docs for scandir \u003chttps://docs.python.org/3.5/library/os.html#os.scandir\u003e`_\n* `PEP 471 \u003chttps://www.python.org/dev/peps/pep-0471/\u003e`_, the\n  (now-accepted) Python Enhancement Proposal that proposed adding\n  ``scandir`` to the standard library -- a lot of details here,\n  including rejected ideas and previous discussion\n\n\nFlames, comments, bug reports\n-----------------------------\n\nPlease send flames, comments, and questions about scandir to Ben Hoyt:\n\nhttp://benhoyt.com/\n\nFile bug reports for the version in the Python 3.5 standard library\n`here \u003chttps://docs.python.org/3.5/bugs.html\u003e`_, or file bug reports\nor feature requests for this module at the GitHub project page:\n\nhttps://github.com/benhoyt/scandir\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenhoyt%2Fscandir","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenhoyt%2Fscandir","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenhoyt%2Fscandir/lists"}