{"id":15708063,"url":"https://github.com/jedie/iterfilesystem","last_synced_at":"2026-02-19T18:04:30.190Z","repository":{"id":137348276,"uuid":"208579456","full_name":"jedie/IterFilesystem","owner":"jedie","description":"Multiprocess directory iteration via os.scandir() with progress indicator via tqdm bars.","archived":false,"fork":false,"pushed_at":"2020-03-17T11:50:49.000Z","size":177,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-22T14:10:33.313Z","etag":null,"topics":["filesystem","python","scandir","walk"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jedie.png","metadata":{"files":{"readme":"README.creole","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-15T10:48:31.000Z","updated_at":"2024-11-28T16:34:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"289acd40-8f86-4ea8-bf3e-9974b44f9588","html_url":"https://github.com/jedie/IterFilesystem","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/jedie/IterFilesystem","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jedie%2FIterFilesystem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jedie%2FIterFilesystem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jedie%2FIterFilesystem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jedie%2FIterFilesystem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jedie","download_url":"https://codeload.github.com/jedie/IterFilesystem/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jedie%2FIterFilesystem/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29626642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-19T18:02:07.722Z","status":"ssl_error","status_checked_at":"2026-02-19T18:01:46.144Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["filesystem","python","scandir","walk"],"created_at":"2024-10-03T20:45:17.921Z","updated_at":"2026-02-19T18:04:30.171Z","avatar_url":"https://github.com/jedie.png","language":"Python","funding_links":["https://www.paypal.me/JensDiemer"],"categories":[],"sub_categories":[],"readme":"== IterFilesystem\n\nMultiprocess directory iteration via {{{os.scandir()}}}\n\nWho's this Lib for?\n\nYou want to process a large number of files and/or a few very big files and give feedback to the user on how long it will take.\n\n\n=== Features:\n\n* Progress indicator:\n** Immediately after start: process files and indication of progress via multiprocess\n** process bars via [[https://pypi.org/project/tqdm/|tqdm]]\n** Estimated time based on file count and size\n* Easy to implement extra process bar for big file processing.\n* Skip directories and file name via fnmatch.\n\n\n=== How it works:\n\nThe main process starts //statistic// processes in background via Python multiprocess and starts directly with the work.\n\nThere are two background //statistic// processes collects information for the process bars:\n\n* Count up all directories and files.\n* Accumulates the sizes of all files.\n\nWhy two processes?\n\nBecause collect only the count of all filesystem items via {{{os.scandir()}}} is very fast. This is the fastest way to predict a processing time.\n\nUse {{{os.DirEntry.stat()}}} to get the file size is significantly slower: It requires another system call.\n\nOK, but why two processed?\n\nUse only the total count of all {{{DirEntry}}} may result in bad estimated time Progress indication.\nIt depends on what the actual work is about: When processing the contents of large files, it is good to know how much total data to be processed.\n\nThat's why we used two ways: the {{{DirEntry}}} count to forecast a processing time very quickly and the size to improve the predicted time.\n\n\n=== requirements:\n\n * Python 3.6 or newer.\n * {{{tqdm}}} for process bars\n * {{{psutils}}} for setting process priority\n * For dev.: [[https://pipenv.readthedocs.io/en/latest/install/#installing-pipenv|Pipenv]]. Packages and virtual environment manager\n\n\n=== contribute\n\nPlease: try, fork and contribute! ;)\n\n| {{https://travis-ci.org/jedie/IterFilesystem.svg|Build Status on travis-ci.org}} | [[https://travis-ci.org/jedie/IterFilesystem/|travis-ci.org/jedie/IterFilesystem]] |\n| {{https://ci.appveyor.com/api/projects/status/py5sl38ql3xciafc?svg=true|Build Status on appveyor.com}} | [[https://ci.appveyor.com/project/jedie/IterFilesystem/history|ci.appveyor.com/project/jedie/IterFilesystem]] |\n| {{https://codecov.io/gh/jedie/IterFilesystem/branch/master/graph/badge.svg|Coverage Status on codecov.io}} | [[https://codecov.io/gh/jedie/IterFilesystem|codecov.io/gh/jedie/IterFilesystem]] |\n| {{https://coveralls.io/repos/jedie/IterFilesystem/badge.svg|Coverage Status on coveralls.io}} | [[https://coveralls.io/r/jedie/IterFilesystem|coveralls.io/r/jedie/IterFilesystem]] |\n| {{https://requires.io/github/jedie/IterFilesystem/requirements.svg?branch=master|Requirements Status on requires.io}} | [[https://requires.io/github/jedie/IterFilesystem/requirements/|requires.io/github/jedie/IterFilesystem/requirements/]] |\n\n\n== Example\n\nUse example CLI, e.g.:\n\n{{{\n~$ git clone https://github.com/jedie/IterFilesystem.git\n~$ cd IterFilesystem\n~/IterFilesystem$ pipenv install\n~/IterFilesystem$ pipenv shell\n(IterFilesystem) ~/IterFilesystem$ print_fs_stats --help\n(IterFilesystem) ~/IterFilesystem$ pip install -e .\n...\nSuccessfully installed iterfilesystem\n\n~/IterFilesystem$ $ poetry run print_fs_stats --help\nusage: print_fs_stats.py [-h] [-v] [--debug] [--path PATH]\n                         [--skip_dir_patterns [SKIP_DIR_PATTERNS [SKIP_DIR_PATTERNS ...]]]\n                         [--skip_file_patterns [SKIP_FILE_PATTERNS [SKIP_FILE_PATTERNS ...]]]\n\nScan filesystem and print some information\n\noptional arguments:\n  -h, --help            show this help message and exit\n  -v, --version         show program's version number and exit\n  --debug               enable DEBUG\n  --path PATH           The file path that should be scanned e.g.: \"~/foobar/\"\n                        default is \"~\"\n  --skip_dir_patterns [SKIP_DIR_PATTERNS [SKIP_DIR_PATTERNS ...]]\n                        Directory names to exclude from scan.\n  --skip_file_patterns [SKIP_FILE_PATTERNS [SKIP_FILE_PATTERNS ...]]\n                        File names to ignore.\n}}}\n\nexample output looks like this:\n\n{{{\n(IterFilesystem) ~/IterFilesystem$ $ print_fs_stats --path ~/IterFilesystem --skip_dir_patterns \".*\" \"*.egg-info\" --skip_file_patterns \".*\"\nRead/process: '~/IterFilesystem'...\nSkip directory patterns:\n\t* .*\n\t* *.egg-info\n\nSkip file patterns:\n\t* .*\n\nFilesystem items..:Read/process: '~/IterFilesystem'...\n\n...\n\nFilesystem items..: 100%|█████████████████████████████████████████|135/135 13737.14entries/s [00:00\u003c00:00, 13737.14entries/s]\nFile sizes........: 100%|██████████████████████████████████████████████████████████████|843k/843k [00:00\u003c00:00, 88.5MBytes/s]\nAverage progress..: 100%|███████████████████████████████████████████████████████████████████████████████████████|00:00\u003c00:00\nCurrent File......:, /home/jens/repos/IterFilesystem/Pipfile\n\n\nProcessed 135 filesystem items in 0.02 sec\nSHA515 hash calculated over all file content: 10f9475b21977f5aea1d4657a0e09ad153a594ab30abc2383bf107dbc60c430928596e368ebefab3e78ede61dcc101cb638a845348fe908786cb8754393439ef\nFile count: 109\nTotal file size: 843.5 KB\n6 directories skipped.\n6 files skipped.\n}}}\n\n\n== History\n\n* [[https://github.com/jedie/IterFilesystem/compare/v1.4.3...master|**dev** - compare v1.4.3...master]]\n** TBC\n* [[https://github.com/jedie/IterFilesystem/compare/v1.4.2...v1.4.3|16.03.2020 - v1.4.3]]\n** Use logging and remove \"verbose mode\"\n** Nicer \"Average progess\" bar\n** Bugfix \"Current File\" bar: remove comma\n* [[https://github.com/jedie/IterFilesystem/compare/v1.4.1...v1.4.2|16.02.2020 - v1.4.2]]\n** iterate over sorted dir entries\n** update CI pipelines\n* [[https://github.com/jedie/IterFilesystem/compare/v1.4.0...v1.4.1|02.02.2020 - v1.4.1]]\n** Bugfix {{{human_filesize}}}\n* [[https://github.com/jedie/IterFilesystem/compare/v1.3.1...v1.4.0|02.02.2020 - v1.4.0]]\n** {{{stats_helper.abort}}} exists always usefull to get information if KeyboardInterrupt was used\n** use poetry and modernize project setup\n* [[https://github.com/jedie/IterFilesystem/compare/v1.3.0...v1.3.1|20.10.2019 - v1.3.1]]\n** Bugfix if scan directory is completely empty\n* [[https://github.com/jedie/IterFilesystem/compare/v1.2.0...v1.3.0|13.10.2019 - v1.3.0]]\n** Set ionice and nice priority via psutils\n* [[https://github.com/jedie/IterFilesystem/compare/v1.1.0...v1.2.0|13.10.2019 - v1.2.0]]\n** Refactor API\n** cleanup statistics and process bar\n** handle access errors like: //Permission denied//\n** fix tests\n* [[https://github.com/jedie/IterFilesystem/compare/v1.0.0...v1.1.0|12.10.2019 - v1.1.0]]\n** don't create separate process for worker: Just do the work in main process\n** dir/file filter uses now {{{fnmatch}}}\n* [[https://github.com/jedie/IterFilesystem/compare/v0.2.0...v1.0.0|12.10.2019 - v1.0.0]]\n** refactoring:\n*** don't use {{{persist-queue}}}\n*** switch from threading to multiprocessing\n*** enhance progress display with multiple {{{tqdm}}} process bars\n* [[https://github.com/jedie/IterFilesystem/compare/v0.1.0...v0.2.0|15.09.2019 - v0.2.0]]\n** store persist queue in temp directory\n** Don't catch {{{process_path_item}}} errors, this should be made in child class\n* [[https://github.com/jedie/IterFilesystem/compare/v0.0.1...v0.1.0|15.09.2019 - v0.1.0]]\n** add some project meta files and tests\n** setup CI\n** fix tests\n* [[https://github.com/jedie/IterFilesystem/commit/db89a467a548a969d9d2cdd48adb92114a8833fe|15.09.2019 - v0.0.1]]\n** first Release on PyPi\n\n\n== Links\n\n* https://pypi.python.org/pypi/IterFilesystem/\n* https://github.com/jedie/IterFilesystem/\n\n\n== Donating\n\n* [[https://www.paypal.me/JensDiemer|paypal.me/JensDiemer]]\n* [[https://flattr.com/submit/auto?uid=jedie\u0026url=https%3A%2F%2Fgithub.com%2Fjedie%2FIterFilesystem%2F|Flattr This!]]\n* Send [[http://www.bitcoin.org/|Bitcoins]] to [[https://blockexplorer.com/address/1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6F|1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6F]]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjedie%2Fiterfilesystem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjedie%2Fiterfilesystem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjedie%2Fiterfilesystem/lists"}