{"id":13821072,"url":"https://github.com/karrick/godirwalk","last_synced_at":"2025-10-03T21:01:24.151Z","repository":{"id":39737850,"uuid":"100993580","full_name":"karrick/godirwalk","owner":"karrick","description":"Fast directory traversal for Golang","archived":false,"fork":false,"pushed_at":"2023-04-26T14:25:28.000Z","size":256,"stargazers_count":711,"open_issues_count":16,"forks_count":69,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-05-14T02:14:28.007Z","etag":null,"topics":["directory-tree","golang-library","symbolic-links","unix","windows"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karrick.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-08-21T21:20:45.000Z","updated_at":"2025-05-08T11:12:26.000Z","dependencies_parsed_at":"2024-01-15T15:18:35.541Z","dependency_job_id":"7e6986d6-1b29-4db2-81ef-f1076e7b175f","html_url":"https://github.com/karrick/godirwalk","commit_stats":{"total_commits":212,"total_committers":10,"mean_commits":21.2,"dds":0.08490566037735847,"last_synced_commit":"9a7752c108e7ea76255201b9f47bd4d4d2df868e"},"previous_names":[],"tags_count":79,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgodirwalk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgodirwalk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgodirwalk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karrick%2Fgodirwalk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karrick","download_url":"https://codeload.github.com/karrick/godirwalk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254119449,"owners_count":22017947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["directory-tree","golang-library","symbolic-links","unix","windows"],"created_at":"2024-08-04T08:01:14.631Z","updated_at":"2025-10-03T21:01:19.097Z","avatar_url":"https://github.com/karrick.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# godirwalk\n\n`godirwalk` is a library for traversing a directory tree on a file\nsystem.\n\n[![GoDoc](https://godoc.org/github.com/karrick/godirwalk?status.svg)](https://godoc.org/github.com/karrick/godirwalk) [![Build Status](https://dev.azure.com/microsoft0235/microsoft/_apis/build/status/karrick.godirwalk?branchName=master)](https://dev.azure.com/microsoft0235/microsoft/_build/latest?definitionId=1\u0026branchName=master)\n\nIn short, why did I create this library?\n\n1. It's faster than `filepath.Walk`.\n1. It's more correct on Windows than `filepath.Walk`.\n1. It's more easy to use than `filepath.Walk`.\n1. It's more flexible than `filepath.Walk`.\n\nDepending on your specific circumstances, [you might no longer need a\nlibrary for file walking in\nGo](https://engineering.kablamo.com.au/posts/2021/quick-comparison-between-go-file-walk-implementations).\n\n## Usage Example\n\nAdditional examples are provided in the `examples/` subdirectory.\n\nThis library will normalize the provided top level directory name\nbased on the os-specific path separator by calling `filepath.Clean` on\nits first argument. However it always provides the pathname created by\nusing the correct os-specific path separator when invoking the\nprovided callback function.\n\n```Go\n    dirname := \"some/directory/root\"\n    err := godirwalk.Walk(dirname, \u0026godirwalk.Options{\n        Callback: func(osPathname string, de *godirwalk.Dirent) error {\n            // Following string operation is not most performant way\n            // of doing this, but common enough to warrant a simple\n            // example here:\n            if strings.Contains(osPathname, \".git\") {\n                return godirwalk.SkipThis\n            }\n            fmt.Printf(\"%s %s\\n\", de.ModeType(), osPathname)\n            return nil\n        },\n        Unsorted: true, // (optional) set true for faster yet non-deterministic enumeration (see godoc)\n    })\n```\n\nThis library not only provides functions for traversing a file system\ndirectory tree, but also for obtaining a list of immediate descendants\nof a particular directory, typically much more quickly than using\n`os.ReadDir` or `os.ReadDirnames`.\n\n## Description\n\nHere's why I use `godirwalk` in preference to `filepath.Walk`,\n`os.ReadDir`, and `os.ReadDirnames`.\n\n### It's faster than `filepath.Walk`\n\nWhen compared against `filepath.Walk` in benchmarks, it has been\nobserved to run between five and ten times the speed on darwin, at\nspeeds comparable to the that of the unix `find` utility; and about\ntwice the speed on linux; and about four times the speed on Windows.\n\nHow does it obtain this performance boost? It does less work to give\nyou nearly the same output. This library calls the same `syscall`\nfunctions to do the work, but it makes fewer calls, does not throw\naway information that it might need, and creates less memory churn\nalong the way by reusing the same scratch buffer for reading from a\ndirectory rather than reallocating a new buffer every time it reads\nfile system entry data from the operating system.\n\nWhile traversing a file system directory tree, `filepath.Walk` obtains\nthe list of immediate descendants of a directory, and throws away the\nnode type information for the file system entry that is provided by\nthe operating system that comes with the node's name. Then,\nimmediately prior to invoking the callback function, `filepath.Walk`\ninvokes `os.Stat` for each node, and passes the returned `os.FileInfo`\ninformation to the callback.\n\nWhile the `os.FileInfo` information provided by `os.Stat` is extremely\nhelpful--and even includes the `os.FileMode` data--providing it\nrequires an additional system call for each node.\n\nBecause most callbacks only care about what the node type is, this\nlibrary does not throw the type information away, but rather provides\nthat information to the callback function in the form of a\n`os.FileMode` value. Note that the provided `os.FileMode` value that\nthis library provides only has the node type information, and does not\nhave the permission bits, sticky bits, or other information from the\nfile's mode. If the callback does care about a particular node's\nentire `os.FileInfo` data structure, the callback can easiy invoke\n`os.Stat` when needed, and only when needed.\n\n#### Benchmarks\n\n##### macOS\n\n```Bash\n$ go test -bench=. -benchmem\ngoos: darwin\ngoarch: amd64\npkg: github.com/karrick/godirwalk\nBenchmarkReadDirnamesStandardLibrary-12   50000       26250  ns/op       10360  B/op       16  allocs/op\nBenchmarkReadDirnamesThisLibrary-12       50000       24372  ns/op        5064  B/op       20  allocs/op\nBenchmarkFilepathWalk-12                      1  1099524875  ns/op   228415912  B/op   416952  allocs/op\nBenchmarkGodirwalk-12                         2   526754589  ns/op   103110464  B/op   451442  allocs/op\nBenchmarkGodirwalkUnsorted-12                 3   509219296  ns/op   100751400  B/op   378800  allocs/op\nBenchmarkFlameGraphFilepathWalk-12            1  7478618820  ns/op  2284138176  B/op  4169453  allocs/op\nBenchmarkFlameGraphGodirwalk-12               1  4977264058  ns/op  1031105328  B/op  4514423  allocs/op\nPASS\nok  \tgithub.com/karrick/godirwalk\t21.219s\n```\n\n##### Linux\n\n```Bash\n$ go test -bench=. -benchmem\ngoos: linux\ngoarch: amd64\npkg: github.com/karrick/godirwalk\nBenchmarkReadDirnamesStandardLibrary-12  100000       15458  ns/op       10360  B/op       16  allocs/op\nBenchmarkReadDirnamesThisLibrary-12      100000       14646  ns/op        5064  B/op       20  allocs/op\nBenchmarkFilepathWalk-12                      2   631034745  ns/op   228210216  B/op   416939  allocs/op\nBenchmarkGodirwalk-12                         3   358714883  ns/op   102988664  B/op   451437  allocs/op\nBenchmarkGodirwalkUnsorted-12                 3   355363915  ns/op   100629234  B/op   378796  allocs/op\nBenchmarkFlameGraphFilepathWalk-12            1  6086913991  ns/op  2282104720  B/op  4169417  allocs/op\nBenchmarkFlameGraphGodirwalk-12               1  3456398824  ns/op  1029886400  B/op  4514373  allocs/op\nPASS\nok      github.com/karrick/godirwalk    19.179s\n```\n\n### It's more correct on Windows than `filepath.Walk`\n\nI did not previously care about this either, but humor me. We all love\nhow we can write once and run everywhere. It is essential for the\nlanguage's adoption, growth, and success, that the software we create\ncan run unmodified on all architectures and operating systems\nsupported by Go.\n\nWhen the traversed file system has a logical loop caused by symbolic\nlinks to directories, on unix `filepath.Walk` ignores symbolic links\nand traverses the entire directory tree without error. On Windows\nhowever, `filepath.Walk` will continue following directory symbolic\nlinks, even though it is not supposed to, eventually causing\n`filepath.Walk` to terminate early and return an error when the\npathname gets too long from concatenating endless loops of symbolic\nlinks onto the pathname. This error comes from Windows, passes through\n`filepath.Walk`, and to the upstream client running `filepath.Walk`.\n\nThe takeaway is that behavior is different based on which platform\n`filepath.Walk` is running. While this is clearly not intentional,\nuntil it is fixed in the standard library, it presents a compatibility\nproblem.\n\nThis library fixes the above problem such that it will never follow\nlogical file sytem loops on either unix or Windows. Furthermore, it\nwill only follow symbolic links when `FollowSymbolicLinks` is set to\ntrue. Behavior on Windows and other operating systems is identical.\n\n### It's more easy to use than `filepath.Walk`\n\nWhile this library strives to mimic the behavior of the incredibly\nwell-written `filepath.Walk` standard library, there are places where\nit deviates a bit in order to provide a more easy or intuitive caller\ninterface.\n\n#### Callback interface does not send you an error to check\n\nSince this library does not invoke `os.Stat` on every file system node\nit encounters, there is no possible error event for the callback\nfunction to filter on. The third argument in the `filepath.WalkFunc`\nfunction signature to pass the error from `os.Stat` to the callback\nfunction is no longer necessary, and thus eliminated from signature of\nthe callback function from this library.\n\nFurthermore, this slight interface difference between\n`filepath.WalkFunc` and this library's `WalkFunc` eliminates the\nboilerplate code that callback handlers must write when they use\n`filepath.Walk`. Rather than every callback function needing to check\nthe error value passed into it and branch accordingly, users of this\nlibrary do not even have an error value to check immediately upon\nentry into the callback function. This is an improvement both in\nruntime performance and code clarity.\n\n#### Callback function is invoked with OS specific file system path separator\n\nOn every OS platform `filepath.Walk` invokes the callback function\nwith a solidus (`/`) delimited pathname. By contrast this library\ninvokes the callback with the os-specific pathname separator,\nobviating a call to `filepath.Clean` in the callback function for each\nnode prior to actually using the provided pathname.\n\nIn other words, even on Windows, `filepath.Walk` will invoke the\ncallback with `some/path/to/foo.txt`, requiring well written clients\nto perform pathname normalization for every file prior to working with\nthe specified file. This is a hidden boilerplate requirement to create\ntruly os agnostic callback functions. In truth, many clients developed\non unix and not tested on Windows neglect this subtlety, and will\nresult in software bugs when someone tries to run that software on\nWindows.\n\nThis library invokes the callback function with `some\\path\\to\\foo.txt`\nfor the same file when running on Windows, eliminating the need to\nnormalize the pathname by the client, and lessen the likelyhood that a\nclient will work on unix but not on Windows.\n\nThis enhancement eliminates necessity for some more boilerplate code\nin callback functions while improving the runtime performance of this\nlibrary.\n\n#### `godirwalk.SkipThis` is more intuitive to use than `filepath.SkipDir`\n\nOne arguably confusing aspect of the `filepath.WalkFunc` interface\nthat this library must emulate is how a caller tells the `Walk`\nfunction to skip file system entries. With both `filepath.Walk` and\nthis library's `Walk`, when a callback function wants to skip a\ndirectory and not descend into its children, it returns\n`filepath.SkipDir`. If the callback function returns\n`filepath.SkipDir` for a non-directory, `filepath.Walk` and this\nlibrary will stop processing any more entries in the current\ndirectory. This is not necessarily what most developers want or\nexpect. If you want to simply skip a particular non-directory entry\nbut continue processing entries in the directory, the callback\nfunction must return nil.\n\nThe implications of this interface design is when you want to walk a\nfile system hierarchy and skip an entry, you have to return a\ndifferent value based on what type of file system entry that node\nis. To skip an entry, if the entry is a directory, you must return\n`filepath.SkipDir`, and if entry is not a directory, you must return\n`nil`. This is an unfortunate hurdle I have observed many developers\nstruggling with, simply because it is not an intuitive interface.\n\nHere is an example callback function that adheres to\n`filepath.WalkFunc` interface to have it skip any file system entry\nwhose full pathname includes a particular substring, `optSkip`. Note\nthat this library still supports identical behavior of `filepath.Walk`\nwhen the callback function returns `filepath.SkipDir`.\n\n```Go\n    func callback1(osPathname string, de *godirwalk.Dirent) error {\n        if optSkip != \"\" \u0026\u0026 strings.Contains(osPathname, optSkip) {\n            if b, err := de.IsDirOrSymlinkToDir(); b == true \u0026\u0026 err == nil {\n                return filepath.SkipDir\n            }\n            return nil\n        }\n        // Process file like normal...\n        return nil\n    }\n```\n\nThis library attempts to eliminate some of that logic boilerplate\nrequired in callback functions by providing a new token error value,\n`SkipThis`, which a callback function may return to skip the current\nfile system entry regardless of what type of entry it is. If the\ncurrent entry is a directory, its children will not be enumerated,\nexactly as if the callback had returned `filepath.SkipDir`. If the\ncurrent entry is a non-directory, the next file system entry in the\ncurrent directory will be enumerated, exactly as if the callback\nreturned `nil`. The following example callback function has identical\nbehavior as the previous, but has less boilerplate, and admittedly\nlogic that I find more simple to follow.\n\n```Go\n    func callback2(osPathname string, de *godirwalk.Dirent) error {\n        if optSkip != \"\" \u0026\u0026 strings.Contains(osPathname, optSkip) {\n            return godirwalk.SkipThis\n        }\n        // Process file like normal...\n        return nil\n    }\n```\n\n### It's more flexible than `filepath.Walk`\n\n#### Configurable Handling of Symbolic Links\n\nThe default behavior of this library is to ignore symbolic links to\ndirectories when walking a directory tree, just like `filepath.Walk`\ndoes. However, it does invoke the callback function with each node it\nfinds, including symbolic links. If a particular use case exists to\nfollow symbolic links when traversing a directory tree, this library\ncan be invoked in manner to do so, by setting the\n`FollowSymbolicLinks` config parameter to `true`.\n\n#### Configurable Sorting of Directory Children\n\nThe default behavior of this library is to always sort the immediate\ndescendants of a directory prior to visiting each node, just like\n`filepath.Walk` does. This is usually the desired behavior. However,\nthis does come at slight performance and memory penalties required to\nsort the names when a directory node has many entries. Additionally if\ncaller specifies `Unsorted` enumeration in the configuration\nparameter, reading directories is lazily performed as the caller\nconsumes entries. If a particular use case exists that does not\nrequire sorting the directory's immediate descendants prior to\nvisiting its nodes, this library will skip the sorting step when the\n`Unsorted` parameter is set to `true`.\n\nHere's an interesting read of the potential hazzards of traversing a\nfile system hierarchy in a non-deterministic order. If you know the\nproblem you are solving is not affected by the order files are\nvisited, then I encourage you to use `Unsorted`. Otherwise skip\nsetting this option.\n\n[Researchers find bug in Python script may have affected hundreds of studies](https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/)\n\n#### Configurable Post Children Callback\n\nThis library provides upstream code with the ability to specify a\ncallback function to be invoked for each directory after its children\nare processed. This has been used to recursively delete empty\ndirectories after traversing the file system in a more efficient\nmanner. See the `examples/clean-empties` directory for an example of\nthis usage.\n\n#### Configurable Error Callback\n\nThis library provides upstream code with the ability to specify a\ncallback to be invoked for errors that the operating system returns,\nallowing the upstream code to determine the next course of action to\ntake, whether to halt walking the hierarchy, as it would do were no\nerror callback provided, or skip the node that caused the error. See\nthe `examples/walk-fast` directory for an example of this usage.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarrick%2Fgodirwalk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarrick%2Fgodirwalk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarrick%2Fgodirwalk/lists"}