{"id":15914524,"url":"https://github.com/bbengfort/urfs","last_synced_at":"2025-04-03T03:43:59.316Z","repository":{"id":69222832,"uuid":"100630205","full_name":"bbengfort/urfs","owner":"bbengfort","description":"Perform computations on files in a large directory","archived":false,"fork":false,"pushed_at":"2017-08-18T22:12:39.000Z","size":64,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-02-08T17:45:47.292Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bbengfort.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-17T17:47:42.000Z","updated_at":"2019-03-09T06:22:17.000Z","dependencies_parsed_at":"2023-04-20T19:07:40.538Z","dependency_job_id":null,"html_url":"https://github.com/bbengfort/urfs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Furfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Furfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Furfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbengfort%2Furfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bbengfort","download_url":"https://codeload.github.com/bbengfort/urfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246933357,"owners_count":20857052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-06T17:03:59.121Z","updated_at":"2025-04-03T03:43:59.286Z","avatar_url":"https://github.com/bbengfort.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# URFS [![CircleCI](https://circleci.com/gh/bbengfort/urfs.svg?style=svg)](https://circleci.com/gh/bbengfort/urfs)\n\n**Perform computations on files in a large directory**\n\n## Usage\n\nTo install the utility, use `go get`:\n\n```bash\n$ go get github.com/bbengfort/urfs/...\n```\n\nIf you have added `$GOPATH/bin` to your `$PATH` then you will have the `urfs` utility installed and available:\n\n```bash\n$ urfs --help\n```\n\nThe `urfs` utility works on all files under a directory except for hidden files that start with a \".\" or a \"~\". Use the `--no-skip-dir` and `--no-skip-hidden` to include directories and hidden files. You can also filter directories using a glob like syntax on the file names. For example:\n\n```bash\n$ urfs -m *.txt cmd dir\n```\n\nWill only match files with a .txt extension. You can also specify a timeout to stop directory processing.\n\n```bash\n$ urfs -t 1m cmd dir\n```\n\nWill limit the command to only 1 minute of processing. There are a number of commands available in the utility, listed as follows:\n\n### Sample\n\nYou can sample a directory with a likelihood of 0.25 (approximately a quarter of the documents), copying the files to the destination directory as follows:\n\n```bash\n$ urfs sample -s 0.25 src/path dst/path\n```\n\nFor very large directories this may take a while, but should be faster than many other utilities.\n\n### Count\n\nYou can count the number of files and bytes in a directory as follows:\n\n```bash\n$ urfs count src/path/*\n```\n\nThis will return the number of files, bytes and average number of bytes per file for each of the paths passed to the utility.\n\n## Writing Commands\n\nURFS stands for \"uniform random file sample\", which was the original purpose of the command, still implemented as the `sample` command. It has since been generalized. To develop a parallel file system utility, simply create a `WalkFunc` and pass it to the `FSWalker.Walk` method.\n\nCreate an `FSWalker` as follows:\n\n```go\nfs := new(FSWalker)\nfs.Init(context.Background())\n```\n\nAt this point you can modify any of the walker's variables such as `Workers` or `Match` to modify operation.\n\nA function that operates concurrently on all paths is written as follows:\n\n```go\nfunc (fs *FSWalker) MyWalker(root string) error {\n\n    fs.Walk(root, func(path string) (string, error) {\n        // Do something with each path, note that this function\n        // is executed concurrently with other functions, so be\n        // sure to use appropriate synchronization mechanisms\n        return path, nil\n    })\n\n    // Access global results of the walker\n    fmt.Printf(\n        \"%d of %d paths executed in %d\\n\",\n        fs.nResults, fs.nPaths, fs.Duration,\n    )\n\n}\n```\n\nNote that each call of the `WalkFunc` happens concurrently, and is limited by the number of workers set in `fs.Workers` (to prevent too many files open or max number of threads reached).\n\nIf the `WalkFunc` returns an error, then processing is canceled. If the `WalkFunc` returns an empty string `\"\"` then the result is not counted. This allows you to correctly use the state of the walker on complete.\n\nIf you execute `fs.Walk` you'll need to reset the `FSWalker` in order to call `fs.Walk` a second time. Use `fs.Reset(nil)` to reset it with the original context.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbengfort%2Furfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbbengfort%2Furfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbengfort%2Furfs/lists"}