{"id":21413062,"url":"https://github.com/rixrix/file-nibbler","last_synced_at":"2026-05-18T01:39:12.557Z","repository":{"id":57235525,"uuid":"94954401","full_name":"rixrix/file-nibbler","owner":"rixrix","description":"Uses Async library called queue in processing hundreds, or thousands files and avoids the \"Too Many Open Files\" limit from your OS.","archived":false,"fork":false,"pushed_at":"2017-07-15T09:58:47.000Z","size":14,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-23T16:20:03.950Z","etag":null,"topics":["async","concurrency","files","filesystem","javascript","nodejs","queue","synchronous"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/file-nibbler","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rixrix.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-06-21T02:27:09.000Z","updated_at":"2017-08-27T08:16:59.000Z","dependencies_parsed_at":"2022-08-31T20:51:12.284Z","dependency_job_id":null,"html_url":"https://github.com/rixrix/file-nibbler","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rixrix%2Ffile-nibbler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rixrix%2Ffile-nibbler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rixrix%2Ffile-nibbler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rixrix%2Ffile-nibbler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rixrix","download_url":"https://codeload.github.com/rixrix/file-nibbler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243910761,"owners_count":20367544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","concurrency","files","filesystem","javascript","nodejs","queue","synchronous"],"created_at":"2024-11-22T18:16:38.996Z","updated_at":"2026-05-18T01:39:07.538Z","avatar_url":"https://github.com/rixrix.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/rixrix/file-nibbler.svg?branch=master)](https://travis-ci.org/rixrix/file-nibbler)\r\n[![npm version](https://badge.fury.io/js/file-nibbler.svg)](https://badge.fury.io/js/file-nibbler)\r\n\r\n# Nibbler\r\n\r\n\u003e /nɪb(ə)lə/ is a tool for cutting sheet metal with minimal distortion.\r\n\r\nIf you're like me you've probably encountered this error message:\r\n\r\n\u003e Error: EMFILE, too many open files\r\n\r\nThere are dime dozen solutions online but I find some of them a bit hard to fit into my workflow. A library/module should be easy enough to drop in and re-use my existing logic, IMHO.\r\n\r\nThis module uses the queue function from [Async](https://caolan.github.io/async/) to process each file bit by bit at your disposal.\r\n\r\nRather than poking around your file systems file descriptor limit - let's get rid of that route and instead do the less severe way by using queue at the user-space level. This is not perfect - I must say but somehow it helped me.\r\n\r\n## Install\r\n\r\nFrom a technical point of view the code is pretty small, just copy/paste and you are done however do the steps below if you want to proceed.\r\n\r\n### NPM\r\n\r\n```bash\r\n$\u003e npm install file-nibbler\r\n```\r\n\r\n### Yarn\r\n```bash\r\n$\u003e yarn add file-nibbler\r\n```\r\n\r\n## Usage\r\n\r\n```javascript\r\nvar fileNibbler = require('file-nibbler').nibbler;\r\nvar MILLION_FILES_DIR = 'full/path/to/my/files/';\r\n\r\n// This is your callback method for processing each file.\r\n// Note that it should return a Promise so that `file-nibbler` knows\r\n// when to call or process the next task from the queue\r\nfunction myFileProcessor(pathToFile) {\r\n    return new Promise(function(resolve, reject) {\r\n        // do some work on the file, and resolve it once done\r\n        resolve();\r\n    });\r\n}\r\n\r\nfileNibbler(MILLION_FILES_DIR, myFileProcessor)\r\n.then(function() {\r\n    // done, do your post processing\r\n})\r\n.catch(function(error) {\r\n    // when an error occurs\r\n});\r\n\r\n```\r\n\r\n## API\r\n\r\n### nibbler(path, [opts], callback(string))\r\nReturns a Promise.\r\n\r\n`path`\r\n\r\nString, path to files folder\r\n\r\nopts - Object, optional flags for various flags\r\n\r\n`opts.filterBy` is a string, for filtering out specific file(s) from the path. It only accepts regex-based string eg: `^((?!flash.exe).)*$`, which returns all the files except flash.exe.\r\n\r\nDefault: all files\r\n\r\n`opts.concurrencyLevel` is a number for setting up the concurrency level.\r\n\r\nDefault 5\r\n\r\n`callback`\r\n\r\nObject, required callback method wrapped in Promise. Nibbler calls this function once the item is in queue and passes the full path of the file as the first argument.\r\n\r\n\r\n## Note\r\n\r\nThe module at its core is pretty basic but I'm using on production for munching thousands and gigabytes of data on low end EC2 machine.\r\n\r\n## Todo\r\n\r\n* ~~better handling of async and sync processing of file/queue. Current use-case is sync-mode for my callback~~\r\n* Webpack\r\n* Test\r\n* Hook with TravisCI\r\n* Auto-publish to NPM, NuGet, Bower\r\n* ES6-ify\r\n* TypeScript definition file\r\n\r\n## References\r\n* [Async#queue](https://caolan.github.io/async/docs.html#queue)\r\n\r\n## License\r\n\r\nMIT\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frixrix%2Ffile-nibbler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frixrix%2Ffile-nibbler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frixrix%2Ffile-nibbler/lists"}