{"id":13471238,"url":"https://github.com/pazdera/gitwalk","last_synced_at":"2025-06-15T21:02:33.074Z","repository":{"id":23904988,"uuid":"27284988","full_name":"pazdera/gitwalk","owner":"pazdera","description":"Bulk processing of git repositories","archived":false,"fork":false,"pushed_at":"2018-10-17T15:13:07.000Z","size":360,"stargazers_count":121,"open_issues_count":7,"forks_count":13,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-15T09:43:03.100Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"CoffeeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pazdera.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-11-28T23:02:56.000Z","updated_at":"2024-11-04T13:37:04.000Z","dependencies_parsed_at":"2022-08-22T07:31:17.143Z","dependency_job_id":null,"html_url":"https://github.com/pazdera/gitwalk","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pazdera%2Fgitwalk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pazdera%2Fgitwalk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pazdera%2Fgitwalk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pazdera%2Fgitwalk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pazdera","download_url":"https://codeload.github.com/pazdera/gitwalk/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245662774,"owners_count":20652080,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:00:42.052Z","updated_at":"2025-03-26T13:30:54.761Z","avatar_url":"https://github.com/pazdera.png","language":"CoffeeScript","readme":"# gitwalk: Bulk processing of git repos\n\n[![npm badge](https://img.shields.io/npm/v/gitwalk.svg)](https://www.npmjs.com/package/gitwalk) \n[![npm downloads](https://img.shields.io/npm/dt/gitwalk.svg)](https://www.npmjs.com/package/gitwalk)\n\n**gitwalk** is a tool to manipulate multiple git repositories at once. You select\na group of repos using an [expression](https://github.com/pazdera/gitwalk#expressions)\nand provide an [operation](https://github.com/pazdera/gitwalk#processors) to be\ncompleted. Gitwalk will download the repos, iterate through them and run the\noperation for each one. This may be searching through files and commits, running tests\nand linters, editing files and pushing the changes back upstream \u0026mdash; whatever you can\nthink of.\n\n![gitwalk in the terminal](https://raw.githubusercontent.com/pazdera/gitwalk/master/screenshot.png)\n\n**gitwalk** is made in CoffeeScript and runs on Node.js. [nodegit](http://www.nodegit.org/) and\n[node-github](https://github.com/mikedeboer/node-github) do most of the heavy lifting in the background.\n\n## Features\n\n* Wildcard matching\n* Integrates directly with GitHub API\n* Works with private repos via GitHub auth\n* Authenticated pushes via ssh and http\n* Lets you define groups of repositories\n* Built-in search tool\n* Easily extensible with new [expressions](#expressions)\n  and [processors](#processors)\n* Usable from the CLI as well as JavaScript\n\n---\n\n* [Installation](#installation)\n* [Usage](#usage)\n* [Expressions](#expressions)\n* [Processors](#processors)\n* [Configuration](#configuration)\n* [JavaScript API](#javascript-api)\n* [Contributing](#contributing)\n* [Credits](#credits)\n* [Licence](#licence)\n\n---\n\n## Installation\n\nGitwalk is distributed as an [npm](https://www.npmjs.com/package/gitwalk) package.\nUse the following command to install it on your system:\n\n```bash\n$ npm install -g gitwalk\n```\n\nMake sure to include `-g` to get the CLI command in your `$PATH`.\n\n## Usage\n\nThe interface is pretty straight-forward. Here's the synopsis:\n\n```\ngitwalk [options] \u003cexpr...\u003e \u003cproc\u003e \u003cproc-args\u003e\n```\n\nThe first positional arguments are **expressions** that determine which\nrepositories will be processed. You can pass one or more of them at the same\ntime. The next one is your preferred **processor** which specifies what will\nhappen with the repositories, followed by any arguments that it takes \u0026mdash;\nusually a command, but different processors support different ones.\n\nCheck out the [expressions](#expressions) and [processors](#processors)\nsections below for the details.\n\n### Examples\n\nHere are a few quick examples that should work out of the box. Paste them\nin your terminal and see what happens.\n\n```bash\n$ gitwalk 'github:pazdera/tco:*' grep TODO\n\n$ gitwalk '**/*' command 'tree .'\n\n$ gitwalk 'github:pazdera/@(catpix|word_wrap)' commits 'echo \"#{sha}: #{summary}\"'\n```\n\n## Expressions\n\nExpressions determine which repositories will be processed by the `gitwalk`\ncommand. You can provide one or more of them and their results will be\ncombined. Check out the following examples:\n\n```bash\n# Matches all branches of the npm repo on GitHub.\n$ gitwalk 'github:npm/npm:*' ...\n\n# Matches all the git repositories in my home dir.\n$ gitwalk '~/**/*' ...\n\n# Use ^ to exclude previously matched repositories.\n# Matches all my repos on GitHub _except_ of scriptster.\n$ gitwalk 'github:pazdera/*' '^github:pazdera/scriptster' ...\n\n# URLs work too.\n$ gitwalk 'https://github.com/pazdera/tco.git:*' ...\n\n# You can predefine custom groups of repositories.\n# Check out the _Groups_ resolver below.\n$ gitwalk 'group:all-js' 'group:all-ruby' ...\n```\n\nWhat comes after the last colon in each expression is interpreted as **branch\nname**. If omitted, gitwalk assumes `master` by default. Also note that you can\nuse [globs](https://github.com/isaacs/node-glob#glob-primer) for certain parts.\nCheck out the detailed description of each resolver below for more information.\n\nUnder the hood, each type of expression is handled by a **resolver**. What\nfollows is a description of those that come with gitwalk by default. However,\nit's really easy to add new ones. See the [contribution\nguidelines](#contributing).\n\n### GitHub\n\nWith this resolver, you can match repositories directly on GitHub. Gitwalk\nwill make queries to their API and clone the repositories automatically.\n\n```\ngithub:\u003cusername\u003e/\u003crepo\u003e:\u003cbranch\u003e\n```\n\n* **username**: GitHub username or organisation.\n* **repo**: Which repositories to process (glob expressions allowed).\n* _[optional]_ **branch**:  Name of the branch to process (glob expressions allowed).\n\n#### Private repos\n\nTo be able to work with private repositories, you need to give gitwalk either\nyour credentials or auth token. Put this into your configuration file:\n\n```json\n{\n  \"resolvers\": {\n    \"github\": {\n      \"username\": \"example\",\n      \"password\": \"1337_h4x0r\",\n      \"token\": \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n    }\n  }\n}\n```\n\n#### Push access\n\nPushing to repositories on GitHub again requires authentication. You'll need\nto give gitwalk either your SSH keys or credentials via the config file. Here's\nhow to do it:\n\n```json\n{\n  \"git\": {\n    \"auth\": {\n      \"github-http-auth\": {\n        \"pattern\": \"^https?\\\\:\\\\/\\\\/github\\\\.com\",\n        \"username\": \"example\",\n        \"password\": \"1337_h4x0r\"\n      },\n      \"github-ssh-auth\": {\n        \"pattern\": \"^git@github\\\\.com\",\n        \"public_key\": \"/Users/radek/.ssh/id_rsa.pub\",\n        \"private_key\": \"/Users/radek/.ssh/id_rsa\",\n        \"passphrase\": \"\"\n      }\n    }\n  }\n}\n```\n\n### Glob\n\nYou can match repositories on your file system using\n[glob](https://en.wikipedia.org/wiki/Glob_(programming)). Note that gitwalk\nwill clone the repository even if it's stored locally on your computer. One\ncan never be too careful.\n\n```\n\u003cpath\u003e:\u003cbranch\u003e\n```\n\n* **path**: Location of the repositories (glob expressions allowed).\n* _[optional]_ **branch**:  Name of the branch to process (glob expressions allowed).\n\n### URL\n\nURLs aren't a problem eiter. Gitwalk will make a local copy of the remote\nrepository and process it. However, you can't do any wildcard matching with\nURLs.\n\n```\n\u003curl\u003e:\u003cbranch\u003e\n```\n\n* **url**: An URL pointing to a git repository (glob **not** allowed).\n* _[optional]_ **branch**:  Name of the branch to process (glob expressions allowed).\n\n### Groups\n\nYou can define custom groups of expressions that you can refer to by name\nlater on. There are no predefined groups.\n\n```\ngroup:\u003cname\u003e\n```\n\n* **name**: Name of the group.\n\n#### Configuring groups\n\nGroups of expressions can be defined in the\n[configuration file](#configuration) like this:\n\n```json\n{\n  \"resolvers\": {\n    \"groups\": {\n      \"all-ruby\": [\n        \"github:pazdera/tco\",\n        \"github:pazdera/scriptster\",\n        \"github:pazdera/catpix\"\n      ],\n      \"c\": [\n          \"github:pazdera/*itree*:*\",\n          \"^github:pazdera/e2fs*\"\n      ],\n      \"c++\": [\n        \"github:pazdera/libcity:*\",\n        \"github:pazdera/OgreCity:*\",\n        \"github:pazdera/pop3client:*\"\n      ]\n    }\n  }\n}\n```\n\n## Processors\n\nProcessors are tasks that you can run on each of the repositories that were\nmatched by your expressions. Just as with expressions, it's really easy to\nadd your own. Here are the ones that ship with gitwalk by default:\n\n```bash\n# Search for unfinished work in all JavaScript files\n$ gitwalk ... grep '(TODO|FIXME)' '**/*.js'\n\n# List all files in the repository\n$ gitwalk ... command 'tree .'\n\n# Another way to search the files\n$ gitwalk ... command 'git grep \"(TODO|FIXME)\"'\n\n# Replace the year in all Ruby files\n$ gitwalk ... files '**/*.rb' 'sed -i s/2015/2016/g #{file}'\n\n# Simple commit message profanity detector\n$ gitwalk ... commits 'grep \"(f.ck|sh.t|b.llocks)\" \u003c\u003c\u003c\"#{message}\"'\n```\n\nThe `#{hashCurlyBraces}` templates will be expanded into values before the\ncommand is executed. Each command exports different set of variables; check\nout the detailed descriptions below to find out more.\n\n### grep\n\nLine-by-line search through all the files in the working directory using\nregular expressions, similar to [grep](https://en.wikipedia.org/wiki/Grep).\n\n```bash\n$ gitwalk ... grep \u003cpattern\u003e [\u003cfiles\u003e]\n```\n\n* **regexp**: The pattern to look for.\n* _(optional)_ **files**: Limit the search to certain files only (glob\nexpressions allowed). Searches all files by default.\n\n### files\n\nRun a custom command for each file in the repository. The `path` parameter\nlets you match only certain files.\n\n```bash\n$ gitwalk ... files \u003cpath\u003e \u003ccommand\u003e\n```\n\n* **path**: Path filter (glob expressions allowed).\n* **command**: A command to run for each file\n  * **Expored vars**\n    * #{file}\n\n### commits\n\nRun a command for each\n\n```bash\n$ gitwalk ... commits \u003ccommand\u003e\n```\n\n* **command**: A command to run for each commit.\n  * **Expored vars**\n    * #{sha}\n    * #{summary}\n    * #{message}\n    * #{author}\n    * #{committer}\n\n### command\n\nRun a command per repository. This is useful for running custom scripts. The\ncurrent working directory for the command is set to the working tree of\ngitwalk's local clone. You should be able to use the `git` command as usual.\n\n```bash\n$ gitwalk ... command \u003ccommand\u003e\n```\n\n* **command**: A command to run for each repository.\n\n\n## Configuration\n\nProviding a config file isn't mandatory. However, you'll need one if you want\nto be able to git-push or access private repositories. Gitwalk loads the\nfollowing files:\n\n* `~/.gitwalk.(json|cson)`\n* `/etc/gitwalk.(json|cson)`\n\nBoth `json` and `cson` formats are accepted. Here's how a basic gitwalk config\nmight look like:\n\n```json\n{\n  \"resolvers\": {\n    \"github\": {\n      \"token\": \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n    },\n  },\n  \"git\": {\n    \"auth\": {\n      \"github-ssh\": {\n        \"pattern\": \"^git@github\\\\.com\",\n        \"public_key\": \"/Users/radek/.ssh/github.pub\",\n        \"private_key\": \"/Users/radek/.ssh/github\",\n        \"passphrase\": \"\"\n      }\n    }\n  },\n  \"logger\": {\n    \"level\": \"debug\"\n  }\n}\n```\n\n## JavaScript API\n\nThe JS API is similar to the CLI interface:\n\n```js\nvar gitwalk = require('gitwalk');\n\ngitwalk('github:pazdera/\\*', gitwalk.proc.grep(/TODO/, '*.js'), function (err) {\n    if (err) {\n        console.log 'gitwalk failed (' + err + ')';\n    }\n});\n```\n\n## Contributing\n\nIf you found a bug, please [open a new issue](https://github.com/pazdera/gitwalk/issues)\nwith as much relevant information as possible. And in case you'd like to jump\nin and get your hands dirty, here are a few ideas to get you started:\n\n* Add new resolvers (BitBucket and other services)\n* Add new processors (profanity search, integration with linters)\n* The tests are atrocious (please help me write them!)\n* Finish writing the API docs\n\nIf you start working on something, feel free to create an issue or drop me\na line to make sure you're not working on the same thing as somebody else.\n\n## Credits\n\nRadek Pazdera \u0026lt;me@radek.io\u0026gt; [http://radek.io](http://radek.io)\n\n## Licence\n\nPlease see [LICENCE](https://github.com/pazdera/gitwalk/blob/master/LICENCE).\n","funding_links":[],"categories":["CoffeeScript","Tools"],"sub_categories":["Comparing workflows"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpazdera%2Fgitwalk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpazdera%2Fgitwalk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpazdera%2Fgitwalk/lists"}