{"id":16965435,"url":"https://github.com/kafkas/firewalk","last_synced_at":"2025-10-19T09:03:25.982Z","repository":{"id":39907696,"uuid":"361182226","full_name":"kafkas/firewalk","owner":"kafkas","description":"A light, fast, and memory-efficient collection traversal library for Firestore and Node.js","archived":false,"fork":false,"pushed_at":"2024-12-17T18:11:32.000Z","size":2983,"stargazers_count":195,"open_issues_count":7,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-08T13:11:13.616Z","etag":null,"topics":["cloud-firestore","firebase","javascript","nodejs","typescript"],"latest_commit_sha":null,"homepage":"https://kafkas.github.io/firewalk","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kafkas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-24T14:24:32.000Z","updated_at":"2025-04-07T07:40:57.000Z","dependencies_parsed_at":"2024-10-26T21:19:56.862Z","dependency_job_id":"44e0dfd7-b56e-418f-aa99-8c812335434a","html_url":"https://github.com/kafkas/firewalk","commit_stats":{"total_commits":506,"total_committers":3,"mean_commits":"168.66666666666666","dds":0.07114624505928857,"last_synced_commit":"bde200fa246b109b01bc1897d78cb4f77ac1d51e"},"previous_names":["kafkas/firewalk"],"tags_count":47,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkas%2Ffirewalk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkas%2Ffirewalk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkas%2Ffirewalk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kafkas%2Ffirewalk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kafkas","download_url":"https://codeload.github.com/kafkas/firewalk/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254459083,"owners_count":22074604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloud-firestore","firebase","javascript","nodejs","typescript"],"created_at":"2024-10-13T23:46:08.863Z","updated_at":"2025-10-19T09:03:25.957Z","avatar_url":"https://github.com/kafkas.png","language":"TypeScript","readme":"\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"images/logo.png\" width=\"240px\" alt=\"header\" /\u003e\n\u003c/div\u003e\n\n\u003ch1 align=\"center\"\u003e\n  firewalk\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  A light, fast, and memory-efficient collection traversal library for Firestore and Node.js.\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/kafkas/firewalk/blob/main/LICENSE\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"Firewalk is released under the MIT license.\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://npmjs.com/package/firewalk\" alt=\"Version\"\u003e\n        \u003cimg src=\"https://img.shields.io/npm/v/firewalk\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://npmjs.com/package/firewalk\" alt=\"Size\"\u003e\n        \u003cimg src=\"https://img.shields.io/bundlephobia/min/firewalk\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://npmjs.com/package/firewalk\" alt=\"Downloads\"\u003e\n        \u003cimg src=\"https://img.shields.io/npm/dm/firewalk\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://\" alt=\"Types\"\u003e\n        \u003cimg src=\"https://img.shields.io/npm/types/firewalk\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/kafkas/firewalk\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/PRs-welcome-brightgreen.svg\" alt=\"PRs welcome!\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nFirewalk is a Node.js library that _walks_ you through Firestore collections.\n\nWhen you have millions of documents in a collection and you need to make changes to them or just read them, you can't just retrieve all of them at once as your program's memory usage will explode. Firewalk's configurable traverser objects let you do this in a simple, intuitive and memory-efficient way using batch processing with concurrency control.\n\nFirewalk is an extremely light and well-typed library that is useful in a variety of scenarios. You can use it in database migration scripts (e.g. when you need to add a new field to all docs) or a scheduled Cloud Function that needs to check every doc in a collection periodically or even a locally run script that retrieves some data from a collection.\n\n[**Read the introductory blog post ▸**](https://anarkafkas.medium.com/traversing-firestore-collections-efficiently-6e43cea1eefd)\n\n[**View the full documentation (docs) ▸**](https://kafkas.github.io/firewalk)\n\n## Overview\n\n1. [Prerequisites](#Prerequisites)\n   1. [Compatibility Map](#Compatibility-Map)\n1. [Installation](#Installation)\n1. [Core Concepts](#Core-Concepts)\n1. [Quick Start](#Quick-Start)\n1. [More Examples](#More-Examples)\n1. [API](#API)\n1. [Upgrading](#Upgrading)\n1. [License](#License)\n\n## Prerequisites\n\nFirewalk is designed to work with the [Firebase Admin SDK](https://github.com/firebase/firebase-admin-node) so if you haven't already installed it, you'll need add it as a dependency to your project.\n\n```bash\nnpm install firebase-admin\n```\n\n### Compatibility Table\n\nMake sure to install the right version of Firewalk depending on the `firebase-admin` version your project is on.\n\n| firewalk | firebase-admin |\n| -------- | -------------- |\n| v1       | v9-10          |\n| v2       | v11-13         |\n\n## Installation\n\nYou can add Firewalk to your project with npm or yarn.\n\n```bash\nnpm install firewalk\n```\n\n## Core Concepts\n\nThere are only 2 kinds of objects you need to be familiar with when using this library:\n\n1. **Traverser**: An object that walks you through a collection of documents (or more generally a [Traversable](https://kafkas.github.io/firewalk/v2/types/Traversable.html)).\n\n2. **Migrator**: A convenience object used for database migrations. It lets you easily write to the documents within a given traversable and uses a traverser to do that. You can easily write your own migration logic in the traverser callback if you don't want to use a migrator.\n\n## Quick Start\n\nSuppose we have a `users` collection and we want to send an email to each user. This is how easy it is to do that efficiently with a Firewalk traverser:\n\n```ts\nimport { firestore } from 'firebase-admin';\nimport { createTraverser } from 'firewalk';\n\nconst usersCollection = firestore().collection('users');\nconst traverser = createTraverser(usersCollection);\n\nconst { batchCount, docCount } = await traverser.traverse(async (batchDocs, batchIndex) =\u003e {\n  const batchSize = batchDocs.length;\n  await Promise.all(\n    batchDocs.map(async (doc) =\u003e {\n      const { email, firstName } = doc.data();\n      await sendEmail({ to: email, content: `Hello ${firstName}!` });\n    })\n  );\n  console.log(`Batch ${batchIndex} done! We emailed ${batchSize} users in this batch.`);\n});\n\nconsole.log(`Traversal done! We emailed ${docCount} users in ${batchCount} batches!`);\n```\n\nWe are doing 3 things here:\n\n1. Create a reference to the `users` collection\n2. Pass that reference to the `createTraverser()` function\n3. Invoke `.traverse()` with an async callback that is called for each batch of document snapshots\n\nThis pretty much sums up the core functionality of this library! The `.traverse()` method returns a Promise that resolves when the entire traversal finishes, which can take a while if you have millions of docs. The Promise resolves with an object containing the traversal details e.g. the number of docs you touched.\n\n## More Examples\n\n### Traverse faster by increasing concurrency\n\n```ts\nconst projectsColRef = firestore().collection('projects');\nconst traverser = createTraverser(projectsColRef, {\n  batchSize: 500,\n  // This means we are prepared to hold 500 * 20 = 10,000 docs in memory.\n  // We sacrifice some memory to traverse faster.\n  maxConcurrentBatchCount: 20,\n});\nconst { docCount } = await traverser.traverse(async (_, batchIndex) =\u003e {\n  console.log(`Gonna process batch ${batchIndex} now!`);\n  // ...\n});\nconsole.log(`Traversed ${docCount} projects super-fast!`);\n```\n\n### Add a new field using a migrator\n\n```ts\nconst projectsColRef = firestore().collection('projects');\nconst migrator = createMigrator(projectsColRef);\nconst { migratedDocCount } = await migrator.update('isCompleted', false);\nconsole.log(`Updated ${migratedDocCount} projects!`);\n```\n\n### Add a new field derived from the previous fields\n\n```ts\ntype UserDoc = {\n  firstName: string;\n  lastName: string;\n};\nconst usersColRef = firestore().collection('users') as firestore.CollectionReference\u003cUserDoc\u003e;\nconst migrator = createMigrator(usersColRef);\nconst { migratedDocCount } = await migrator.updateWithDerivedData((snap) =\u003e {\n  const { firstName, lastName } = snap.data();\n  return {\n    fullName: `${firstName} ${lastName}`,\n  };\n});\nconsole.log(`Updated ${migratedDocCount} users!`);\n```\n\n### Migrate faster by increasing concurrency\n\n```ts\nconst projectsColRef = firestore().collection('projects');\nconst migrator = createMigrator(projectsColRef, { maxConcurrentBatchCount: 25 });\nconst { migratedDocCount } = await migrator.update('isCompleted', false);\nconsole.log(`Updated ${migratedDocCount} projects super-fast!`);\n```\n\n### Change traversal config\n\n```ts\nconst walletsWithNegativeBalance = firestore().collection('wallets').where('money', '\u003c', 0);\nconst migrator = createMigrator(walletsWithNegativeBalance, {\n  // We want each batch to have 500 docs. The size of the very last batch may be less than 500\n  batchSize: 500,\n  // We want to wait 500ms before moving to the next batch\n  sleepTimeBetweenBatches: 500,\n});\n// Wipe out their debts!\nconst { migratedDocCount } = await migrator.set({ money: 0 });\nconsole.log(`Set ${migratedDocCount} wallets!`);\n```\n\n### Rename a field\n\n```ts\nconst postsColGroup = firestore().collectionGroup('posts');\nconst migrator = createMigrator(postsColGroup);\nconst { migratedDocCount } = await migrator.renameField('postedAt', 'publishedAt');\nconsole.log(`Updated ${migratedDocCount} posts!`);\n```\n\n## [API](https://kafkas.github.io/firewalk)\n\nYou can find the full API reference for Firewalk [here](https://kafkas.github.io/firewalk). We maintain detailed docs for each major version. Here are some of the core functions that this library provides.\n\n### [createTraverser](https://kafkas.github.io/firewalk/v2/functions/createTraverser.html)\n\nCreates an object which can be used to traverse a Firestore collection or, more generally, a [Traversable](https://kafkas.github.io/firewalk/v2/types/Traversable.html).\n\nFor each batch of document snapshots in the traversable, the traverser invokes a specified async callback and immediately moves to the next batch. It does not wait for the callback Promise to resolve before moving to the next batch. That is, when `maxConcurrentBatchCount` \u003e 1, there is no guarantee that any given batch will finish processing before a later batch.\n\nThe traverser becomes faster as you increase `maxConcurrentBatchCount`, but this will consume more memory. You should increase concurrency when you want to trade some memory for speed.\n\n#### Complexity:\n\n- Time complexity: _O_((_N_ / `batchSize`) \\* (_Q_(`batchSize`) + _C_(`batchSize`) / `maxConcurrentBatchCount`))\n- Space complexity: _O_(`maxConcurrentBatchCount` \\* (`batchSize` \\* _D_ + _S_))\n- Billing: _max_(1, _N_) reads\n\nwhere:\n\n- _N_: number of docs in the traversable\n- _Q_(`batchSize`): average batch query time\n- _C_(`batchSize`): average callback processing time\n- _D_: average document size\n- _S_: average extra space used by the callback\n\n### [createMigrator](https://kafkas.github.io/firewalk/v2/functions/createMigrator.html)\n\nCreates a migrator that facilitates database migrations. The migrator accepts a custom traverser to traverse the collection. Otherwise it will create a default traverser with your desired traversal config. This migrator does not use atomic batch writes so it is possible that when a write fails other writes go through.\n\n#### Complexity:\n\n- Time complexity: _TC_(`traverser`) where _C_(`batchSize`) = _W_(`batchSize`)\n- Space complexity: _SC_(`traverser`) where _S_ = _O_(`batchSize`)\n- Billing: _max_(1, _N_) reads, _K_ writes\n\nwhere:\n\n- _N_: number of docs in the traversable\n- _K_: number of docs that passed the migration predicate (_K_\u003c=_N_)\n- _W_(`batchSize`): average batch write time\n- _TC_(`traverser`): time complexity of the underlying traverser\n- _SC_(`traverser`): space complexity of the underlying traverser\n\n### [createBatchMigrator](https://kafkas.github.io/firewalk/v2/functions/createBatchMigrator.html)\n\nCreates a migrator that facilitates database migrations. The migrator accepts a custom traverser to traverse the collection. Otherwise it will create a default traverser with your desired traversal config. This migrator uses atomic batch writes so the entire operation will fail if a single write isn't successful.\n\n#### Complexity:\n\n- Time complexity: _TC_(`traverser`) where _C_(`batchSize`) = _W_(`batchSize`)\n- Space complexity: _SC_(`traverser`) where _S_ = _O_(`batchSize`)\n- Billing: _max_(1, _N_) reads, _K_ writes\n\nwhere:\n\n- _N_: number of docs in the traversable\n- _K_: number of docs that passed the migration predicate (_K_\u003c=_N_)\n- _W_(`batchSize`): average batch write time\n- _TC_(`traverser`): time complexity of the underlying traverser\n- _SC_(`traverser`): space complexity of the underlying traverser\n\n## Upgrading\n\nThis project adheres to [SemVer](https://semver.org). Before upgrading to a new major version, make sure to check out the [Releases](https://github.com/kafkas/firewalk/releases) page to view all the breaking changes.\n\n## License\n\nThis project is made available under the MIT License.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkafkas%2Ffirewalk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkafkas%2Ffirewalk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkafkas%2Ffirewalk/lists"}