{"id":17068923,"url":"https://github.com/localnerve/google-drive-folder","last_synced_at":"2025-07-31T00:38:41.304Z","repository":{"id":43044629,"uuid":"354736246","full_name":"localnerve/google-drive-folder","owner":"localnerve","description":"Download a Google Drive Folder","archived":false,"fork":false,"pushed_at":"2025-05-23T11:22:17.000Z","size":1197,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-27T14:06:58.603Z","etag":null,"topics":["build-tool","google-drive","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/localnerve.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-04-05T06:02:51.000Z","updated_at":"2025-05-23T11:21:41.000Z","dependencies_parsed_at":"2023-10-15T02:13:02.222Z","dependency_job_id":"42d7cb72-2dd7-4e1c-903f-b6635266c5e6","html_url":"https://github.com/localnerve/google-drive-folder","commit_stats":{"total_commits":27,"total_committers":1,"mean_commits":27.0,"dds":0.0,"last_synced_commit":"b7668f482cebac1e3171ed6001b0a1902681325c"},"previous_names":[],"tags_count":44,"template":false,"template_full_name":null,"purl":"pkg:github/localnerve/google-drive-folder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localnerve%2Fgoogle-drive-folder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localnerve%2Fgoogle-drive-folder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localnerve%2Fgoogle-drive-folder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localnerve%2Fgoogle-drive-folder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/localnerve","download_url":"https://codeload.github.com/localnerve/google-drive-folder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/localnerve%2Fgoogle-drive-folder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267967720,"owners_count":24173566,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-30T02:00:09.044Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build-tool","google-drive","nodejs"],"created_at":"2024-10-14T11:15:35.292Z","updated_at":"2025-07-31T00:38:41.277Z","avatar_url":"https://github.com/localnerve.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# google-drive-folder\n\n\u003e Streams files from a google drive folder\n\n[![npm version](https://badge.fury.io/js/%40localnerve%2Fgoogle-drive-folder.svg)](https://badge.fury.io/js/%40localnerve%2Fgoogle-drive-folder)\n![Verify](https://github.com/localnerve/google-drive-folder/workflows/Verify/badge.svg)\n[![Coverage Status](https://coveralls.io/repos/github/localnerve/google-drive-folder/badge.svg?branch=master)](https://coveralls.io/github/localnerve/google-drive-folder?branch=master)\n\n## Contents\n+ [Overview](#overview)\n  + [Why](#why-this-exists)\n  + [Authentication and Authorization](#authentication-and-authorization)\n  + [Conversions](#conversions)\n+ [API](#api)\n  + [Input Types](#input)\n  + [Output Stream](#output)\n    + [Stream Data Format](#stream-data-format)\n+ [Input Detail](#input-detail)\n  + [Google Drive Parameters](#google-drive-parameters)\n  + [Options](#options)\n    + [Conversion Options](#conversion-options)\n      + [Transformer Function](#transformer-function)\n        + [Transformer Input Object](#transformer-input-object)\n        + [Transformer Example](#transformer-example)\n+ [Example Usage](#example-usage)\n  + [Example Minimal Usage](#code-example-using-minimal-options)\n  + [Example with Most Options](#code-example-using-most-options)\n+ [MIT License](#license)\n\n## Overview\nThis library streams files from a google drive folder, and applies conversions (if desired).\n\n### Why This Exists\n\nI wrote this library to codify my solution to a painful experience in which [403](https://en.wikipedia.org/wiki/HTTP_403) errors developed during larger folder downloads. This was initially puzzling (403 code not sufficient to understand the precise issue), but I solved it by serially streaming the files. The original problem must have been that I was making too many requests all at once over a very small time period, resulting in a 403 from the Google Drive service by policy.\n\n### Authentication and Authorization\n\nThis library uses the [google api node client](https://github.com/googleapis/google-api-nodejs-client), and therefore requires credentials. By default, this library relies on Environment variable SVC_ACCT_CREDENTIALS to point to a valid Google service account credential file. Further, this credential must have the permissions to impersonate the user email passed in as `userId` for the folder `folderId`.\n\nHowever, If you have different requirements (OAuth2, JWT, etc), you can supply your own pre-resolved auth reference by passing [google auth](https://github.com/googleapis/google-auth-library-nodejs) as an 'auth' option. For more information, all options are [detailed here](#input-detail).\n\n### Conversions\n\nBy default, no conversion occurs, data is just passed through as a Buffer (if binary) or a utf8 encoded string.\nTo change this, supply a `transformer` function and an optional `exportMimeMap` if the source file on Google Drive is a Google Workspace file type (Docs, Spreadsheet, Presentation, Drawing, etc). These options are supplied using the [conversion options](#conversion-options).\n\n## API\n\nThe library exports a single function that returns a Promise that resolves to a Stream.  \nHere's the signature:\n\n```\nPromise\u003cStream\u003e GoogleDriveFolder (folderId, userId, Options)\n```\n\n### Input\n\nA quick look at all the input and the types. All options are optional. For a more detailed explanation of input [skip to here](#input-detail).\n\n```\nfolderId: String,\nuserId: String,\n[Options: Object]\n  outputDirectory: String,\n  scopes: Array\u003cString\u003e,\n  fileQuery: String,\n  auth: GoogleAuth | OAuth2Client | JWT | String, \n  exportMimeMap: Object\u003ckey:String, value:String\u003e,\n  transformer: Function\n```\n\n### Output\n\nThe returned Promise resolves to a `Stream` in object mode to receive data objects for each downloaded file as it arrives.\n\n#### Stream Data Format\n\nThe format of the data objects you will receive on `data` events:\n\n```\n  {\n    input: {\n      data,         // Buffer | String, data from Google Drive, Buffer if binary, 'utf-8' String otherwise\n      name,         // String of the file name\n      ext,          // String of the file extension\n      binary,       // Boolean, true if binary data content\n      downloadMeta, // Object\n        mimeType | alt  // String, mimeType for export, alt for get\n    },\n    // If converted is true, the converted data. Otherwise, a reference to input\n    output: {\n      data,     // String (utf-8) or Buffer if input was binary\n      name,     // String of the file name\n      ext       // String of the file extension\n    },\n    converted   // Boolean, true if conversion occurred, false otherwise\n  }\n```\n\nThe `input` is data as downloaded from Google Drive.\nThe `output` is data as converted by a [transformer](#transformer-function).\nIf no conversion occurs (`converted === false`), output is a referece to `input`.\n\n## Input Detail\n\n`SVC_ACCT_CREDENTIALS` environment variable must point to a valid Google Service Account credential file **UNLESS** Auth is supplied using the `auth` option.\n\n### Google Drive Parameters\n\n* `folderId` {String} -  Uniquely identifies your folder in the google drive service. Found on the web in the url.\n* `userId` {String} - The email address of the folder owner that SVC_ACCT_CREDENTIALS will impersonate. If you supply the `auth` option, this parameter is ignored.\n\n### Options\n\nAll options are optional.\n\n* `[Options]` {Object} - The general options object.\n* `[Options.scopes]` {Array\u003cString\u003e} - Scopes to use for auth (if required) in a special case. Defaults to the `drive.readonly` scope.\n* `[Options.fileQuery]` {String} - A file query string used to filter files to download by specific characteristics. Defaults to downloading all files in the `folderId` that are NOT deleted (`trashed = false`). @see [file reference search terms](https://developers.google.com/drive/api/v3/ref-search-terms).\n* `[Options.auth]` {GoogleAuth | OAuth2Client | JWT | String} - Given directly to the Google Drive NodeJS Client `auth` option. Use this option to override the default behavior of the `SVC_ACCT_CREDENTIALS` environment variable path to a service account credentials. This will also cause the `userId` parameter to be ignored.\n* `[Options.outputDirectory]` {String} - Absolute path to the output directory. Defaults to falsy. If supplied, files are written out as data arrives. Does not touch the directory other than to write files. The directory must already exist.\n\n#### Conversion Options\n\n* `[Options.exportMimeMap]` {Object} - A name-value map of Google Workspace mime-types to a conversion mime-type to be performed by the Google Drive service prior to sending the data to the `transformer` function. If this option is supplied, the Google Drive Files 'export' method is used, and therefore the types are presumed to conform to the service capabilities outlined in the [Google Export Reference](https://developers.google.com/drive/api/v3/ref-export-formats). For detail on Google Workspace mime-types, see [Google Workspace MimeTypes](https://developers.google.com/drive/api/v3/mime-types). If this option is not supplied, the Google Drive Files 'get' method is used for download.\n* `[Options.transformer]` {Function} - Transforms the input after download (or optional export conversion) and before it goes out to the stream (and optional `outputDirectory`, if supplied). Defaults to pass-through. Returns a Promise that resolves to an object that conforms to the [stream data format](#stream-data-format).\n\n##### Transformer Function\n\nA Transformer function receives input from the download and returns a Promise that resolves to the [data stream object format](#data-stream-format)\n\n###### Transformer Input Object\n\nA supplied `transformer` function receives a single object from the download of the following format:\n\n```\n{ \n  name: String,\n  ext: String,\n  data: \u003cBuffer | String\u003e,  // Buffer if binary, 'utf-8' String otherwise\n  binary: Boolean,\n  downloadMeta: Object\n    method: String,         // 'get' or 'export'\n    parameters: Object      // parameters used in the download (mimeType from exportMimeMap or alt='media')\n}\n```\n\n###### Transformer Example\nThis example downloads all files from the given Google Drive Folder.\nIt presumes they are all Google Workspace GoogleDocs files that are markdown documents.\nDownloads them as `text/plain`, converts them to 'html', and outputs the result to the stream.\n\n```js\nimport googleDriveFolder from '@localnerve/google-drive-folder';\n\nprocess.env.SVC_ACCT_CREDENTIALS = '/path/to/svcacctcredential.json';\n\nconst folderId = 'ThEfOlDeRiDyOuSeEiNyOuRbRoWsErOnGoOgLeDrIvE';\nconst userId = 'email-of-the-folder-owner@will-be-impersonated.by-svc-acct';\n\n// Use remark for markdown to html conversion\nconst remark = require('remark');\nconst remarkHtml = require('remark-html');\n\n/**\n * The transformer definition.\n * Receives an input object from the Google Drive download, outputs a conversion object\n * in the form of #stream-data-format defined in this readme.md\n * \n * @param input (Object) The file input object\n * @param input.name {String} The name of the file downloaded\n * @param input.ext {String} The extension of the file download (if available)\n * @param input.data {String} The utf-8 encoded string data from the 'text/plain' download.\n * @param input.binary {Boolean} False in this case, true if content binary.\n * @param input.downloadMeta {Object} download meta data\n * @param input.downloadMeta.method {String} 'export' or 'get', 'export' in this case.\n * @param input.downloadMeta.parameters {Object} the download parameters\n * @param input.downloadMeta.parameters.mimeType {String} The mimeType used, 'text/plain' in this case.\n * @param input.downloadMeta.parameters.alt {String} 'meta' when 'get' method is used, undefined in this case.\n */\nfunction myTransformer (input) {\n  return new Promise((resolve, reject) =\u003e {\n    remark()\n      .use(remarkHtml)\n      .process(input.data, (err, res) =\u003e {\n        if (err) {\n          err.message = `Failed to convert '${input.name}'` + err.message;\n          return reject(err);\n        }\n        resolve({\n          input,\n          output: {\n            name: input.name,\n            ext: '.html',\n            data: String(res)\n          },\n          converted: true\n        });\n      });\n  });\n}\n\n// let's do this:\ntry {\n  // blocks until object flow begins\n  const stream = await googleDriveFolder(folderId, userId, {\n    exportMimeMap: {\n      'application/vnd.google-apps.document': 'text/plain'\n    },\n    transformer: myTransformer \n  });\n  stream.on('data', data =\u003e {\n    // data.input.data has markdown\n    // data.output.data has html\n    console.log(`Received converted data for '${data.output.name}'`, data);\n  });\n  stream.on('end', () =\u003e {\n    console.log('downloads are done, we got all the files.');\n  });\n  stream.on('error', e =\u003e {\n    throw e;\n  });\n}\ncatch (e) {\n  console.error(e); // something went wrong\n}\n```\n\n## Example Usage\n\n### Code Example using minimal options\n\n```js\nimport googleDriveFolder from '@localnerve/google-drive-folder';\n\nprocess.env.SVC_ACCT_CREDENTIALS = '/path/to/svcacctcredential.json';\n\nconst folderId = 'ThEfOlDeRiDyOuSeEiNyOuRbRoWsErOnGoOgLeDrIvE';\nconst userId = 'email-of-the-folder-owner@will-be-impersonated.by-svc-acct';\n\ntry {\n  // Blocks until object flow begins (while auth and file list is downloaded)\n  const stream = await googleDriveFolder(folderId, userId);\n  stream.on('data', data =\u003e {\n    console.log(`Received a data object for '${data.input.name}'`, data);\n  });\n  stream.on('end', () =\u003e {\n    console.log('downloads are done, we got all the files');\n  });\n  stream.on('error', e =\u003e {\n    throw e;\n  });\n} catch (e) {\n  console.error(e); // something went wrong\n}\n```\n\n### Code Example using most options\n\nAll the prerequisites and possible arguments as code (except for `auth` option):\n\n```js\nimport googleDriveFolder from '@localnerve/google-drive-folder';\nimport myTransformer from './myTransformer'; // your transform function\n\nprocess.env.SVC_ACCT_CREDENTIALS = '/path/to/svcacctcredential.json';\n\nconst folderId = 'ThEfOlDeRiDyOuSeEiNyOuRbRoWsErOnGoOgLeDrIvE';\nconst userId = 'email-of-the-folder-owner@will-be-impersonated.by-svc-acct';\n\n// all optional, if outputDirectory omitted, returns ReadableStream\nconst options = {\n  outputDirectory: '/tmp/mydrivefolder/mustexist',\n  scopes: [\n    'special/google.auth/scope/you/might/need/other/than/drive.readonly',\n    'https://www.googleapis.com/auth/drive.readonly'\n  ],\n  fileQuery: 'name contains \".md\"', // download GoogleDocs markdown files\n  exportMimeMap: {\n    'application/vnd.google-apps.document': 'text/plain'\n  },\n  transformer: myTransformer\n};\n\ntry {\n  // Blocks until object flow begins\n  const stream = await googleDriveFolder(folderId, userId, options);\n  stream.on('data', data =\u003e {\n    console.log(`Received a data object for '${data.input.name}'`, data);\n  });\n  stream.on('end', () =\u003e {\n    console.log('downloads are done, we got all the files');\n  });\n  stream.on('error', e =\u003e {\n    throw e;\n  });\n} catch (e) {\n  console.error(e); // something went wrong\n}\n\n```\n\n## LICENSE\n\n* [MIT, Alex Grant, LocalNerve, LLC](license.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalnerve%2Fgoogle-drive-folder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flocalnerve%2Fgoogle-drive-folder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flocalnerve%2Fgoogle-drive-folder/lists"}