{"id":13554854,"url":"https://github.com/IBM/couchbackup","last_synced_at":"2025-04-03T07:33:07.670Z","repository":{"id":21648344,"uuid":"87912220","full_name":"IBM/couchbackup","owner":"IBM","description":"Cloudant backup and restore library and command-line utility","archived":false,"fork":false,"pushed_at":"2024-10-29T09:18:39.000Z","size":1865,"stargazers_count":76,"open_issues_count":5,"forks_count":21,"subscribers_count":66,"default_branch":"main","last_synced_at":"2024-10-29T11:37:16.829Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IBM.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-11T08:55:36.000Z","updated_at":"2024-10-29T09:18:41.000Z","dependencies_parsed_at":"2023-10-04T21:31:16.389Z","dependency_job_id":"519077e9-9094-4e18-a515-00b61d02ee02","html_url":"https://github.com/IBM/couchbackup","commit_stats":{"total_commits":769,"total_committers":23,"mean_commits":33.43478260869565,"dds":0.6814044213263979,"last_synced_commit":"96607176303393ae5c8125977c00138a4f187ae8"},"previous_names":["ibm/couchbackup","cloudant/couchbackup"],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fcouchbackup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fcouchbackup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fcouchbackup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fcouchbackup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IBM","download_url":"https://codeload.github.com/IBM/couchbackup/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222935627,"owners_count":17060398,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T12:02:56.261Z","updated_at":"2024-11-04T03:31:24.011Z","avatar_url":"https://github.com/IBM.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","others"],"sub_categories":[],"readme":"# CouchBackup\n\n[![npm (scoped)](https://img.shields.io/npm/v/@cloudant/couchbackup.svg?colorB=0000ff)](https://www.npmjs.com/package/@cloudant/couchbackup)\n[![npm (scoped with tag)](https://img.shields.io/npm/v/@cloudant/couchbackup/snapshot.svg?colorB=666699)](https://www.npmjs.com/package/@cloudant/couchbackup)\n\n```\n _____                  _    ______            _\n/  __ \\                | |   | ___ \\          | |\n| /  \\/ ___  _   _  ___| |__ | |_/ / __ _  ___| | ___   _ _ __\n| |    / _ \\| | | |/ __| '_ \\| ___ \\/ _` |/ __| |/ / | | | '_ \\\n| \\__/\\ (_) | |_| | (__| | | | |_/ / (_| | (__|   \u003c| |_| | |_) |\n \\____/\\___/ \\__,_|\\___|_| |_\\____/ \\__,_|\\___|_|\\_\\\\__,_| .__/\n                                                         | |\n                                                         |_|\n```\n\nCouchBackup is a command-line utility that backs up a Cloudant or CouchDB database to a text file.\nIt comes with a companion command-line utility that can restore the backed up data.\n\n## Limitations\n\nCouchBackup has some restrictions in the data it's able to backup:\n\n* **`couchbackup` does not do CouchDB replication as such, it simply streams through a database's `_changes` feed, and uses `POST /db/_bulk_get` to fetch the documents, storing the documents it finds on disk.**\n* **`couchbackup` does not support backing up or restoring databases containing documents with attachments. The recommendation is to store attachments directly in an object store. The \"attachments\" option is provided as-is and is not supported. This option is for Apache CouchDB only and is experimental. DO NOT USE THIS OPTION WITH IBM Cloudant backups.** [Note](#note-on-attachments)\n\n## Installation\n\nTo install the latest released version use npm:\n\n```sh\nnpm install -g @cloudant/couchbackup\n```\n\n### Requirements\n* Node.js LTS version 18, 20, or 22.\n* The minimum required CouchDB version is 2.0.0.\n\n### Snapshots\n\nThe latest builds of the `main` branch are available on npm with the `snapshot` tag. Use the `snapshot` tag if you want to experiment with an unreleased fix or new function, but please note that snapshot versions are **not supported**.\n\n## Usage\n\nUse either environment variables or command-line options to specify the URL of the CouchDB or Cloudant instance, and the database to work with.\n\n### The URL\n\nTo define the URL of the CouchDB instance set the `COUCH_URL` environment variable:\n\n```sh\nexport COUCH_URL=http://localhost:5984\n```\nor\n\n```sh\nexport COUCH_URL=https://myusername:mypassword@myhost.cloudant.com\n```\n\nOr use the `--url` command-line parameter.\n\nWhen passing credentials in the user information subcomponent of the URL\nthey must be [percent encoded](https://tools.ietf.org/html/rfc3986#section-3.2.1).\nSpecifically, within either the username or password, the characters `: / ? # [ ] @ %`\n_MUST_ be precent-encoded, other characters _MAY_ be percent-encoded.\n\nFor example, for the username `user123` and password `colon:at@321`:\n```\nhttps://user123:colon%3aat%40321@localhost:5984\n```\n\nNote take extra care to escape shell reserved characters when\nsetting the environment variable or command-line parameter.\n\n### The Database name\n\nTo define the name of the database to backup or restore, set the `COUCH_DATABASE` environment variable:\n\n```sh\nexport COUCH_DATABASE=animaldb\n```\n\nOr use the `--db` command-line parameter\n\n## Backup\n\nTo backup a database to a text file, use the `couchbackup` command, directing the output to a text file:\n\n```sh\ncouchbackup \u003e backup.txt\n```\n\nAnother way of backing up is to set the `COUCH_URL` environment variable only and supply the database name on the command-line:\n\n```sh\ncouchbackup --db animaldb \u003e animaldb.txt\n```\n\n## Logging \u0026 resuming backups\n\nYou may also create a log file which records the progress of the backup with the `--log` parameter, for example:\n\n```sh\ncouchbackup --db animaldb --log animaldb.log \u003e animaldb.txt\n```\n\nUse this log file to resume backups with `--resume true`:\n\n```sh\ncouchbackup --db animaldb --log animaldb.log --resume true \u003e\u003e animaldb.txt\n```\n\nThe `--resume true` option works for a backup that has finished spooling changes, but has not yet completed downloading all the necessary batches of documents. It _is not an incremental backup_ solution.\n\nYou may also specify the name of the output file, rather than directing the backup data to *stdout*:\n\n```sh\ncouchbackup --db animaldb --log animaldb.log --resume true --output animaldb.txt\n```\n\n### Compatibility note\n\nWhen using `--resume` use the same version of `couchbackup` that started the backup.\n\n## Restore\n\nNow restore the backup text file to a new, empty, existing database using the `couchrestore`:\n\n```sh\ncat animaldb.txt | couchrestore\n```\n\nor specifying the database name on the command-line:\n\n```sh\ncat animaldb.txt | couchrestore --db animaldb2\n```\n\n### Compatibility note\n\n**Do not use an older version of `couchbackup` to restore a backup created with a newer version.**\n\nNewer versions of `couchbackup` can restore backups created by older versions within the same major version.\n\n## Compressed backups\n\nTo compress the backup data before storing to disk pipe the contents through `gzip`:\n\n```sh\ncouchbackup --db animaldb | gzip \u003e animaldb.txt.gz\n```\n\nand restore the file with:\n\n```sh\ncat animaldb.tar.gz | gunzip | couchdbrestore --db animaldb2\n```\n\n## Encrypted backups\n\nSimilarly to compression it is possible to pipe the backup content through an\nencryption or decryption utility. For example with `openssl`:\n\n```sh\ncouchbackup --db animaldb | openssl aes-128-cbc -pass pass:12345 \u003e encrypted_animal.db\n```\n\n```sh\nopenssl aes-128-cbc -d -in encrypted_animal.db -pass pass:12345 | couchrestore --db animaldb2\n```\n\nNote that the content is not encrypted in the\nbackup tool before piping to the encryption utility.\n\n## What's in a backup file?\n\nA backup file is a text file where each line is either a JSON object of backup metadata\nor a JSON array of backed up document revision objects, for example:\n\n```json\n{\"name\":\"@cloudant/couchbackup\",\"version\":\"2.9.10\",\"mode\":\"full\"}\n[{\"_id\": \"1\",\"a\":1},{\"_id\": \"2\",\"a\":2},...]\n[{\"_id\": \"501\",\"a\":501},{\"_id\": \"502\",\"a\":502}]\n```\n\nThe number of document revisions in a backup array varies. It typically has\n`buffer_size` elements, but may be more if there are also leaf revisions returned\nfrom the server or fewer if it is the last batch.\n\n## What's in a log file?\n\nA log file has a line:\n\n- for every batch of document ids that `couchbackup` needs to fetch, for example: `:t batch56 [{\"id\":\"a\"},{\"id\":\"b\"}]`\n- for every batch that `couchbackup` has fetched and stored, for example: `:d batch56`\n- to indicate that the changes feed was fully consumed, for example: `:changes_complete`\n\n## What's shallow mode?\n\nWhen you run `couchbackup` with `--mode shallow` `couchbackup` performs a simpler backup.\nIt only backs up the winning revisions and ignores any conflicting revisions.\nThis is a faster, but less complete backup.\n\n_Note:_ The `--log`,  `--resume`, and `--parallelism` are invalid for `--mode shallow` backups.\n\n## Why use CouchBackup?\n\nThe easiest way to backup a CouchDB database is to copy the \".couch\" file. This is fine on a single-node instance, but when running multi-node\nCloudant or using CouchDB 2.0 or greater, the \".couch\" file only has a single shard of data. This utility allows simple backups of CouchDB\nor Cloudant database using the HTTP API.\n\nThis tool can script the backup of your databases. Move the backup and log files to cheap Object Storage so that you have copies of your precious data.\n\n## Options reference\n\n### Environment variables\n\n* `COUCH_URL` - the URL of the CouchDB/Cloudant server, for example: `http://127.0.0.1:5984`\n* `COUCH_DATABASE` - the name of the database to act upon, for example: `mydb` (default `test`)\n* `COUCH_PARALLELISM` - the number of HTTP requests to perform in parallel when restoring a backup, for example: `10` (Default `5`)\n* `COUCH_BUFFER_SIZE` - the number of documents fetched and restored at once, for example: `100` (default `500`).\n* `COUCH_REQUEST_TIMEOUT` - the number of milliseconds to wait for a response to a HTTP request before retrying the request, for example: `10000` (Default `120000`)\n* `COUCH_LOG` - the file to store logging information during backup\n* `COUCH_RESUME` - if `true`, resumes an earlier backup from its last known position (requires a log file)\n* `COUCH_OUTPUT` - the file name to store the backup data (defaults to stdout)\n* `COUCH_MODE` - if `shallow`, does only a superficial backup ignoring conflicts. Defaults to `full` - a full backup.\n* `COUCH_QUIET` - if `true`, suppresses the individual batch messages to the console during CLI backup and restore\n* `CLOUDANT_IAM_API_KEY` - optional [IAM API key](https://console.bluemix.net/docs/services/Cloudant/guides/iam.html#ibm-cloud-identity-and-access-management)\n to use to access the Cloudant database instead of user information credentials in the URL. The endpoint used to retrieve the token defaults to\n `https://iam.cloud.ibm.com/identity/token`, but can be overridden if necessary using the `CLOUDANT_IAM_TOKEN_URL` environment variable.\n* `COUCH_ATTACHMENTS` - _EXPERIMENTAL \u0026 UNSUPPORTED_ (see [Note](#note-on-attachments)) if `true` will include attachments as part of the backup or restore process.\n* `DEBUG` - if set to `couchbackup`, all debug messages print on `stderr` during a backup or restore process\n\n_Note:_ Environment variables are only used with the CLI. When\n[using programmatically](#using-programmatically) use the `opts` dictionary.\n\n### Command-line parameters\n\n* `--url` - same as `COUCH_URL` environment variable\n* `--db` - same as `COUCH_DATABASE`\n* `--parallelism` - same as `COUCH_PARALLELISM`\n* `--buffer-size` - same as `COUCH_BUFFER_SIZE`\n* `--request-timeout` - same as `COUCH_REQUEST_TIMEOUT`\n* `--log` - same as `COUCH_LOG`\n* `--resume` - same as `COUCH_RESUME`\n* `--output` - same as `COUCH_OUTPUT`\n* `--mode` - same as `COUCH_MODE`\n* `--iam-api-key` - same as `CLOUDANT_IAM_API_KEY`\n* `--quiet` - same as `COUCH_QUIET`\n* `--attachments` - _EXPERIMENTAL \u0026 UNSUPPORTED_ (see [Note](#note-on-attachments)) same as `COUCH_ATTACHMENTS`\n\n## Using programmatically\n\nYou can use `couchbackup` programmatically. First install\n`couchbackup` into your project with `npm install --save @cloudant/couchbackup`.\nThen you can import the library into your code:\n\n```js\n  const couchbackup = require('@cloudant/couchbackup');\n```\n\nThe library exports two main functions:\n\n1. `backup` - backup from a database to a writable stream.\n2. `restore` - restore from a readable stream to an empty database.\n\n### Examples\n\nSee [the examples folder](./examples) for example scripts showing how to\nuse the library.\n\n### Backup\n\nThe `backup` function takes a source database URL, a stream to write to,\nbackup options and a callback for completion.\n\n```javascript\nbackup: function(srcUrl, targetStream, opts, callback) { /* ... */ }\n```\n\nThe `opts` dictionary can contain values which map to a subset of the\nenvironment variables defined above. Those related to the source and\ntarget locations are not required.\n\n* `parallelism`: see `COUCH_PARALLELISM`.\n* `bufferSize`: see `COUCH_BUFFER_SIZE`.\n* `requestTimeout`: see `COUCH_REQUEST_TIMEOUT`.\n* `log`: see `COUCH_LOG`.\n* `resume`: see `COUCH_RESUME`.\n* `mode`: see `COUCH_MODE`.\n* `iamApiKey`: see `CLOUDANT_IAM_API_KEY`.\n* `iamTokenUrl`: optionally used with `iamApiKey` to override the default URL for\n retrieving IAM tokens.\n* `attachments`: _EXPERIMENTAL \u0026 UNSUPPORTED_ (see [Note](#note-on-attachments)), see `CLOUDANT_ATTACHMENTS`.\n\nWhen the backup completes or fails the callback functions gets called with\nthe standard `err, data` parameters.\n\nThe `backup` function returns an event emitter. You can subscribe to:\n\n* `changes` - when a batch of changes has been written to log stream.\n* `written` - when a batch of documents has been written to backup stream.\n* `finished` - emitted once when all documents are backed up.\n\nBackup data to a stream:\n\n```javascript\ncouchbackup.backup(\n  'https://examples.cloudant.com/animaldb',\n  process.stdout,\n  {parallelism: 2},\n  function(err, data) {\n    if (err) {\n      console.error(\"Failed! \" + err);\n    } else {\n      console.error(\"Success! \" + data);\n    }\n  });\n```\n\nOr to a file:\n\n```javascript\ncouchbackup.backup(\n  'https://examples.cloudant.com/animaldb',\n  fs.createWriteStream(filename),\n  {parallelism: 2},\n  function(err, data) {\n    if (err) {\n      console.error(\"Failed! \" + err);\n    } else {\n      console.error(\"Success! \" + data);\n    }\n  });\n```\n\n### Restore\n\nThe `restore` function takes a readable stream containing the data emitted\nby the `backup` function and uploads that to a Cloudant database.\n\n_Note:_ A target database must be a **new and empty** database.\n\n```javascript\nrestore: function(srcStream, targetUrl, opts, callback) { /* ... */ }\n```\n\nThe `opts` dictionary can contain values which map to a subset of the\nenvironment variables defined above. Those related to the source and\ntarget locations are not required.\n\n* `parallelism`: see `COUCH_PARALLELISM`.\n* `bufferSize`: see `COUCH_BUFFER_SIZE`.\n* `requestTimeout`: see `COUCH_REQUEST_TIMEOUT`.\n* `iamApiKey`: see `CLOUDANT_IAM_API_KEY`.\n* `iamTokenUrl`: optionally used with `iamApiKey` to override the default URL for\n retrieving IAM tokens.\n* `attachments`: _EXPERIMENTAL \u0026 UNSUPPORTED_ (see [Note](#note-on-attachments)), see `CLOUDANT_ATTACHMENTS`.\n\nWhen the restore completes or fails the callback functions gets called with\nthe standard `err, data` parameters.\n\nThe `restore` function returns an event emitter. You can subscribe to:\n\n* `restored` - when a batch of documents is restored.\n* `finished` - emitted once when all documents are restored.\n\nThe `srcStream` for the restore is a [backup file](#whats-in-a-backup-file).\nIn the case of an incomplete backup the file could be corrupt and in that\ncase the restore emits a `BackupFileJsonError`.\n\nRestore data from a stream:\n\n```javascript\ncouchbackup.restore(\n  process.stdin,\n  'https://examples.cloudant.com/new-animaldb',\n  {parallelism: 2},\n  function(err, data) {\n    if (err) {\n      console.error(\"Failed! \" + err);\n    } else {\n      console.error(\"Success! \" + data);\n    }\n  });\n```\n\nOr from a file:\n\n```javascript\ncouchbackup.restore(\n  fs.createReadStream(filename),\n  'https://examples.cloudant.com/new-animaldb',\n  {parallelism: 2},\n  function(err, data) {\n    if (err) {\n      console.error(\"Failed! \" + err);\n    } else {\n      console.error(\"Success! \" + data);\n    }\n  });\n```\n\n## Error Handling\n\nThe `couchbackup` and `couchrestore` processes are able to tolerate many errors even over an unreliable network.\nFailed requests retry at least twice after a back-off delay.\nHowever, certain errors can't tolerate failures:\n- invalid configuration\n- failed validation checks (for example: auth, database existence, `_bulk_get` endpoint avaialbility)\n\n### API\n\nWhen using the library programmatically in the case of a fatal error\nthe callback function gets called with `null, error` arguments.\n\n### CLI Exit Codes\n\nOn fatal errors, `couchbackup` and `couchrestore` exit with non-zero exit codes. This section\ndetails them.\n\n### common to both `couchbackup` and `couchrestore`\n\n* `1`: unknown CLI option or generic error.\n* `2`: invalid CLI option.\n* `10`: backup source or restore target database does not exist.\n* `11`: unauthorized credentials for the database.\n* `12`: invalid permissions for the database.\n* `40`: database returned a fatal HTTP error.\n\n### `couchbackup`\n\n* `20`: `--resume` without a log file.\n* `21`: the resume log file does not exist.\n* `22`: incomplete changes in log file.\n* `23`: the log file already exists, but `--resume` was not used.\n* `30`: error spooling changes from the database.\n* `50`: source database does not support `/_bulk_get` endpoint.\n\n### `couchrestore`\n\n* `13`: restore target database is not new and empty.\n* `60`: `attachments` option used for backup, but wasn't used for restore.\n* `61`: `attachments` option used for restore, but wasn't used for backup.\n\n## Note on attachments\n\nTLDR; If you backup a database that has attachments without using the `attachments` option `couchbackup` can't restore it.\n\nAs documented above `couchbackup` does not support backing up or restoring databases containing documents with attachments.\n\nThe recommendation is to store attachments directly in an object store with a link in the JSON document instead of using the\nnative attachment API.\n\n### With experimental `attachments` option\n\nThe `attachments` option is provided as-is and is not supported. This option is for Apache CouchDB only and is experimental. Do not use this option with IBM Cloudant backups.\n\n### Without experimental `attachments` option\n\nBacking up a database that includes documents with attachments appears to complete successfully. However, the attachment\ncontent is not downloaded and the backup file contains attachment metadata. So attempts to\nrestore the backup result in errors because the attachment metadata references attachments that are not present\nin the restored database.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIBM%2Fcouchbackup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FIBM%2Fcouchbackup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIBM%2Fcouchbackup/lists"}