{"id":13998488,"url":"https://github.com/TimMikeladze/gist-database","last_synced_at":"2025-07-23T06:31:50.670Z","repository":{"id":65526121,"uuid":"584034634","full_name":"TimMikeladze/gist-database","owner":"TimMikeladze","description":"✨ Transform gist into your personal key/value datastore. 💡Pair this with Next.js static sites and incremental static regeneration. Built with TypeScript.","archived":false,"fork":false,"pushed_at":"2023-06-03T03:35:29.000Z","size":367,"stargazers_count":108,"open_issues_count":4,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-26T09:35:23.185Z","etag":null,"topics":["datastore","gist","gist-data","gist-database","gist-db","github-database","key-value","keyvalue"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TimMikeladze.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null},"funding":{"ko_fi":"linesofcodedev","custom":["https://www.paypal.me/TimMikeladze"]}},"created_at":"2023-01-01T02:34:59.000Z","updated_at":"2024-10-08T11:31:05.000Z","dependencies_parsed_at":"2024-01-15T19:44:48.113Z","dependency_job_id":"f5a132da-494b-41b3-b6e9-0d4651a57bb0","html_url":"https://github.com/TimMikeladze/gist-database","commit_stats":{"total_commits":59,"total_committers":2,"mean_commits":29.5,"dds":0.05084745762711862,"last_synced_commit":"103476a1948920fc91e6623ef89e60213ad1f1bf"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimMikeladze%2Fgist-database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimMikeladze%2Fgist-database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimMikeladze%2Fgist-database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TimMikeladze%2Fgist-database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TimMikeladze","download_url":"https://codeload.github.com/TimMikeladze/gist-database/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227245180,"owners_count":17753239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datastore","gist","gist-data","gist-database","gist-db","github-database","key-value","keyvalue"],"created_at":"2024-08-09T19:01:42.904Z","updated_at":"2024-11-30T00:31:38.784Z","avatar_url":"https://github.com/TimMikeladze.png","language":"TypeScript","funding_links":["https://ko-fi.com/linesofcodedev","https://www.paypal.me/TimMikeladze"],"categories":["TypeScript"],"sub_categories":[],"readme":"# 🗄️ Gist Database\n\n✨ Transform [gist](https://gist.github.com/) into your personal key/value data store.\n\n```console\nnpm install gist-database\n\nyarn add gist-database\n\npnpm add gist-database\n```\n\n## 🚪 Introduction\n\nSometimes all a project needs is the ability to read/write small amounts of JSON data and have it saved in some persistent storage. Imagine a simple data-model which receives infrequent updates and could be represented as JSON object. It doesn't demand a full-blown database, but it would be neat to have a way to interact with this data and have it persist across sessions.\n\nThis is where `gist-database` comes in handy, by leveraging the power of the [gist api](https://gist.github.com/) you can easily create a key/value data-store for your project.\n\nThis is a perfect solution for low write / high read scenarios when serving static site content with [Next.js](https://nextjs.org/) and using [Incremental Static Regeneration](https://nextjs.org/docs/basic-features/data-fetching/incremental-static-regeneration) to keep your cached content fresh.\n\n\u003e 👋 Hello there! Follow me [@linesofcode](https://twitter.com/linesofcode) or visit [linesofcode.dev](https://linesofcode.dev) for more cool projects like this one.\n\n## ⚖️‍ Acceptable use policy\n\nWhen using this library you **must comply** with Github's [acceptable use policy](https://docs.github.com/en/github/site-policy/github-acceptable-use-policies). Do not use this library to store data that violates Github's guidelines, violates laws, is malicious, unethical, or harmful to others.\n\n## 🏃 Getting started\n\nIn order to communicate with the Gist API you need to create a [personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) with the `gist` scope or use the [beta tokens](https://github.com/settings/tokens?type=beta) with the `gist read/write scope`.\n\nSave this token somewhere safe, you will need it to authenticate with the Gist API.\n\nNow let's create a new database. The empty database will be created as single gist containing a single file called `database.json` with an empty JSON object: `{}`.\n\nThis package comes with a cli tool to help you perform common database operations.\n\n```console\nUsage: gist-database [options]\n\nTransform gist into a key/value datastore.\n\nOptions:\n  -c --create                      Create a new gist database.\n  -p --public                      Make the gist public. (default: false)\n  -de --description \u003cdescription\u003e  Description of the gist. (default: \"\")\n  -des --destroy \u003cdestroy\u003e         Destroy a gist database. Provide the gist id of the database.\n  -t --token \u003ctoken\u003e               Gist token. Required for all operations.\n  -h, --help                       display help for command\n```\n\nTo create a new database run the following command in your terminal:\n\n```console\nnpx gist-database -c -t \u003cyour-token\u003e\n```\n\nIf successful, you should see output similar to:\n\n```json\n{\n  \"id\": \"xxxxxxxxxxxx\",\n  \"rawUrl\": \"https://api.github.com/gists/xxxxxxxxxxxx\",\n  \"url\": \"https://gist.github.com/xxxxxxxxxxxx\",\n  \"public\": false,\n  \"description\": \"\"\n}\n```\n\nThis is the gist containing your main database file. Save the `id` somewhere safe. You will need it to initialize your database instance.\n\n## 📖 API\n\n```ts\nimport { GistDatabase, CompressionType } from 'gist-database'\n\n// Initialize the database\n\nconst db = new GistDatabase({\n  token: process.env.GIST_TOKEN,\n  id: process.env.GIST_ID,\n  encryptionKey: process.env.GIST_ENCRYPTION_KEY, // Optional - Encrypt your data\n  compression: CompressionType.pretty // Optional - Compress your data\n})\n\n// Before we begin let's define an optional Tyescript interface to add some type-safety to the shape of our data. Tip: combine this with Zod for even more safety around your data and business logic.\n\ninterface ExampleData {\n  hello: string\n  foo?: string\n}\n\nconst original = await db.set\u003cExampleData\u003e('key', {\n  value: {\n    hello: 'world'\n  }\n})\n\nconst found = await db.get\u003cExampleData\u003e('key')\n\n/**\n {\n  value : {\n      hello: \"world\"\n  },\n  id: \"xxxxxxxxxxxxxxxxxxx\",\n  url: \"https://api.github.com/gists/xxxxxxxxxxx\",\n  rev: \"xxxxx\"\n}\n **/\n\nconst updated = await db.set\u003cExampleData\u003e('key', {\n  value: {\n    hello: 'world',\n    foo: 'bar'\n  }\n})\n\n/**\n {\n  value : {\n      hello: \"world\"\n      foo: \"bar\"\n  },\n  id: \"xxxxxxxxxxxxxxxxxxx\",\n  url: \"https://api.github.com/gists/xxxxxxxxxxx\"\n  rev: \"yyyyy\",\n}\n **/\n\n// A rev can be used to ensure that the data is not overwritten by another process. If the rev does not match the current rev, the update will fail.\ntry {\n  await updated.set\u003cExampleData\u003e('key', {\n    value: {\n      hello: 'world',\n      foo: 'bar'\n    },\n    rev: original.rev // this will throw an error\n    // rev: Database.rev() // leave field blank or manually generate a new rev\n  })\n} catch (err) {\n  // An error will be thrown due to the rev mismatch\n  console.log(err)\n}\n\n// Trying to fetch an outdated rev will also throw an error\ntry {\n  await updated.get\u003cExampleData\u003e('key', {\n    rev: original.rev // this will throw an error\n    // rev: updated.rev // this will succeed\n  })\n} catch (err) {\n  // An error will be thrown due to the rev mismatch\n  console.log(err)\n}\n\n// It's possible to pass arrays as key names. This is especially useful if you want to scope your data to a specific user or group. Internally all array keys will be joined with a `.` character.\n\nawait db.set\u003cExampleData\u003e(['user', 'testUserId'], {\n  value: {\n    displayName: 'Test User'\n  }\n})\n\nawait db.get\u003cExampleData\u003e(['user', 'testUserId'])\n\nawait db.has('key') // true\n\nawait db.keys() // ['key']\n\nawait db.delete('key') // void\n\nawait db.set\u003cExampleData\u003e('key_with_ttl', {\n  ttl: 1000, // 1 second\n  description: \"I'll expire soon and be deleted upon retrieval\"\n})\n\n// Get or delete many keys at once. `undefined` will be returned for keys that don't exist.\nawait db.getMany(['key1', 'key2', 'key3'])\n\nawait db.deleteMany(['key1', 'key2', 'key3'])\n\n// Remove all gist files and delete the database\nawait db.destroy()\n```\n\n## 🏗️ How it works\n\nThe gist of it: each database is stored as multiple `.json` files with one or more of these files maintaining additional metadata about the database.\n\nThe main file is called `database.json` (this is the file corresponding to the id you provided during initialization). It serves multiple purposes, but is primarily used as a lookup table for gistIds with a specific key. It also contains additional metadata such as associating TTL values with keys. Take care when editing or removing this file as it is the source of truth for your database.\n\nWhen a value is created or updated a new `.json` gist is created for the document. It contains the provided value plus additional metadata such as TTL. The id of this newly created gist is then added to the lookup table in `database.json`.\n\nEach gist can contain up to 10 files, with each file having a maximum size of 1mb.\n\nWhen data is written or read for a specific key, this library will chunk the data and pack it into multiple files within the gist to optimize storage.\n\n## 📄 Storing markdown files\n\nGists lend themselves perfectly to storing markdown files which you can then revise over time. This is a great way to keep track of your notes, ideas, or even use Gist as a headless CMS to manage your blog posts.\n\n\u003e **❗Important note:** These files will be stored as is without any **compression** or **encryption** and there is no additional guarding around inconsistent writes using revision ids when writing to the gist containing these files.\n\nOut of the box this library supports storing additional files beyond the `value` argument passed to `set`. They will be stored as a separate gist file and be part of the response when calling `get` or `set`.\n\nThis is useful for storing `.md` files or other assets like `.yml` files alongside some data while circumventing the packing, compression and encryption that is typically applied to the `value` argument.\n\n```ts\nconst blogId = 'xxxxxx'\n\nawait db.set(`blog_${blogId}`, {\n  value: {\n    tags: ['javascript', 'typescript', 'gist-database'],\n    title: 'My blog post'\n  },\n  files: [\n    {\n      name: `blog_${id}.md`,\n      content: `# My blog post\nGist Database is pretty cool.`\n    }\n  ]\n})\n\nawait db.get(`blog_${blogId}`)\n\n/**\n {\n  value : {\n    tags: ['javascript', 'typescript', 'gist-database'],\n    title: 'My blog post',\n  },\n  id: \"xxxxxxxxxxxxxxxxxxx\",\n  url: \"https://api.github.com/gists/xxxxxxxxxxx\"\n  rev: \"xxxxx\",\n  files: [\n    {\n      name: 'blog_${id}.md',\n      content: `# My blog post\nGist Database is pretty cool.`\n    }\n  ]\n}\n **/\n```\n\n## 🗜️ Compression\n\nWhen initializing `GistDatabase` you can pass an optional parameter `compression` to control how data is serialized and deserialized. By default, the data is not compressed at all and is stored as plain JSON.\n\n**Available compression options:**\n\n- `none` - no compression\n- `msgpck` - [msgpack](https://msgpack.org/) compression using [msgpackr](https://www.npmjs.com/package/msgpackr)\n- `pretty` - Store data as well-formatted JSON, this is useful for debugging purposes or databases where the content needs to be easily human-readable.\n\n## 🔐 Encryption\n\nWhen initializing `GistDatabase` you can pass an optional parameter called `encryptionKey` to enable `aes-256-gcm` encryption and decryption using the [cryptr](https://github.com/MauriceButler/cryptr) package.\n\n```ts\nconst db = new GistDatabase({\n  token: process.env.GIST_TOKEN,\n  id: process.env.GIST_ID,\n  encryptionKey: process.env.GIST_ENCRYPTION_KEY\n})\n```\n\n## 🧮 Revisions\n\nEach time a value is set, a new `rev` id is generated using the [nanoid](https://github.com/ai/nanoid) package. This revision is used to ensure that the data is not overwritten by another process. Before data is written the document for the corresponding key will be fetched its revision id checked with one provided. If they do not match the update will fail and an error will be thrown.\n\nBy default, revisions are not checked when getting or setting data. To enable revision checking, pass the `rev` parameter to `get` or `set`. Typically, this would be the `rev` value returned from the previous `get` or `set` call for the same key.\n\nThis is a dirty implementation of optimistic locking. It is not a perfect solution, but it is a simple way of **trying** to keep data consistent during concurrent writes. If you're looking for consistency guarantees then you should use a proper database solution, not this library.\n\n## ⚠️ Limitations\n\n1. This is **not** a replacement for a **production database!** Do not store data that you cannot afford to lose or that needs to remain consistent. If it's important, use the proper database solution for your problem.\n1. This is not intended for **high write** scenarios. You will be rate limited by the GitHub API. This is package is intended for **low write**, **single session** scenarios.\n1. The maximum size that a value can be is approximately 10mb. However, I suspect a request that large would simply be rejected by the API. It's not a scenario I'm building for as sophisticated storage is beyond the scope of this library. Once again this is not a real database, it should not be used for storing large documents.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTimMikeladze%2Fgist-database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTimMikeladze%2Fgist-database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTimMikeladze%2Fgist-database/lists"}