{"id":20697880,"url":"https://github.com/fly-apps/litefs-js","last_synced_at":"2025-04-05T20:09:37.392Z","repository":{"id":65597690,"uuid":"594818647","full_name":"fly-apps/litefs-js","owner":"fly-apps","description":"JavaScript utilities for working with LiteFS on Fly.io","archived":false,"fork":false,"pushed_at":"2024-02-13T16:52:04.000Z","size":76,"stargazers_count":161,"open_issues_count":3,"forks_count":4,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-29T19:05:47.223Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://fly.io/blog/introducing-litefs","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fly-apps.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-29T18:26:59.000Z","updated_at":"2025-03-11T07:23:39.000Z","dependencies_parsed_at":"2024-02-13T18:13:01.760Z","dependency_job_id":"655857e8-676a-42c3-8cd3-782b5a55dac5","html_url":"https://github.com/fly-apps/litefs-js","commit_stats":{"total_commits":14,"total_committers":2,"mean_commits":7.0,"dds":0.0714285714285714,"last_synced_commit":"bf8369a8fae5cb17a1539c36e34308965c131782"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fly-apps%2Flitefs-js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fly-apps%2Flitefs-js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fly-apps%2Flitefs-js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fly-apps%2Flitefs-js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fly-apps","download_url":"https://codeload.github.com/fly-apps/litefs-js/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393573,"owners_count":20931813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-17T00:20:24.554Z","updated_at":"2025-04-05T20:09:37.373Z","avatar_url":"https://github.com/fly-apps.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003elitefs-js 🎈\u003c/h1\u003e\n\n\u003cp\u003e\nJavaScript utilities for working with\n\u003ca href=\"https://fly.io/docs/litefs/getting-started/\"\u003eLiteFS\u003c/a\u003e\non \u003ca href=\"https://fly.io\"\u003eFly.io\u003c/a\u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n---\n\n\u003c!-- prettier-ignore-start --\u003e\n[![Build Status][build-badge]][build]\n[![version][version-badge]][package]\n[![MIT License][license-badge]][license]\n\u003c!-- prettier-ignore-end --\u003e\n\n## The problem\n\nDeploying your app to multiple regions along with your data is a great way to\nmake your app really fast, but there are two issues:\n\n1. Read replica instances can only read from the database, they cannot write to\n   it.\n2. There's an edge case where the user could write to the primary instance and\n   then read from a replica instance before replication is finished.\n\nThe first problem is as simple as making sure you use a special `fly-replay`\nresponse so Fly can pass the request to the primary instance:\n\n![a visualization of the user making a request which is sent to a read replica and replayed to the primary instance](https://user-images.githubusercontent.com/1500684/215623618-85620188-b7f7-458b-90cf-d1844b3d6d63.png)\n\nBut the second problem is a little harder. Here's how we visualize that:\n\n![continuing the previous visualization with the edge case that the read replica responds to a get request before the replication has finished](https://user-images.githubusercontent.com/1500684/215623612-68909248-67ae-483c-8e92-1e9f292ee3e9.png)\n\n## This solution\n\nThis module comes with several utilities to help you work around these issues.\nSpecifically, it allows you an easy way to add a special cookie to the client\nthat identifies the client's \"transaction number\" which is then used by read\nreplicas to compare to their local transaction number and force the client to\nwait until replication has finished if necessary (with a timeout).\n\nHere's how we visualize that:\n\n![a visualization that shows the primary server sending a transaction number to the client and then the subsequent get request is sent to the replica which waits for replication to finish before responding](https://user-images.githubusercontent.com/1500684/215623623-3815a1bf-2263-4d5f-9720-cd8dc23eb027.png)\n\n## The even better `proxy` solution\n\nAt the time of this writing, LiteFS just released experimental support for a\nproxy server that will handle much of this stuff for you. You simply configure\nthe proxy server in your `litefs.yml` and then you don't need to bother with the\ntx number cookie or ensuring primary on non-get requests at all. The `litefs-js`\nmodule is still useful for one-off situations where you're making mutations in\nGET requests for example, or if you need to know more about the running\ninstances of your application, but for most of the use cases, you can get away\nwith using the proxy.\n[Learn more about using the proxy from this PR](https://github.com/superfly/litefs/pull/271).\n\n## Installation\n\nThis module is distributed via [npm][npm] which is bundled with [node][node] and\nshould be installed as one of your project's `dependencies`:\n\n```\nnpm install --save litefs-js\n```\n\nUnless you plan on using lower-level utilities, you'll need to set two\nenvironment variables on your server:\n\n- `LITEFS_DIR` - the directory where the `.primary` file is stored. This should\n  be what you set your `fuse.dir` config to in the `litefs.yml` config.\n- `DATABASE_FILENAME` - the filename of your sqlite database. This is used to\n  determine the location of the `-pos` file which LiteFS uses to track the\n  transaction number.\n- `INTERNAL_PORT` - the port set in the fly.toml (can be different from `PORT`\n  if you're using the litefs proxy). This is useful for the\n  `getInternalInstanceDomain` utility.\n\n## Usage\n\nThe best way to use this is with the latest version of LiteFS which supports the\n`proxy` server:\n\n```yml\n# litefs.yml\n# ...\nproxy:\n  # matches the internal_port in fly.toml\n  addr: ':${INTERNAL_PORT}'\n  target: 'localhost:${PORT}'\n  db: '${DATABASE_FILENAME}'\n# ...\n```\n\nFrom there all you really need `litefs-js` for is when you run your database\nmigrations. For example:\n\n```js\n// start.js\nconst fs = require('fs')\nconst { spawn } = require('child_process')\nconst os = require('os')\nconst path = require('path')\nconst { getInstanceInfo } = require('litefs-js')\n\nasync function go() {\n\tconst { currentInstance, currentIsPrimary, primaryInstance } =\n\t\tawait getInstanceInfo()\n\n\tif (currentIsPrimary) {\n\t\tconsole.log(\n\t\t\t`Instance (${currentInstance}) in ${process.env.FLY_REGION} is primary. Deploying migrations.`,\n\t\t)\n\t\tawait exec('npx prisma migrate deploy')\n\t} else {\n\t\tconsole.log(\n\t\t\t`Instance (${currentInstance}) in ${process.env.FLY_REGION} is not primary (the primary instance is ${primaryInstance}). Skipping migrations.`,\n\t\t)\n\t}\n\n\tconsole.log('Starting app...')\n\tawait exec('node ./build')\n}\ngo()\n\nasync function exec(command) {\n\tconst child = spawn(command, { shell: true, stdio: 'inherit' })\n\tawait new Promise((res, rej) =\u003e {\n\t\tchild.on('exit', code =\u003e {\n\t\t\tif (code === 0) {\n\t\t\t\tres()\n\t\t\t} else {\n\t\t\t\trej()\n\t\t\t}\n\t\t})\n\t})\n}\n```\n\nThe only other thing you need to worry about is if you ever perform a mutation\nas a part of a GET request. The proxy will handle forwarding non-GET requests\nfor you automatically, but if you need to write to a non-primary in a GET,\nyou'll need to use some of the utilities to ensure the server replays the\nrequest to the primary.\n\n### Lower Level Usage\n\nIntegrating this with your existing server requires integration in two places:\n\n1. Setting the transaction number cookie on the client after mutations have\n   finished\n2. Waiting for replication to finish before responding to requests\n\nLow-level utilities are exposed, but higher level utilities are also available\nfor `express` and `remix`.\n\nAdditionally, any routes that trigger database mutations will need to ensure\nthey are running on the primary instance, which is where `ensurePrimary` comes\nin handy.\n\n### Express\n\n```ts\nimport express from 'express'\nimport {\n\tgetSetTxNumberMiddleware,\n\tgetTransactionalConsistencyMiddleware,\n\tgetEnsurePrimaryMiddleware,\n} from 'litefs-js/express'\n\nconst app = express()\n// this should appear before any middleware that mutates the database\napp.use(getEnsurePrimaryMiddleware())\n\n// this should appear before any middleware that retrieves something from the database\napp.use(getTransactionalConsistencyMiddleware())\n\n// ... other middleware that might mutate the database here\napp.use(getSetTxNumberMiddleware())\n\n// ... middleware that send the response here\n```\n\nThe tricky bit here is that often your middleware that mutates the database is\nalso responsible for sending the responses, so you may need to use a lower-level\nutility like `setTxCookie` to set the cookie after mutations.\n\n### Remix\n\nUntil we have proper middleware support in Remix, you'll have to use the express\nor other lower-level utilities. You cannot currently use this module with the\nbuilt-in Remix server because there's no way to force the server to wait before\ncalling your loaders. Normally, you just need to use\n`getTransactionalConsistencyMiddleware` in express, and then you can use\n`appendTxNumberCookie` as shown below.\n\nOf course, instead of using express with\n`getTransactionalConsistencyMiddleware`, you could use\n`await handleTransactionalConsistency(request)` to the top of every loader if\nyou like:\n\n```tsx\n// app/root.tsx (and app/routes/*.tsx... and every other loader in your app)\nexport function loader({ request }: DataFunctionArgs) {\n\tawait handleTransactionalConsistency(request)\n\t// ... your loader code here\n}\n```\n\nThe same thing applies to `getEnsurePrimaryMiddleware` as well. If you need or\nlike, you can use `await ensurePrimary()` in every `action` call or any\n`loader`s that mutate the database (of which, there should be few because you\nshould avoid mutations in loaders).\n\nWe're umm... really looking forward to Remix middleware...\n\nThe `appendTxNumberCookie` utility should be used in the `entry.server.ts` file\nin both the `default` export (normally people call this `handleDocumentRequest`\nor `handleRequest`) and the `handleDataRequest` export.\n\n```tsx\n// app/entry.server.ts\nimport { appendTxNumberCookie } from 'litefs-js/remix'\n\nexport default async function handleRequest(\n\trequest: Request,\n\tresponseStatusCode: number,\n\tresponseHeaders: Headers,\n\tremixContext: EntryContext,\n) {\n\t// Most of the time, all mutations are finished by now, but just make sure\n\t// you're finished with all mutations before this line:\n\tawait appendTxNumberCookie(request, responseHeaders)\n\t// send the response\n}\n\nexport async function handleDataRequest(\n\tresponse: Response,\n\t{ request }: Parameters\u003cHandleDataRequestFunction\u003e[1],\n) {\n\t// Most of the time, all mutations are finished by now, but just make sure\n\t// you're finished with all mutations before this line:\n\tawait appendTxNumberCookie(request, response.headers)\n\treturn response\n}\n```\n\n### Other\n\nThere are several other lower-level utilities that you can use. They allow for\nmore customization and are documented via jsdoc. Utilities you may find helpful:\n\n- `ensurePrimary` - Use this to ensure that the server that's handling the\n  request is the primary server. This is useful if you know you need to do a\n  mutation for that request.\n- `getInstanceInfo` - get the `currentInstance` and `primaryInstance` hostnames\n  from the filesystem.\n- `waitForUpToDateTxNumber` - wait for the local transaction number to match the\n  one you give it\n- `getTxNumber` - read the transaction number from the filesystem.\n- `getTxSetCookieHeader` - get the `Set-Cookie` header value for the transaction\n  number\n- `checkCookieForTransactionalConsistency` - the logic used to check the\n  transaction number cookie for consistency and wait for replication if\n  necessary.\n- `getAllInstances` - get all the instances of your app currently running\n- `getInternalInstanceDomain` - get the internal domain for the current instance\n  so you can communicate between instances of your app (ensure you've set the\n  `INTERNAL_PORT` environment variable to what appears in your `fly.toml`).\n\n## How it works\n\nThis module uses the special `.primary` directory in your Fuse filesystem to\ndetermine the primary\n([litefs primary docs](https://fly.io/docs/litefs/primary/)), and the `-pos`\nfile to determine the transaction number\n([litefs transaction number docs](https://fly.io/docs/litefs/position/)).\n\nWhen necessary, replay requests are made by responding with a 409 status code\nand a `fly-replay` header\n([docs on dynamic request routing](https://fly.io/docs/reference/dynamic-request-routing/)).\n\n## Inspiration\n\nThis was built to make it much easier for people to take advantage of\ndistributed SQLite with LiteFS on Fly.io. The bulk of the logic was extracted\nfrom\n[kentcdodds/kentcdodds.com](https://github.com/kentcdodds/kentcdodds.com/blob/96d76de72a4a48089f2eb22a88a6ad1c6f847fa1/server/fly.ts).\n\n## LICENSE\n\nMIT\n\n\u003c!-- prettier-ignore-start --\u003e\n[npm]: https://www.npmjs.com\n[node]: https://nodejs.org\n[build-badge]: https://img.shields.io/github/actions/workflow/status/fly-apps/litefs-js/validate.yml?logo=github\u0026style=flat-square\n[build]: https://github.com/fly-apps/litefs-js/actions?query=workflow%3Avalidate\n[version-badge]: https://img.shields.io/npm/v/litefs-js.svg?style=flat-square\n[package]: https://www.npmjs.com/package/litefs-js\n[license-badge]: https://img.shields.io/npm/l/litefs-js.svg?style=flat-square\n[license]: https://github.com/fly-apps/litefs-js/blob/main/LICENSE\n\u003c!-- prettier-ignore-end --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffly-apps%2Flitefs-js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffly-apps%2Flitefs-js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffly-apps%2Flitefs-js/lists"}