{"id":13516616,"url":"https://github.com/gajus/postloader","last_synced_at":"2025-04-19T16:53:58.835Z","repository":{"id":57339124,"uuid":"110373744","full_name":"gajus/postloader","owner":"gajus","description":"A scaffolding tool for projects using DataLoader, Flow and PostgreSQL.","archived":false,"fork":false,"pushed_at":"2020-06-15T21:41:40.000Z","size":98,"stargazers_count":50,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-12T00:36:14.279Z","etag":null,"topics":["dataloader","flowtype","postgresql"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gajus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null},"funding":{"github":"gajus","patreon":"gajus"}},"created_at":"2017-11-11T19:29:28.000Z","updated_at":"2024-10-25T10:32:29.000Z","dependencies_parsed_at":"2022-08-28T07:11:12.131Z","dependency_job_id":null,"html_url":"https://github.com/gajus/postloader","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fpostloader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fpostloader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fpostloader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gajus%2Fpostloader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gajus","download_url":"https://codeload.github.com/gajus/postloader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249453705,"owners_count":21275008,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataloader","flowtype","postgresql"],"created_at":"2024-08-01T05:01:24.172Z","updated_at":"2025-04-19T16:53:58.811Z","avatar_url":"https://github.com/gajus.png","language":"JavaScript","readme":"# PostLoader\n\n[![GitSpo Mentions](https://gitspo.com/badges/mentions/gajus/postloader?style=flat-square)](https://gitspo.com/mentions/gajus/postloader)\n[![Travis build status](http://img.shields.io/travis/gajus/postloader/master.svg?style=flat-square)](https://travis-ci.org/gajus/postloader)\n[![Coveralls](https://img.shields.io/coveralls/gajus/postloader.svg?style=flat-square)](https://coveralls.io/github/gajus/postloader)\n[![NPM version](http://img.shields.io/npm/v/postloader.svg?style=flat-square)](https://www.npmjs.org/package/postloader)\n[![Canonical Code Style](https://img.shields.io/badge/code%20style-canonical-blue.svg?style=flat-square)](https://github.com/gajus/canonical)\n[![Twitter Follow](https://img.shields.io/twitter/follow/kuizinas.svg?style=social\u0026label=Follow)](https://twitter.com/kuizinas)\n\nA scaffolding tool for projects using [DataLoader](https://github.com/facebook/dataloader), [Flow](https://flow.org/) and [PostgreSQL](https://www.postgresql.org/).\n\n* [Motivation](#motivation)\n  * [What makes this different from using an ORM?](#what-makes-this-different-from-using-an-orm)\n* [Behaviour](#behaviour)\n  * [Unique key loader](#unique-key-loader)\n  * [Non-unique `_id` loaders](#non-unique-id-loader)\n  * [Non-unique joining table loader](#non-unique-joining-table-loader)\n* [Naming conventions](#naming-conventions)\n  * [Type names](#type-names)\n  * [Property names](#property-names)\n  * [Loader names](#loader-names)\n* [Usage examples](#usage-examples)\n  * [Generate DataLoader loaders for all database tables](#generate-dataloader-loaders-for-all-database-tables)\n  * [Consume the generated code](#consume-the-generated-code)\n  * [Handling non-nullable columns in materialized views](#handling-non-nullable-columns-in-materialized-views)\n\n## Motivation\n\nKeeping database and codebase in sync is hard. Whenever changes are done to the database schema, these changes need to be reflected in the codebase's type declarations.\n\nMost of the loaders are needed to perform simple PK look ups, e.g. `UserByIdLoader`. Writing this logic for every table is a mundane task.\n\nPostLoader solves both of these problems by:\n\n1. Creating type declarations for all database tables.\n1. Creating loaders for the most common lookups.\n\nIf you are interested to learn more, I have written an article on the subject: [I reduced GraphQL codebase size by 40% and increased type coverage to 90%+. Using code generation to create data loaders for all database resources.](https://medium.com/@gajus/i-reduced-graphql-codebase-size-by-40-and-increased-type-coverage-to-90-a2c87fdc78d3).\n\n### What makes this different from using an ORM?\n\n1. ORM is not going to give you strict types and code completion.\n1. ORM has runtime overhead for constructing the queries and formatting the results.\n\n## Behaviour\n\nPostLoader is a CLI program (and a collection of utilities) used to generate code based on a PostgreSQL database schema.\n\nThe generated code consists of:\n\n1. Flow type declarations describing every table in the database.\n1. A factory function used to construct a collection of loaders.\n\n### Unique key loader\n\nA loader is created for every column in a unique index ([unique indexes including multiple columns are not supported](https://github.com/gajus/postloader/issues/1)), e.g. `UserByIdLoader`.\n\n### Non-unique `_id` loader\n\nA loader is created for every column that has name ending with `_id`.\n\nA non-unique loader is used to return multiple rows per lookup, e.g. `CitiesByCountryIdLoader`. The underlying data in this example comes from a table named \"city\". PostLoader is using [`pluralize`](https://www.npmjs.com/package/pluralize) module to pluralize the table name.\n\n### Non-unique joining table loader\n\nA loader is created for every resource discoverable via a joining table.\n\n1. A joining table consists of at least 2 columns that have names ending `_id`.\n1. The table name is a concatenation of the column names (without `_id` suffix) (in alphabetical order, i.e. `genre_movie`, not `movie_genre`).\n\n#### Example\n\nAssume a many-to-many relationship of movies and genres:\n\n```sql\nCREATE TABLE movie (\n  id integer NOT NULL,\n  name text\n);\n\nCREATE TABLE venue (\n  id integer NOT NULL,\n  name text\n);\n\nCREATE TABLE genre_movie (\n  id integer NOT NULL,\n  genre_id integer NOT NULL,\n  movie_id integer NOT NULL\n);\n\n```\n\nProvided the above schema, PostLoader will create two non-unique loaders:\n\n* `MoviesByGenreIdLoader`\n* `GenresByMovieIdLoader`\n\n## Naming conventions\n\n### Type names\n\nType names are created from table names.\n\nTable name is camel cased, the first letter is uppercased and suffixed with \"RecordType\", e.g. \"movie_rating\" becomes `MovieRatingRecordType`.\n\n### Property names\n\nProperty names of type declarations are derived from the respective table column names.\n\nColumn names are camel cased, e.g. \"first_name\" becomes `firstName`.\n\n### Loader names\n\nLoader names are created from table names and column names.\n\nTable name is camel cased, the first letter is uppercased, suffixed with \"By\" constant, followed by the name of the property (camel cased, the first letter is uppercased) used to load the resource, followed by \"Loader\" constant, e.g. a record from \"user\" table with \"id\" column can be loaded using `UserByIdLoader` loader.\n\n## Usage examples\n\n### Generate DataLoader loaders for all database tables\n\n```bash\nexport POSTLOADER_DATABASE_CONNECTION_URI=postgres://postgres:password@127.0.0.1/test\nexport POSTLOADER_COLUMN_FILTER=\"return /* exclude tables that have a _view */ !columns.map(column =\u003e column.tableName).includes(tableName + '_view')\"\nexport POSTLOADER_TABLE_NAME_MAPPER=\"return tableName.endsWith('_view') ? tableName.slice(0, -5) : tableName;\"\nexport POSTLOADER_DATA_TYPE_MAP=\"{\\\"email\\\":\\\"text\\\"}\"\n\npostloader generate-loaders \u003e ./PostLoader.js\n\n```\n\nThis generates a file containing a factory function used to construct a DataLoader for every table in the database and Flow type declarations in the following format:\n\n```js\n// @flow\n\nimport {\n  getByIds,\n  getByIdsUsingJoiningTable\n} from 'postloader';\nimport DataLoader from 'dataloader';\nimport type {\n  DatabaseConnectionType\n} from 'slonik';\n\nexport type UserRecordType = {|\n  +id: number,\n  +email: string,\n  +givenName: string | null,\n  +familyName: string | null,\n  +password: string,\n  +createdAt: string,\n  +updatedAt: string | null,\n  +pseudonym: string\n|};\n\n// [..]\n\nexport type LoadersType = {|\n  +UserByIdLoader: DataLoader\u003cnumber, UserRecordType\u003e,\n  +UsersByAffiliateIdLoader: DataLoader\u003cnumber, $ReadOnlyArray\u003cUserRecordType\u003e\u003e,\n  // [..]\n|};\n\n// [..]\n\nexport const createLoaders = (connection: DatabaseConnectionType) =\u003e {\n  const UserByIdLoader = new DataLoader((ids) =\u003e {\n    return getByIds(connection, 'user', ids, 'id', '\"id\", \"email\", \"given_name\" \"givenName\", \"family_name\" \"familyName\", \"password\", \"created_at\" \"createdAt\", \"updated_at\" \"updatedAt\", \"pseudonym\"', false);\n  });\n  const UsersByAffiliateIdLoader = new DataLoader((ids) =\u003e {\n    return getByIdsUsingJoiningTable(connection, 'affiliate_user', 'user', 'user', 'affiliate', 'r2.\"id\", r2.\"email\", r2.\"given_name\" \"givenName\", r2.\"family_name\" \"familyName\", r2.\"password\", r2.\"created_at\" \"createdAt\", r2.\"updated_at\" \"updatedAt\", r2.\"pseudonym\"', ids);\n  });\n\n  // [..]\n\n  return {\n    UserByIdLoader,\n    UsersByAffiliateIdLoader,\n    // [..]\n  };\n};\n\n\n```\n\nNotice that the generated file depends on `postloader` package, i.e. you must install `postloader` as the main project dependency (as opposed to a development dependency).\n\n### Consume the generated code\n\n1. Dump the generated code to a file in your project tree, e.g. `/generated/PostLoader.js`.\n1. Create PostgreSQL connection resource using [Slonik](https://github.com/gajus/slonik).\n1. Import `createLoaders` factory function from the generated file.\n1. Create the loaders collections.\n1. Consume the loaders.\n\nExample:\n\n```js\n// @flow\n\nimport {\n  createPool\n} from 'slonik';\nimport {\n  createLoaders\n} from './generated/PostLoader';\nimport type {\n  UserRecordType\n} from './generated/PostLoader';\n\nconst pool = createPool('postgres://');\n\nconst loaders = createLoaders(pool);\n\nconst user = await loaders.UserByIdLoader.load(1);\n\nconst updateUserPassword = (user: UserRecordType, newPassword: string) =\u003e {\n  // [..]\n};\n\n```\n\nYou can optionally pass a second parameter to `createLoaders` – loader configuration map, e.g.\n\n```js\nconst loaders = createLoaders(connection, {\n  UserByIdLoader: {\n    cache: false\n  }\n});\n\n```\n\n### Handling non-nullable columns in materialized views\n\nUnfortunately, PostgreSQL does not describe materilized view columns as non-nullable even when you add a constraint that enforce this contract ([see this Stack Overflow question](https://stackoverflow.com/q/47242219/368691)).\n\nFor materialied views, you need to explicitly identify which collumns are non-nullable. This can be done by adding `POSTLOAD_NOTNULL` comment to the column, e.g.\n\n```sql\nCOMMENT ON COLUMN user.id IS 'POSTLOAD_NOTNULL';\nCOMMENT ON COLUMN user.email IS 'POSTLOAD_NOTNULL';\nCOMMENT ON COLUMN user.password IS 'POSTLOAD_NOTNULL';\nCOMMENT ON COLUMN user.created_at IS 'POSTLOAD_NOTNULL';\nCOMMENT ON COLUMN user.pseudonym IS 'POSTLOAD_NOTNULL';\n\n```\n\nAlternatively, update the `pg_attribute.attnotnull` value of the target columns, e.g.\n\n```sql\nCREATE OR REPLACE FUNCTION set_attribute_not_null(view_name TEXT, column_names TEXT[])\nRETURNS void AS\n$$\nBEGIN\n  UPDATE pg_catalog.pg_attribute\n  SET attnotnull = true\n  WHERE attrelid IN (\n    SELECT\n      pa1.attrelid\n    FROM pg_class pc1\n    INNER JOIN pg_namespace pn1 ON pn1.oid = pc1.relnamespace\n    INNER JOIN pg_attribute pa1 ON pa1.attrelid = pc1.oid AND pa1.attnum \u003e 0 AND NOT pa1.attisdropped\n    WHERE\n      pn1.nspname = 'public' AND\n      pc1.relkind = 'm' AND\n      pc1.relname = view_name AND\n      pa1.attname = ANY(column_names)\n  );\nEND;\n$$ language 'plpgsql';\n\nset_attribute_not_null('person_view', ARRAY['id', 'imdb_id', 'tmdb_id', 'headshot_image_name', 'name']);\n\n```\n","funding_links":["https://github.com/sponsors/gajus","https://patreon.com/gajus"],"categories":["JavaScript"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgajus%2Fpostloader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgajus%2Fpostloader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgajus%2Fpostloader/lists"}