{"id":28391381,"url":"https://github.com/wearebraid/drupal-jsonapi-extractor","last_synced_at":"2025-06-25T21:31:57.617Z","repository":{"id":44129814,"uuid":"191802992","full_name":"wearebraid/drupal-jsonapi-extractor","owner":"wearebraid","description":"Library to extract, transform, and download an entire Drupal json:api graph.","archived":false,"fork":false,"pushed_at":"2023-01-04T00:39:10.000Z","size":745,"stargazers_count":7,"open_issues_count":15,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-24T05:58:17.664Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wearebraid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-06-13T17:07:04.000Z","updated_at":"2024-04-18T13:57:32.000Z","dependencies_parsed_at":"2023-02-01T15:00:52.121Z","dependency_job_id":null,"html_url":"https://github.com/wearebraid/drupal-jsonapi-extractor","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/wearebraid/drupal-jsonapi-extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wearebraid%2Fdrupal-jsonapi-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wearebraid%2Fdrupal-jsonapi-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wearebraid%2Fdrupal-jsonapi-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wearebraid%2Fdrupal-jsonapi-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wearebraid","download_url":"https://codeload.github.com/wearebraid/drupal-jsonapi-extractor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wearebraid%2Fdrupal-jsonapi-extractor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261956148,"owners_count":23236024,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T09:11:51.548Z","updated_at":"2025-06-25T21:31:57.609Z","avatar_url":"https://github.com/wearebraid.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Drupal JSON:API Extractor\n\n[![CircleCI](https://circleci.com/gh/wearebraid/drupal-jsonapi-extractor.svg?style=svg)](https://circleci.com/gh/wearebraid/drupal-jsonapi-extractor)\n![node](https://img.shields.io/node/v/drupal-jsonapi-extractor.svg)\n\nThis package is a Drupal [json:api](https://www.drupal.org/project/jsonapi)\nclient library with one primary responsibility - to crawl through a Drupal\nproduced json:api and save the resulting data to static json files in\ndirectory structures to allow easy access to the files.\n\nWhy all the trouble? For Drupal sites with only hundreds or low thousands of\npages (the majority) enabling the (now core) [json:api](https://www.drupal.org/project/jsonapi)\nmodule in conjunction with this library allows for fully static front ends.\nHaving a way to export all of a site's data to static json files allows\nthose files to be deployed, statically, along with a site's decoupled front\nend.\n\nIt also presents an opportunity to transform the standard json:api output\nto something a little more friendly for developers to work with. Ideally this\nlibrary is used during the static generation process.\n\n## Getting started\n\nCrawling all drupal nodes of a given content type with each node's associated\nrelationships (including [paragraphs](https://www.drupal.org/project/paragraphs))\nis pretty easy.\n\n```js\nconst { Spider } = require('drupal-jsonapi-extractor')\n\nconst baseURL = 'https://example.org/jsonapi/'\nconst spider = new Spider({ baseURL })\n\nspider.crawl('/node/blog')\n// or to crawl every published node\nspider.crawlNodes()\n```\nWhile the above `Spider` does crawl through an entire set of content types it does\nnot actually do anything with the results. This is where we introduce the\n`Extractor` object.\n\n```js\nconst { Spider, Extractor } = require('drupal-jsonapi-extractor')\n\nconst baseURL = 'https://example.org/jsonapi/'\nconst spider = new Spider({ baseURL })\nconst extractor = new Extractor(spider, { location: './downloads' })\n\nextractor.wipe().then(() =\u003e spider.crawl('/node/content-type'))\n```\n\n\u003e Note: The extractor has a helpful utility function `wipe()` which will returns a\n\u003e `Promise` and ensures the target directory is completely empty before resolving.\n\nThe above code will output a new `downloads` directory with the structure:\n\n```\ndownloads/\n  _resources/\n    node/\n      blog/\n        0ef56bbd-b2d6-475e-8b83-e1fa9bc1e7fb.json\n    paragraph/\n      hero/\n        425a6dc1-5158-4f12-8d54-eb8a7af369f0.json\n    taxonomy_term/\n      tags/\n        2d850e4b-9d2f-4b8f-b1e7-ad959de8b393.json\n  _slugs/\n    node/\n      1.json\n    blogs/\n      my-first-blog-post.json\n```\n\nThis structure is intended to serve static sites well by allowing lookup by\nthe unique json:api global unique id, as well as the more traditional drupal\npath (`node/1`) and a node's alias \"slug\" (`/blogs/my-first-blog-post`).\n\nThe extractor by default saves the exact output of the json:api. However, when\ndeveloping your decoupled front end you may prefer a slightly less verbose json\nschema. This package includes a transformer that allows easily \"cleaning\" of the\noutput:\n\n```js\nconst extractor = new Extractor(spider, {\n  location: './downloads'\n  clean: true\n})\n```\n\nSometimes it is nice to see the progress of the download process. This package\nincludes a console logger as well.\n\n```js\nconst { Spider, Extractor, Logger } = require('drupal-jsonapi-extractor')\n\nconst baseURL = 'https://example.org/jsonapi/'\nconst spider = new Spider({ baseURL })\nconst extractor = new Extractor(spider, { location: './downloads' })\nconst logger = new Logger([spider, extractor])\n\nspider.crawl('/node/content-type')\n```\nThe logger in our example would print to the command line:\n\n```sh\n✔️  node: 1\n✔️  taxonomy_term: 1\n✔️  paragraph: 1\n----------------------------\n🎉   Crawl complete!\nErrors.................0\nnode...................1\nparagraph..............1\ntaxonomy_term..........1\n```\n\n## Configuration options\n\nEach of the provided classes have a number of configuration options.\n\n### Spider\n\nYou pass options as the first argument when instantiating a new `Spider`.\n\n```js\nnew Spider(options)\n```\n\n```js\n{\n  // (required) Should include the /jsonapi/ segment\n  baseURL: 'https://example.org/jsonapi/'\n\n  // (optional) Instance of axios with baseURL already applied\n  api: axios,\n\n  // Quite the program on a crawl error\n  terminateOnError: false,\n\n  // What is the maximum number of concurrent api requests the spider can open.\n  // you get timeout errors from the api, reduce this number.\n  maxConcurrent: 5,\n\n  // (optional) Resource class configuration options\n  resourceConfig: {\n    // (optional) Array of regex that is used to determine which relationships should be crawled\n    relationships: [\n      // By default, only relationships that start with field_ are crawled\n      new RegExp(/^field_/)\n    ]\n  }\n}\n```\n\n### Extractor\n\nYou pass options as the second argument when instantiating a new `Extractor`.\n\n```js\nconst extractor = new Extractor(spider, options)\nextractor.wipe().then(() =\u003e spider.crawlNodes())\n// To limit the depth of a crawl, pass a max depth (rarely needed since the\n// package handles recursive references)\nextractor.wipe().then(() =\u003e spider.crawlNodes(5))\n```\n\n\u003e Note: above we use a helpful utility method `wipe()` which will returns a\n\u003e `Promise` and ensures the target directory is completely empty before resolving.\n\n```js\n{\n  // The location to save files (will create directories automatically)\n  location: './',\n\n  // Should the data be transformed or \"cleaned\" before being saved to disk?\n  clean: false,\n\n  // Sometimes it's helpful to see pretty-printed json, just flip this to true.\n  pretty: false,\n\n  // The function to pass each Resource through before saving it if clean is true\n  // By default we use our own transform function, this function takes a number of\n  // options itself, or you can choose to use your own callback altogether.\n  transformer: transformer({\n\n    // Array of regular expressions to keep or reject each key in the \"attributes\"\n    // section of a json:api response. Matches are kept. These are the defaults.\n    attributeFilters: [\n      /^field_/, // Common field prefix\n      /^(title|created|changed|langcode|body)$/, // Common for node entities\n      /^(name|weight|description)$/, // Common for taxonomies\n      /^(parent_type|parent_id)$/ // Common for paragraphs,\n    ],\n    \n    // Same functionality as the attributeFilters but applied to the\n    // \"relationships\" section of the json:api response.\n    relationshipFilters: [\n      /^field_/ // Common field prefix\n    ],\n\n    // Within each \"field\" we can remove certain fields we no longer want, in\n    // this case properties of field that contains an object. Matches are removed.\n    fieldPropertyFilters: [\n      /^links$/\n    ],\n\n    // A callback that is passed a Resource object and expected to return a\n    // cleaned up \"fields\" object. The default applies the above filters, but\n    // a custom callback could be used here.\n    cleanFields: callback\n  })\n}\n```\n\nInternally this library represents every crawled response with a `Resource`\nobject. If you choose to override the `transformer` callback it will be given\na `Resource` as an argument. You can read the source code for details on it's\n[functionality](./src/Resource.js). If you want change the configuration options\nof our transformer, you can customize it:\n\n```js\nconst { Spider, Extractor, transformer } = require('drupal-jsonapi-extractor')\n\nconst baseURL = 'https://example.org/jsonapi/'\nconst spider = new Spider({ baseURL })\nconst extractor = new Extractor(spider, {\n  location: './downloads'\n  clean: true,\n  transformer: transformer({\n    attributeFilters: [\n      /^custom_attribute_to_keep$/\n    ]\n  })\n})\n\nspider.crawl('/node/content-type')\n```\n\n## Logger\n\nThe logger, at the moment, is pretty simple with just one configuration option:\n\n```js\nnew Logger([...emitters], {\n  // Set the verbosity of the logger:\n  // 0 - Log nothing\n  // 1 - (default) Show a simple tally of number of downloads and number of errors\n  // 2 - Log each entity and error as its downloading\n  // 3 - Log every event being listened to by the logger\n  verbosity: 1\n})\n```\n\n## To do\n\nCurrently there is effectively no test coverage, although test files for the\nclasses have been written with an instantiation check in each.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwearebraid%2Fdrupal-jsonapi-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwearebraid%2Fdrupal-jsonapi-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwearebraid%2Fdrupal-jsonapi-extractor/lists"}