{"id":18712908,"url":"https://github.com/mrsteele/contentful-big-data","last_synced_at":"2025-10-04T20:50:41.685Z","repository":{"id":60685255,"uuid":"540258103","full_name":"mrsteele/contentful-big-data","owner":"mrsteele","description":"Use Contentful's CDA and GraphQL to do big queries","archived":false,"fork":false,"pushed_at":"2025-06-10T12:56:59.000Z","size":658,"stargazers_count":0,"open_issues_count":8,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-12T02:53:05.884Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mrsteele.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-09-23T03:03:57.000Z","updated_at":"2025-05-12T12:28:43.000Z","dependencies_parsed_at":"2024-01-26T14:25:04.104Z","dependency_job_id":"4c6feefa-090f-4341-93a7-6632cc87e0a9","html_url":"https://github.com/mrsteele/contentful-big-data","commit_stats":{"total_commits":72,"total_committers":4,"mean_commits":18.0,"dds":0.5833333333333333,"last_synced_commit":"80af7b90a64026c7c9798bd4f5246438f3fe162d"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/mrsteele/contentful-big-data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsteele%2Fcontentful-big-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsteele%2Fcontentful-big-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsteele%2Fcontentful-big-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsteele%2Fcontentful-big-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mrsteele","download_url":"https://codeload.github.com/mrsteele/contentful-big-data/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mrsteele%2Fcontentful-big-data/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261396817,"owners_count":23152455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T12:44:18.113Z","updated_at":"2025-10-04T20:50:36.628Z","avatar_url":"https://github.com/mrsteele.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# contentful-big-data 🥦\n\n\u003cimg width=\"100\" align=\"right\" src=\"https://user-images.githubusercontent.com/5257284/194174575-ff35d11e-94b1-43a0-a7d4-626432a6f7bc.png\" /\u003e\n\nUse Contentful's CDA and GraphQL to do big queries\n\n[![npm version](https://badge.fury.io/js/contentful-big-data.svg)](https://badge.fury.io/js/contentful-big-data)\n[![codecov](https://codecov.io/gh/mrsteele/contentful-big-data/branch/main/graph/badge.svg?token=MKM3ID7DVP)](https://codecov.io/gh/mrsteele/contentful-big-data)\n\n## About\n\nHave you ever made a CDA request to Contentful, assuming you got all the data, only to later found out you had a total of 1001 entries and Contentful only supports 1000?\n\nThis module is used to wrap around Contentful's CDA and GraphQL to make the most effective use of both of the APIs to perform powerful searches and more conservative response selectors.\n\n### Disclaimers\n\n1. I do not work for Contentful. I am a developer who professionally and personally use the service and this is something I was seeing as a \"missing link\" to get the data in its entirety.\n2. Use at your own risk. Contentful has limitations on how many API requests you can do in any given period. If you are charged for overages I recommend you pay attention to how you use this tool as it is meant to do a lot of queries.\n\n## Usage\n\n1. First add this library to your node project:\n\n```bash\nnpm i contentful-big-data\n```\n\n2. Use the library like so\n\n```js\n// es6\nimport CBD from 'contentful-big-data'\n// const CBD = require('contentful-big-data')\n\nconst cbd = CBD({\n  space: 'space-id-here'\n  key: 'space-key-here'\n})\n\nconst results = await cbd.fetch({\n  // Regular CDA query parameters (read me here: https://www.contentful.com/developers/docs/references/content-delivery-api/#/reference/search-parameters/inclusion)\n  'fields.author[in]': 'authorid1,authorid2,authorid3'\n}, `{\n  name\n  image { url }\n}`)\n\n// You can also do this\n// cbd.fetch({ ... }).then (results =\u003e ...)\n\n// will return ALL the results in your space\nconsole.log(results)\n// [{ name: 'name-here', image: { url: 'https://cfassets.com/image/is/here' } }]\n```\n\n## Features\n\nThis library comes from a need due to limitations with Contentful\n1. Request size too large\n2. Response size too large\n3. Response collection too narrow (not the full set)\n4. Response includes `null` entries\n\nBecause of these limitations, this library has been created to satisfy these shortcomings.\n\n* **Multi-Query** - The concept revolves around making a CDA request with `limit=0` with the purpose of getting the count of results. This tells us how many additional requests are needed to complete the dataset.\n* **Multi-Service** - Instead of using the CDN or GraphQL, **use both**. The CDN is great for requesting all the IDs, and the GraphQL is useful for limiting the response size.\n* **Retry-Oriented** - Sometimes you hit a limit with Contentful. This could be from the size of the response, or because you hit the rate limiter. If that happens we should adjust where appropriate and try again.\n* **Size-Aware** - There are limitations on both the request and response sizes. We should break apart requests that are too large, and dynamically respond if we get a reponse too large error from Contentful. *(coming soon)*\n\n## Examples\n\nThe following examples can be used as a guide in your own implementation.\n\n```js\n// get ALL entries title (careful, if you have 1 mil entries you get them all!)\nconst allEntries = await cbd.fetch({\n  content_type: 'page'\n}, `{\n  title\n}`)\n\n// assuming you have paginated data, get only the second 100 assuming you are on page 2\nconst secondPage = await cbd.fetch({\n  content_type: 'page',\n  skip: 100,\n  limit: 100\n}, `{\n  title\n}`)\n\n// Get the thumbnail, title and url of all pages that have specific authors\nconst linkedPages = await cbd.fetch({\n  content_type: 'page',\n  'fields.author.sys.id[in]': ['qwerty','yuiop']\n  }, `{\n    title\n    url\n    thumbnail {\n      url\n      width\n      height\n    }\n}`)\n\n```\n\n## Docs\n\n### CBD (constructor)\n\n```js\nconst cbd = CBD(config={\n  space: 'space-id-here', // (required) - Your space id in Contentful\n  key: 'space-key-here', // (possibly required) - Your api access key to your Contentful space\n  previewKey: 'preview-key-here', // (possibly required) - Your preview api access key to your Contentful space\n  env: 'master', // (optional) - Your Contentful env. Defaults to `master`\n  retry: 3, //  (optional) - The amount of retries after hitting the rate limiter for all requests. Defaults to `3`.\n  failSilently: false // (optional) - If you want your requests to fail silently or throw an error. Defaults to false (and throws errors).\n})\n```\n\n\u003e Note - You can populate both `key` and `previewKey` but if you only plan to use the tool for one purpose, only one key is required.\n\n### CBD.fetch\n\n```js\n// filters, selectors, and options\ncbd.fetch(\n  // filters - Please refer to the [Content Delivery API](https://www.contentful.com/developers/docs/references/content-delivery-api/) to look up all the properties. This library adheres to all of them, but will ignore the `select` property because we opt to use the GraphQL selectors for more granularity. Note that the `content_type` property is required due to limitations with Contentful.\n{\n  custom: 'see below'\n},\n// selectors - Please adhere to the [Contentful GraphQL API](https://www.contentful.com/developers/docs/references/graphql/) to determine how your query should be formatted. We always do a `Collection` request, just wrap your schema in curly braces.\n`{\n  fields\n  goHere\n  graphQL {\n    entries\n  }\n  seeSelectorsBelow\n}`,\n// options - Comments below...\n{\n  isPreview: false, // (optional) - If `true`, will use the `previewKey` and access the preview API for both the CDA and GraphQL services.\n  retry: 3, // (optional) - If set, this number will be used at retry attempts. Default retry attempts is `3` but depends on your use-case and how many other services are hitting Contentful APIs at the same time.\n  verbose: false, // If `true`, we will return the full GraphQL response object, otherwise we only return the array of results.\n  failSilently: false, // (optional) - If `true`, this will not throw any errors when requests fail.\n})\n```\n\n## References\n\n* https://www.contentful.com/developers/docs/references/content-delivery-api\n* https://www.contentful.com/developers/docs/references/graphql/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrsteele%2Fcontentful-big-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmrsteele%2Fcontentful-big-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmrsteele%2Fcontentful-big-data/lists"}