{"id":19174583,"url":"https://github.com/equalitie/bundler","last_synced_at":"2025-05-07T18:20:55.552Z","repository":{"id":26244531,"uuid":"29691518","full_name":"equalitie/bundler","owner":"equalitie","description":"Site bundling system for use in Ceno and DDeflect","archived":false,"fork":false,"pushed_at":"2016-11-05T07:01:19.000Z","size":346,"stargazers_count":9,"open_issues_count":3,"forks_count":4,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-14T04:54:29.658Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/equalitie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-22T17:46:55.000Z","updated_at":"2017-12-21T05:14:24.000Z","dependencies_parsed_at":"2022-08-24T11:01:02.680Z","dependency_job_id":null,"html_url":"https://github.com/equalitie/bundler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fbundler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fbundler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fbundler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/equalitie%2Fbundler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/equalitie","download_url":"https://codeload.github.com/equalitie/bundler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931812,"owners_count":21827170,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T10:18:24.856Z","updated_at":"2025-05-07T18:20:55.528Z","avatar_url":"https://github.com/equalitie.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bundler\n\nA utility for bundling resources in web pages into a single page by inlining them\nusing the [data URI scheme](https://en.wikipedia.org/wiki/Data_URI_scheme).\n\nIt is designed to be easy to use in all kinds of scenarios, ranging from ones\nwhere one simply wants to produce a web page with certain resources bundled to\nscenarios where one would like to proxy resource requests, include caching\nfunctionality, modify request headers, and more.\n\n# Basic Usage\n\nThe simplest use case for the bundler is the case where one would like to:\n\n1. Fetch a web page, given a URL\n2. Replace references to certain kinds of resources with their data URIs\n\nThis can be accomplished very simply.  In the following example, we can\nfetch [Hacker News](https://news.ycombinator.com)'s main page and bundle the\nimages on the site into the HTML.\n\n```javascript\nvar bundleMaker = new bundler.Bundler('https://news.ycombinator.com');\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(bundle);\n});\n```\n\nCurrently, there are five resource handlers exported in the bundler module:\n\n1. replaceImages\n2. replaceCSSFiles\n3. replaceJSFiles\n4. replaceLinks\n5. replaceURLCalls\n\nThe first three are pretty straightforward. You register all of them the same way (like in the example of `repaceImages` above).\n\n`replaceLinks` accepts a function that will be called to generate new links.  The signature of this function is\n\n```javascript\nfunction linkReplacer(baseURL, resourceURL) {\n  return combine(baseURL, resourceURL);\n}\n```\n\nHere, you are free to define how you combine the baseURL and resourceURL as long as you return the value at the end.\n\nFor example, you might register a link replacer to transform all urls to the form \"http://localhost:9001?url=URL\".\n\n`replaceURLCalls` does not accept any parameters.  It will search through the inline `style` attributes of tags and bundle the resources referenced in the CSS `url()` function.\n\nMore on writing your own resource handlers in the *Writing Your Own Hooks*\nsection below.\n\n# Terminology\n\nA `diff` is an object whose key is a resource URL and whose value is a data URI.\n\nA `handler` is a function that analyzes a web page to produce diffs.\n\nA `hook` is a function that can manipulate request option data or diffs.\n\n# Using Hooks\n\n![Architectural diagram](https://raw.githubusercontent.com/equalitie/bundler/master/architecture.png)\n\nBundler provides five opportunities to inject new functionality into the\nbundling process.\n\n1. `originalRequest`  - Before fetching the first, original document\n2. `originalReceived` - Specify handlers used to scan the document and produce diffs \n3. `resourceRequest`  - Before fetching each resource referenced by the original document\n4. `resourceReceived` - After retrieving each resource referenced by the original document\n5. `diffsReceived`    - After accumulating a collection of resource URLs and their data URIs\n\nAs seen above, you can register handlers using the `originalReceived` event.\n\n## Before fetching the original document\n\nBecause one may wish to modify request headers, among other things, bundler\nallows for hooks to be called before making a request for the original document.\nFor example, to replace the `Referer` header with `https://duckduckgo.com`:\n\n```javascript\nvar bundleMaker = new bundler.Bundler('https://yahoo.com');\n\nbundleMaker.on('originalRequest', bundler.spoofHeaders({\n  'Referer': 'https://duckduckgo.com'\n}));\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\nbundleMaker.on('originalReceived', bundler.replaceCSSFiles);\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(bundle);\n});\n```\n\nCurrently, three request modifiers are exported.\n\n### stripHeaders\n\n`stripHeaders` accepts an array of header names to replace with blank values in the\nrequest object and returns the hook function that `Bundler.on` expects.\n\n### spoofHeaders\n\n`spoofHeaders` accepts an object mapping header names to the value to insert in\ntheir place and returns the hook function that `Bundler.on` expects.\n\n### proxyTo\n\n`proxyTo` accepts a URL that requests will be configured to use as a proxy.\nSee [request's documentation on proxies](https://github.com/request/request#proxies)\nto understand how that works.  The function accepts the URL and returns the handler\nfunction that `Bundler.on` expects.\n\n### followRedirects\n\n`followRedirects` confgures the redirect-handling options for the `request` object. It accepts:\n\n1. `first`: bool - Whether or not to follow the first redirect resulting from a request.\n2. `all`: bool - Whether or not to follow all redirects that might result from requests.\n3. `limit`: int - The maximum number of redirects to follow.\n\nSee [request's option documentation](https://github.com/request/request#requestoptions-callback)\nto learn more about what request defaults to.\n\n## Handling resources\n\nResource handlers are used to extract references to resources in a document, such as those in script tags, link tags, and so on.\nThey are responsible for producing diff objects that bundler will go on to\nuse to replace references to such resources with data-URIs.\n\nAs mentioned in the *Basic Usage* section, resource handlers are registered\nusing the `originalReceived` event. Bundler currently exports the following handlers.\n\n1. replaceImages\n2. replaceCSSFiles\n3. replaceJSFiles\n4. replaceLinks\n5. replaceURLCalls\n\n## Before fetching each resource\n\nBundler allows request options to be set for each resource that is to be retrieved.\nThe functions from `bundler.modifyRequests` can be reused here.  For example:\n\n```javascript\nvar bundleMaker = new bundler.Bundler('https://yahoo.com');\n\nbundleMaker.on('resourceRequest', bundler.spoofHeaders({\n  'Referer': 'https://duckduckgo.com'\n}));\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\nbundleMaker.on('originalReceived', bundler.replaceCSSFiles);\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(bundle);\n});\n```\n\nNote that the only difference between this example and the one in the first section\nis that this one registers hooks for the `resourceRequest` method. You can reuse\nhandlers written for `originalRequest` here.\n\n## After retrieving each resource\n\nBundler allows hooks to be registered to directly manipulate the body of a fetched resource through the `resourceReceived` event.  Currently no \nhandlers are exported directly by bundler for these events.\n\nCurrently there is one function exported by bundler to operate on retrieved resources. This is the `bundleCSSRecursively` function, which takes no arguments to use.  It will find instances of calls to `url()` in CSS documents\nand replace the URL within with a data URI.\n\n## After building data URIs\n\nIn case one would like to prevent bundler from making certain kinds of replacements,\nfor example if a data URI is too long or the resource is hosted on a particular site,\none can register hooks to be run when the collection of diffs has been compiled.\n\nCurrently, `bundler.filterDiffs` is the only existing hook exported\nfor this purpose. An example of filtering out resources that appear to be hosted\non `google.com`:\n\n```javascript\nvar bundleMaker = new bundler.Bundler(url);\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\nbundleMaker.on('originalReceived', bundler.replaceCSSFiles);\n\nbundleMaker.on('diffsReceived', bundler.filterDiffs(function (src, dest) {\n  return src.indexOf('google.com') \u003c 0;\n}));\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(bundle);\n});\n```\n\n# Writing your own hooks\n\nIt is, of course, possible to write your own hooks to use in any of the cases outlined\nabove. Each type of hook has a different signature but are expected to behave in\napproximately the same ways.\n\n## Before fetching the original document\n\nHooks to be added to the Bundler object using the `originalRequest` event \nshould have the following form:\n\n```javascript\nfunction handlerName(options, callback) {\n  // Do something with options\n  callback(err, options);\n}\n```\n\nThe `callback` provided is used to iterate through hooks, and so must be called\nwith any error that might occur (or `null` otherwise) and the modified `options`\nobject.\n\nThe `options` object is the object passed to the [request](https://github.com/request/request)\nlibrary as the first argument to `request` as seen in the library's\n[Custom HTTP Headers](https://github.com/request/request#custom-http-headers)\ndocumentation.\n\nFor example, we could register the following hook to increment a global count of the total\nnumber of bundle requests the server has received.\n\n```javascript\nvar bundlerCalls = 0;\n\nvar bundleMaker = new bundler.Bundler(url);\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\n\nbundleMaker.on('originalRequest', function (options, callback) {\n  bundlerCalls++;\n  callback(null, options);\n});\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(bundle);\n});\n```\n\n## Handling resources\n\nHandlers for replacing resources in a document, like `bundler.replaceImages` can also be supplied to the bundler.  Such functions have the following form.\n\n```javascript\nfunction handlerName(request, originalDoc, url, callback) {\n  var resourceURL = findResourceURL(originalDoc);\n  request({url: resourceURL}, function (err, response, body) {\n    // produce a diff object\n    var diff = { 'source-url': 'replacement' };\n    callback(errorIfAny, diff);\n  });\n}\n```\n\nThe `request` parameter is a wrapper around the [request](https://github.com/request) function that will invoke all hooks inserted via `resourceRequest`\nand `resourceRetrieved`\nto modify the `options` object and to produce diffs for the resource before and after making the request. It accepts an  \n`options` parameter (or resource URL) and a callback to handle the response.\n\nThe `originalDoc` parameter contains the document fetched by the original request.\n\nThe `url` parameter is the URL originally requested, used to produce resolved\npaths to discovered resources.\n\nThe `callback` parameter is used to iterate through a call to `async.reduce` and must be invoked with any error that occurs (or null) and the diff object\ncreated.\n\nSuch handlers tend to become quite complicated quickly as multiple requests will need to be made for resources.  In the bundler library, `async.reduce` is\nused to build the diff object.\n\n## Before fetching each resource\n\nHooks to be added to the Bundler object using the `resourceRequest` event \nshould have the following form:\n\n```javascript\nfunction handlerName(options, callback, originalDocument, response) {\n  // Do something with options\n  callback(err, options);\n}\n```\n\nYou can reuse hooks for the `originalRequest` event here.\n\nThe `options` and `callback` arguments here are the same as they are for the\n`originalRequest` handlers.\n\nThe `originalDocument` argument here contains the content of the originally\nfetched document.\n\nThe `response` argument is the response object provided by the call to `request` \nfor the original document, which is an instance of\n[http.IncomingMessage](http://nodejs.org/api/http.html#http_http_incomingmessage).\nThis may be useful for obtaining response headers and the status code.\n\nFor example, you could set the Referer header of the resource request to the\nvalue of the Host header in the response.\n\n```javascript\nvar bundleMaker = new bundler.Bundler(url);\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\n\nbundleMaker.on('resourceRequest', function (options, callback, doc, response) {\n  if (!options.hasOwnProperty('headers')) {\n    options.headers = {};\n  }\n  options.headers['Referer'] = response.headers['host'];\n  callback(null, options);\n});\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(bundle);\n});\n```\n\n## After retrieving each resource\n\nBundler allows hooks to be registered to directly manipulate the body of a fetched resource through the `resourceReceived` event.  Such hooks have\nthe following signature.\n\n```javascript\nfunction handlerName(requestFn, options, body, diffs, response, callback) {\n  // Make a diff object for the resource body\n  callback(err, diff);\n}\n```\n\n`requestFn` is a wrapper around the `request` library's exported function and can be used to fetch resources.\n\n`options` is the options object passed to the request made to fetch the resource for which the `resourceReceived` event was triggered.\n\n`body` contains the string contents of the resource in question.\n\n`diffs` is a diff object containing the diffs for the resource assembled by previously-invoked hooks.\n\n`response` contains the response object corresponding to the request for the resource in question.\n\n`callback` is the `async.reduce` callback and must be invoked with any error that might have occurred (or null) and the new diff object to pass onto the next hook.\n\n## After building data URIs\n\nHooks to be added to the Bundler object using the `diffsReceived` event \nshould have the following form.\n\n```javascript\nfunction handlerName(diffs, callback) {\n  // Do something with diffs\n  callback(err, diffs);\n}\n```\n\nOne could write a handler to count the number of images, CSS files, and JS files\nhaving replacements made with the following hook.\n\n```javascript\nvar cssReplaces = 0;\nvar jsReplaces = 0;\nvar imgReplaces = 0;\n\nvar bundleMaker = new bundler.Bundler(url);\n\nbundleMaker.on('originalReceived', bundler.replaceImages);\nbundleMaker.on('originalReceived', bundler.replaceCSSFiles);\nbundleMaker.on('originalReceived', bundler.replaceJSFiles);\n\nbundleMaker.on('diffsReceived', function (diffs, callback) {\n  var sources = Object.keys(diffs);\n  for (var i = 0, len = sources.length; i \u003c len; ++i) {\n    switch (bundler.mimetype(sources[i])) {\n    case 'text/css':\n      cssReplaces++;\n      break;\n    case 'application/javascript':\n      jsReplaces++;\n      break;\n    default:\n      imgReplaces++;\n    }\n  }\n  callback(null, diffs);\n});\n\nbundleMaker.bundle(function (err, bundle) {\n  console.log(cssReplaces + '\\t CSS files replaced.');\n  console.log(jsReplaces + '\\t JS files replaced.');\n  console.log(imgReplaces + '\\t Image files replaced.');\n});\n```\n\n# Helper functions\n\nTo make writing handlers and hooks a little bit easier, Bundler exports the\nfollowing functions.\n\n## bundler.mimetype(url)\n\nInfers, where possible, the mimetype of a resource based on its URL.  It is better to determine this information from the Content-Type header in a response, however this function is provided as a useful helper.\n\n## bundler.dataURI(response, baseURL, content)\n\nProduces the data URI for a resource given the response object corresponding to the request for the resource, the base URL (e.g. www.google.com) for the resource, and the content of the resource as a [Buffer obect](http://nodejs.org/api/buffer.html).\n\n## bundler.strReplaceAll(string, str1, str2)\n\nReplaces all instances of `str1` in `string` with `str2`.\n\n## bundler.applyDiffs(string, diffs)\n\nApplies all the replacements provided by a diff object to a given string.\nFor example, the string `\"abc\"` with diff `{'c': 'd'}` would become `\"abd\"`.\n\n## bundler.htmlFinder(source, selector, attr)\n\nReturns a function that accepts a callback. Will scan through an HTML document \n`source` and invoke the callback with the value of the attribute of each element obtained using the provided selector. This works using the \n[Cheerio](https://github.com/cheeriojs/cheerio) library, so selectors should be supplied accordingly.\n\n## bundler.cssReferenceFinder(source)\n\nLike `htmlFinder`, will return a function that accepts a callback.  The callback will be invoked with the URL found within all instances of calls to \n`url()` in the CSS document `source` provided.\n\n## bundler.replaceAll(request, url, finder, callback)\n\nUses a finder (the callback-accepting function provided by a call to either `htmlFinder` or `cssReferenceFinder`) to identify and request resources. The `url` argument must be the URL of the original document (e.g. www.google.com). The callback will be invoked with any error that occurs (or null) and a diff object.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalitie%2Fbundler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fequalitie%2Fbundler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fequalitie%2Fbundler/lists"}