{"id":13471164,"url":"https://github.com/WebReflection/tiny-cdn","last_synced_at":"2025-03-26T13:30:52.396Z","repository":{"id":35763310,"uuid":"40042792","full_name":"WebReflection/tiny-cdn","owner":"WebReflection","description":"A tiny static files serving handler","archived":true,"fork":false,"pushed_at":"2020-04-24T07:26:45.000Z","size":110,"stargazers_count":38,"open_issues_count":0,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-16T06:43:08.643Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WebReflection.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-01T10:04:41.000Z","updated_at":"2024-09-02T13:17:53.000Z","dependencies_parsed_at":"2022-09-16T11:32:22.837Z","dependency_job_id":null,"html_url":"https://github.com/WebReflection/tiny-cdn","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WebReflection%2Ftiny-cdn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WebReflection%2Ftiny-cdn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WebReflection%2Ftiny-cdn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WebReflection%2Ftiny-cdn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WebReflection","download_url":"https://codeload.github.com/WebReflection/tiny-cdn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245662756,"owners_count":20652075,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:00:40.840Z","updated_at":"2025-03-26T13:30:52.020Z","avatar_url":"https://github.com/WebReflection.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","others"],"sub_categories":[],"readme":"# Deprecated\n\nCheck the new, 100% code covered, **[µcdn](https://github.com/WebReflection/ucdn#readme)** instead 👍\n\n## tiny-cdn [![build status](https://secure.travis-ci.org/WebReflection/tiny-cdn.svg)](http://travis-ci.org/WebReflection/tiny-cdn)\n\nA tiny static files serving handler\n\n\n### About\nThis module has been developed on top of [holdon](https://github.com/WebReflection/holdon) utility.\n\nWhile this script is the fastest on Internet of Things devices such Arduino Yun or Raspberry PI, it has also been tested on regular servers and it has demonstrated to be both reliable and on average 2X or more faster than common nodejs solutions.\n\n\n\n#### How to install\n```\nnpm install -g tiny-cdn\n```\n\n\n## The Module API\nUsed as module, `tiny-cdn` exports a function with a public `.create` method which is simply an alternative way to invoke the module.\n\nThe function will return a function usable as serving middle-ware.\n\n```js\nvar\n  tinyCDN = require('./tiny-cdn'),\n  siteCDN = tinyCDN({ ..configuration object ..})\n;\nrequire('http')\n  .createServer(function (request,  response) {\n    if (/^\\/(?:js|css|img|assets)/.test(request.url)) {\n      siteCDN(request,  response);\n    } else {\n      // dynamic content generation in here\n    }\n  })\n  .listen(8080, '0.0.0.0')\n;\n```\n\nIf the purpose of the machine is to serve static files only, this is also possible.\n```js\nrequire('http')\n  .createServer(require('./tiny-cdn').create({\n    // the configuration object\n  }))\n  .listen(8080, '0.0.0.0');\n```\n\n\n#### The Configuration Object\n`tinyCDN({...})` function accepts a configuration object.\nFollowing a description of all accepted properties:\n\n```js\nvar tinyCDN = require('./tiny-cdn').create({\n\n\n  // where are original files ?\n  // this should be the root of client requests\n  // as example, /js/main.js should be a file\n  // originally saved in ./source/js/main.js\n  // the folder does not have to be publicly reachable\n  source: './source',\n\n\n  // where should ETags and Compressed files be stored ?\n  // this folder will be populated with respective etags\n  // per each requested file and their gzip or deflate version\n  dest: './dest',\n\n\n  // optional, if specified will create\n  // gzip and deflate version of every request\n  // in case the requesting UA is compatible.\n  //\n  // If specified as string,\n  // it can be one of these values:\n  //    best     // best compression\n  //    speed    // fastest compression\n  //    default  // default compression\n  //    no       // no compression ( kinda pointeless )\n  //\n  // If specified as number,\n  // it could be an integer from 1 to 9\n  // that will mirror the compression level\n  // accordingly with current zlib API compression values\n  compression: 'best',\n\n  // optional, if specified will activate compression\n  // and it will perform it only for the list of known extensions\n  // such list can be passed as Array of extensions\n  // with values like 'txt' or '.css'\n  // by default, if compression is enabled,\n  // the list of known files will be the following one\n  compress: [\n    'js', 'css', 'txt', 'html',\n    'svg', 'md', 'htm', 'xml', 'json', 'yml'\n  ],\n\n  // optional, if specified will create\n  // an etag related file per each request kind\n  // (raw, gzip, deflate)\n  // it can be any valid crypt hash\n  // and by default it will be the sha256 one\n  etag: 'sha256',\n\n\n  // optional, if specified will limit\n  // the amount of connection that cpould be piped\n  // as streams when any file is served\n  maxListeners: Infinity, // as deafault\n\n\n  // optional, if specified will define\n  // the masimum age any  ETag should last\n  maxAge: 30672000, // as deafault\n\n\n  // optional, if set every file will send\n  // the Access-Control-Allow-Origin header\n  // with the specified value\n  accessControlAllowOrigin: '', // as deafault\n\n\n  // if cluster is used and you'd like\n  // to not use optimizations made for multiple workers\n  // feel free to force this flag to true (false by default)\n  ignoreCluster: false, // as deafault\n\n  // optional, if specified will define\n  // the Not Found content per each page\n  // served accordingly with common readind standards\n  404: {\n\n    // either html or HTML, could be any HTML page\n    html: 'Not Found',\n    // or\n    HTML: 'Not Found',\n\n    // either json or JSON, could be any JSON\n    json: '{\"error\":\"Not found\"}',\n    // or\n    JSON: '{\"error\":\"Not found\"}',\n\n    // either txt or TXT, could be any TXT\n    txt: 'Not Found',\n    // or\n    TXT: 'Not Found'\n  },\n\n\n  // optional, if specified will be invoked\n  // every time an erro occurs.\n  // by default tinyCDN will take care of everything\n  onError: function (\n    // the error that triggered this callback\n    err,\n    // the requested url that generated the error\n    url,\n    // the list of requestes waiting for a response.\n    // do not use an error handler if you don't have\n    // a good way to serve failures.\n    // Use simply `onResponse` handler instead\n    // to be notified, if that's what you are looking for\n    held\n  ) {\n    // do your stuff here ...\n  },\n\n\n  // optional, if specified will be invoked\n  // every time a request completes.\n  // It will eventually provide the error object\n  // and the url as second argument.\n  // But there won't be anything to do once this triggers\n  onResponse: function (\n    // the error, if any, that happened while serving\n    err,\n    // the url that successfully, eventually, got delivered\n    url\n  ) {\n    // do your stuff here ...\n  }\n});\n```\n\n## The CLI API\nIf used directly as executable, `tinyCDN` is capable of creating an *http* or *https* server, pre-build a file and its compressed versions, or generate a configuration JSON/object content to read and test.\n\n```\n./tiny-cdn [run] [options]\n\nMandatory options\n\n  -s | --source           the source folder with static content\n  -d | --dest             the destination folder for ETags / gzip\n\nConfiguration options\n\n  -c | --compression      use compression and optionally specify its level\n                          best, speed, default, or an integer\n  -cl | --compress-list   a comma separated list of extension to compress\n  -e | --etag             use etag and optionally specify its algorithm\n                          by default it's sha256\n  -ma | --max-age         the cache max-age header in seconds (default: 30672000)\n  -ml | --max-listeners   the maximum amount of listeners to use per each stream\n                          by default there is no limit\n  -ic | --ignore-cluster  if true (or empty) will never use master/cluster logic\n\nSSL options\n\n  -ssl-cert               will use the provided cert file to run HTTPS instead of HTTP\n  -ssl-key                will use the provided key file to run HTTPS instead of HTTP\n\nNetwork options\n\n  -h | --host | -ip       if specified, will be used as server address\n  -p | --port             if specified, will be used as port\n\nBuild file option\n  -b | --build            if specified will pre build a file\n                          creating in the destination folder\n                          the gzip and deflate version of the file\n                          plus the ETag per each version\n```\n\n#### run examples\n\nThis is the minimum requirement to run an http server:\n\n```sh\ntiny-cdn run\n```\nAbove snippet will start `tinyCDN` static file serving in the current folder without creating etags or compressed files.\nHandy to quickly test some directory content statically.\n\nPlease note that point at a generic `/folder/` will automatically check for an `index.html` file within that folder.\n\nReaching the default shown url will also look for an `index.html` file and  will return a Not Found error if none is provided.\n\n\nIt is possible to specify both source and destination though:\n```sh\ntiny-cdn run -s=./source -d=./dest\n```\n\nTo use compression and etags, we can add related flags too:\n```sh\ntiny-cdn run -s=./source -d=./dest -e -c\n```\n\nTo use a specific version of the etag or compression\n```sh\ntiny-cdn run -s=./source -d=./dest -e=md5 -c=speed\n```\n\nTo specify a different host or port\n```sh\ntiny-cdn run -s=./source -d=./dest -h=192.168.1.10 -p=4321 -e -c\n```\n\n#### build examples\nThe build flag will create only what will be served to every request accordingly with the configuration  options.\n\nFor instannce, assuming we have `source/js/main.js` file in the source folder,\nthe following command will do basically nothing:\n```sh\ntiny-cdn -s=./source -d=./dest -b=/js/main.js\n```\nbecause neither etag nor compression flag is required.\n\nHowever, if we use the ETag flag:\n```sh\ntiny-cdn -s=./source -d=./dest -b=/js/main.js -e\n```\nthe builder will create the file `./dest/js/main.js.raw.sha256` with its ETag content.\n\nUsing the compression flag will create both gzip and  deflate version of the file.\n```sh\ntiny-cdn -s=./source -d=./dest -b=/js/main.js -c\n```\nNow we'll have also  `./dest/js/main.js.raw.gzip` and `./dest/js/main.js.raw.deflate` files.\n\nPutting everything together will have also `./dest/js/main.js.raw.gzip.sha256` and `./dest/js/main.js.raw.deflate.sha256` files generated.\n```sh\ntiny-cdn -s=./source -d=./dest -b=/js/main.js -c -e\n```\n\n### Running tinyCDN through SSL\nIf the `-ssl-cert=file.crt` and the `-ssl-key=file.key` are provided, the CLI will create a **https** server instead.\nPlease note that if you created the certificate using your local network IP address, you might need to specify it as `-h=192.168.1.5` in order to have a link in  console that will point to the authorized page.\n\nOtherwise you can always write down manually that IP address when visiting the CDN.\n\nIf you have no idea how to create a certificate, feel free to [read this \"How To\" page](https://www.webreflection.co.uk/blog/2015/08/08/bringing-ssl-to-your-private-network).\n\nFollowing an example on how to run SSL and HTTPS server\n```bash\n./tiny-cdn run \\\n  -s=source/ \\\n  -ssl-key=~/.server/192.168.1.5.key \\\n  -ssl-cert=~/test/https/192.168.1.5.crt \\\n  -h=192.168.1.5\n\n# will log\n# [tinyCDN] running on https://192.168.1.5:7151/\n```\n\n\n#### configuration example\nIn case you'd like to automatically generate a configuration file for your CDN, or simply read how it looks like, skip `run` and build options and just execute the cli\n```sh\ntiny-cdn -s=./source -d=./dest -c -e\n\n/* [tinyCDN] */{\n  \"source\": \"/home/yourname/cdn/source\",\n  \"dest\": \"/home/yourname/cdn/dest\",\n  \"compression\": \"best\",\n  \"etag\": \"sha256\"\n}\n\n```\nIt is helpful to also understand defaults and verify folders.\n\n\n\n### tinyCDN and cluster\nThe main reason tinyCDN is 2X faster is disk access which,\neven if executed in an  asynchronous non-blocking way,\nis usually repeated per each request.\n\nAll common operations like file stats access, runtime etag or gzip compression,\nand even piping the output, are performed through a single-request model,\nwhich will serve every other connection that meanwhile asked for the same file.\n\nThis is a great model specially on Internet of Things devices, where every\ndisk operation might have a huge cost, specially the writing one.\n\nWhen the `cluster` module is in place, we can use every available core to serve our CDN.\n\nThis is great, and it works already out of the box with [common solutions](benchmark/express-multi),\nhowever there could be cases where a single file that should be compressed at runtime,\nwill be asked potentially from different workers.\n\nWhat usually happens is that each worker will perform the same operation  if the file wasn't there,\nunable to know another worker maybe is going to create such file.\n\nDue to the nature of cluster and workers, the holdon module (as it is) cannot coordinate\nin a single core requests from every CPU, so here it comes [the converger](cli/tiny-cdn-converger.js),\nwhich role is to be a cluster director.\n\n#### The Converger Utility\nThis helper does basically one thing: it forces asynchronous methods to be **execute only in Master**,\nproviding a regular JS looking interface that is transparent to human  eyes.\n```js\nvar\n  cluster = require('cluster'),\n  fs = require('fs'),\n  numCPUs = require('os').cpus().length,\n  Converger = require('tiny-cdn').Converger,\n\n  // it should be defined no matter if this is master or worker\n  // it will perform in master anyway\n  master = new Converger({\n    readFile: function (path, onFileRead) {\n      // just to be sure we are reading from master\n      console.log('reading from ' +\n        (cluster.isMaster ? 'Master' : 'Worker'));\n      fs.readFile(path, function (err, file) {\n        onFileRead(err, err || file.toString());\n      });\n    }\n  })\n;\n\n// if we have one CPU there's no  point to fork this process\nif (cluster.isMaster \u0026\u0026 1 \u003c numCPUs) {\n  while (numCPUs--) cluster.fork();\n} else {\n  // but even with a single CPU we can use Converger as it is\n  master.readFile(__filename, function (err, content) {\n    // however, if we have multiple cores\n    // we will receive data per each of them\n    console.log(\n      'Read file with length ' + content.length +\n      (cluster.isMaster ? '' :\n        (' [worker ' + cluster.worker.id + ']'))\n    );\n  });\n}\n```\n\nIf we run [above example](test/example.converger.js) we'll see this kind of output:\n```\nreading from Master\nreading from Master\nreading from Master\nreading from Master\nRead file with length 1098 [worker 2]\nRead file with length 1098 [worker 1]\nRead file with length 1098 [worker 3]\nRead file with length 1098 [worker 4]\n```\n\nWhich means the `Converger` instance worked as expected, but we have 4 workers asking for the same file so that\nin order to satisfy all of them the system will access and read from scratch each time.\n\nAdding [holdon](https://github.com/WebReflection/holdon#holdon) to this equation will result in less reading operations.\n```js\n// ...\n\n  // let's hold on readFile\n  heldFile = cluster.isMaster ?\n    // since we'd like to do this on master only ...\n    require('holdon').create(['callback']) : null,\n\n  // it should be defined no matter if this is master or worker\n  // it will perform in master anyway\n  master = new Converger({\n    readFile: function (path, onFileRead) {\n      // absolute path is a unique identifier\n      if (heldFile.add(path, onFileRead)) {\n        // only if added as new record\n        console.log('reading from ' +\n          (cluster.isMaster ? 'Master' : 'Worker'));\n        fs.readFile(path, function (err, file) {\n          // per each worker waiting for a result\n          heldFile.remove(path).callback.forEach(\n            function (callback) {\n              // send already processed arguments\n              callback.apply(null, this);\n            },\n            [err, err || file.toString()]\n          );\n        });\n      }\n    }\n  })\n\n// ...\n```\n\nAccordingly with your machine performance, your output should be now like the following one:\n```\nreading from Master\nRead file with length 1603 [worker 1]\nRead file with length 1603 [worker 2]\nRead file with length 1603 [worker 3]\nRead file with length 1603 [worker 4]\n```\nWhere `reading from Master` could happen twice and the workers order could be completely different.\n```\nreading from Master\nreading from Master\nRead file with length 1603 [worker 2]\nRead file with length 1603 [worker 3]\nRead file with length 1603 [worker 1]\nRead file with length 1603 [worker 4]\n```\nIn latter case two workers asked in different reading time the same file ... which is OK, it means my machine was that fast (cache).\n\n\n#### Converger Performance\n\nWhile this might not look like a huge win on the reading path,\nthis might actually save your Raspberry PI, as well as your server,\nfor all cases when you need some runtime operation like compressing\nand storing a file or creating its etag.\n\nThose kind of slow operation that specially on SD cards might take \"forever\".\n\nThe **writing** is indeed a wonderful fit for `Converger` which is, together with `holdon` module,\ncapable of bringing in a very good performance compromise even for IoT projects.\n\n**However**, the serialization channel can be very slow,\nand serving files entirely read on a single thread broadcasted\nper each worker becomes a bottleneck,  rather than a feature.\n\nAccordingly, beside little content read,  how an ETag could be,\nor some little information, as a `fs.Stats` object could contain,\nare a good fit for `Converger` pattern but every other reading should be avoided.\n\nThe `tinyCDN` strategy when it comes to file serving is to pipe all requests at once\nwhen the read stream is ready, and the etag has been retrieved.\n\nThe compression, and the etag generation, is done only once, holding every request.\n\nI also need to better benchmark the get stats path and see if it makes sense to\npass through the channel, or if I could simply use the thread for every read-only opration.\n\n\n\n### License\n```\nCopyright (C) 2015 by Andrea Giammarchi - @WebReflection\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWebReflection%2Ftiny-cdn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FWebReflection%2Ftiny-cdn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWebReflection%2Ftiny-cdn/lists"}