Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/bahamas10/node-fs-caching-server
A caching HTTP server/proxy that stores data on the local filesystem
https://github.com/bahamas10/node-fs-caching-server
Last synced: about 2 months ago
JSON representation
A caching HTTP server/proxy that stores data on the local filesystem
- Host: GitHub
- URL: https://github.com/bahamas10/node-fs-caching-server
- Owner: bahamas10
- Created: 2015-05-17T00:39:39.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2021-04-13T17:28:42.000Z (over 3 years ago)
- Last Synced: 2024-10-30T10:45:14.835Z (about 2 months ago)
- Language: JavaScript
- Size: 20.5 KB
- Stars: 5
- Watchers: 3
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
fs-caching-server
=================
A caching HTTP server/proxy that stores data on the local filesystem
Installation
------------
[sudo] npm install [-g] fs-caching-server
`v1.0.0` Release Notes
----------------------
`v1.0.0` adds the following changes as well as bug fixes.
- Fixes HEAD before GET caching - old behavior would cache 0-byte files
- Handles redirects (or more accurately, doesn't handle - just proxies them)
- Can retrieve from an HTTPS backend URL
- Tests! there were none before - now a lot of tests have been added to ensure functionality
- Can be used as a module (this added mostly for testing)
- Cache dir can be specified as an argument/env variable - CWD not required anymore
- Access logs now contain debug UUID if debug is specified
CLI
---
### Description
The `fs-caching-server` program installed can be used to spin up an HTTP server
that acts a proxy to any other HTTP(s) server - with the added ability to
cache GET and HEAD requests that match a given regex.
### Example
This will create a caching proxy that fronts Joyent's pkgsrc servers
$ mkdir cache
$ fs-caching-server -c cache/ -d -U https://pkgsrc.joyent.com
listening on http://0.0.0.0:8080
proxying requests to https://pkgsrc.joyent.com
caching matches of /\.(png|jpg|jpeg|css|html|js|tar|tgz|tar\.gz)$/
caching to /home/dave/dev/fs-caching-server/cache
`-d` enables debug output which can be used to determine if a file was a cache
hit, cache miss, or skipped the cache completely. For example, we can request
a file twice to see that it will be proxied and downloaded the first time, and
the second time it will just be streamed from the local cache.
[58b93965-b7de-4669-9cb1-aff39e16a4fb] INCOMING REQUEST - GET /packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz
[58b93965-b7de-4669-9cb1-aff39e16a4fb] proxying GET to http://pkgsrc.joyent.com/packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz
[58b93965-b7de-4669-9cb1-aff39e16a4fb] saving local file to ./packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz.in-progress
10.0.1.35 - - [16/May/2015:20:31:39 -0400] "GET /packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz HTTP/1.1" 200 12432 "-" "libfetch/2.0"
[58b93965-b7de-4669-9cb1-aff39e16a4fb] renamed ./packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz.in-progress to ./packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz
...
[ff8a1519-597f-4f9a-a999-bd05677896c2] INCOMING REQUEST - GET /packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz
[ff8a1519-597f-4f9a-a999-bd05677896c2] ./packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz is a file (cached) - streaming to client
10.0.1.35 - - [16/May/2015:20:32:48 -0400] "GET /packages/SmartOS/2014Q4/x86_64/All/watch-3.2.6nb1.tgz HTTP/1.1" 200 12432 "-" "curl/7.39.0"
The lines that begin with `[]` are only printed when debug (`-d`) is
enabled - each UUID represents a unique incoming request. The first request
shows the file was proxied to pkgsrc.joyent.com and streamed to both the client
and the local filesystem. The second request shows the file was already
present so it was streamed to the client without ever reaching out to
pkgsrc.joyent.com.
### Usage
$ fs-caching-server -h
usage: fs-caching-server [options]
options
-c, --cache-dir [env FS_CACHE_DIR] directory to use for caching data, defaults to CWD
-d, --debug enable debug logging to stderr
-H, --host [env FS_CACHE_HOST] the host on which to listen, defaults to 0.0.0.0
-h, --help print this message and exit
-p, --port [env FS_CACHE_PORT] the port on which to listen, defaults to 8080
-r, --regex [env FS_CACHE_REGEX] regex to match to cache files, defaults to /\.(png|jpg|jpeg|css|html|js|tar|tgz|tar\.gz)$/
-U, --url [env FS_CACHE_URL] URL to proxy to
-u, --updates check npm for available updates
-v, --version print the version number and exit
Module
------
### Description
This module can also be used as a JavaScript module.
### Example
``` js
var FsCachingServer = require('fs-caching-server').FsCachingServer;
// proxy to Joyent's pkgsrc
var opts = {
cacheDir: '/home/dave/cache-dir',
host: '0.0.0.0',
port: 8080,
backendUrl: 'https://pkgsrc.joyent.com'
};
var cachingServer = new FsCachingServer(opts);
cachingServer.once('start', function () {
console.log('server started');
});
if (process.env.NODE_DEBUG) {
// debug messages go to stderr
cachingServer.on('log', console.error);
}
// access log messages to stdout
cachingServer.on('access-log', console.log);
cachingServer.start();
```
### Usage
``` js
/*
* FsCachingServer
*
* Create an instance of an FS Caching Server
*
* Aurguments
* opts Object
* opts.host String (Required) Host to bind to. ex: '0.0.0.0',
* '127.0.0.1', etc.
* opts.port Number (Required) Port to bind to. ex: 80, 8080, etc.
* opts.backendUrl String (Required) URL of the backend to proxy
* requests to. ex:
* 'http://1.2.3.4:5678'
* opts.cacheDir String (Required) Directory for the cached items. ex:
* '/tmp/fs-caching-server'
* opts.regex RegExp (Optional) Regex to match to enable caching,
* defaults to REGEX above.
* opts.noProxyHeaders Array (Optional) An array of headers to not proxy to
* the backend, default is [date,
* server, host].
* opts.cacheMethods Array (Optional) An array of methods to proxy,
* default is [GET, HEAD].
*
* Methods
*
* .start()
* - Start the server.
*
* .stop()
* - Stop the server.
*
* .onIdle(cb)
* - Call the callback when the caching server is "idle" (see events below).
*
* Events
*
* 'start'
* - Called when the listener is started.
*
* 'stop'
* - Called when the listener is stopped.
*
* 'access-log'
* - Called per-request with a CLF-formatted apache log style string.
*
* 'log'
* - Called with debug logs from the server - useful for debugging.
*
* 'idle'
* - Called when the server is idle. "idle" does not mean there are not
* pending web requests, but instead means there are no pending filesystem
* actions remaining. This is useful for writing automated tests.
*/
```
Testing
-------
```
$ NODE_DEBUG=1 npm test
> [email protected] test /home/dave/dev/node-fs-caching-server
> ./node_modules/tape/bin/tape tests/*.js
TAP version 13
# start cachingServer
ok 1 tmp dir "/home/dave/dev/node-fs-caching-server/tests/tmp" cleared
starting server
listening on http://127.0.0.1:8081
proxying requests to http://127.0.0.1:8080
caching matches of /\.(png|jpg|jpeg|css|html|js|tar|tgz|tar\.gz)$/
caching to /home/dave/dev/node-fs-caching-server/tests/tmp
ok 2 cachingServer started
# start backendServer
ok 3 backendServer started on http://127.0.0.1:8080
# simple cached request
[63e44996-ac28-4a0f-b306-61aeeb88b53c] INCOMING REQUEST - GET /hello.png
[63e44996-ac28-4a0f-b306-61aeeb88b53c] proxying GET to http://127.0.0.1:8080/hello.png
[63e44996-ac28-4a0f-b306-61aeeb88b53c] saving local file to /home/dave/dev/node-fs-caching-server/tests/tmp/hello.png.in-progress
[63e44996-ac28-4a0f-b306-61aeeb88b53c] 127.0.0.1 - - [13/Mar/2021:20:58:52 -0500] "GET /hello.png HTTP/1.1" 200 48 "-" "-"
...
...
...
ok 50 backendServer closed
# stop cachingServer
ok 51 cachingServer stopped
1..51
# tests 51
# pass 51
# ok
```
License
-------
MIT License