Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/scripting/pagePark

I wrote this simple Node.js folder-based HTTP server that serves static and dynamic pages for domains I've bought but not yet used. Then I kept going...
https://github.com/scripting/pagePark

Last synced: 5 days ago
JSON representation

I wrote this simple Node.js folder-based HTTP server that serves static and dynamic pages for domains I've bought but not yet used. Then I kept going...

Awesome Lists containing this project

README 

        
# PagePark 



I wrote  this simple HTTP server to park domains I've bought but not yet used. 



Then I kept going and added all the features I want to help me manage my own websites, far beyond just parking them. I liked the name so I kept it. Think of it as a nice park where you keep your pages. ;-)



It's written in JavaScript and runs in Node.js.



Each domain is in its own folder. The content for that domain is in the folder. It can serve Markdown docs, or optionally run JavaScript code. And of course HTML, text files, images, movies, etc. 



Yet it's still very simple. Which is the point. ;-)



It's 90 percent of what all web servers do, so if you learn how to run PagePark, you're learning how to run a web server. A real one you can use to host your sites. And it's easy to hack the code if you want to.



### How to



How to get started quickly with PagePark.



0. Install Node.js if you don't already have it running, including the NPM package manager.



1. Download the PagePark folder from the GitHub repository.



2. At the command line enter npm install.



3. To run the server, enter node pagepark.js.



4. Create a sub-folder of the domains folder called localhost. 



5. Using a text editor add a file to the localhost folder called index.md. Put whatever you like in the file. Save it.



6. In a browser, running on the same machine, enter localhost:1339. You should see the text you entered in the previous step. 



If this worked, congratulations -- you just installed a web server. :-)



### Screen shot



Here's a screen shot of an example PagePark domains folder. 



### How it works



1. PagePark will automatically create a *prefs* sub-folder and a *domains* sub-folder. 



2. Add your web content under domains. Each folder's name is the name of a domain. The contents within the folder are what we serve. Screen shot.



3. Serves all major media types including audio and video. Files whose names end with .md are passed through the built-in Markdown processor. Files ending with .js are interpreted as scripts only if the associated config setting is turned on. The text they return is what we serve. Here's an example of a script that I have running on one of my servers.



4. The prefs folder contains a file of settings you can change, prefs.json. These include the port that the server runs on and the name of the index file (see below).



5. stats.json contains information generated by the server including the number of times the server has started, how many hits it's received (all time and today), and hits by domain.



6. mdTemplate.txt is the template we use to serve Markdown text. You can edit this file to provide a common template for all your Markdown documents.



7. If a request comes in for a folder, we scan the folder for a file whose name begins with *index* and serve the first one we find. So the index file can be HTML, Markdown or a script, or any other type PagePark can serve. 



8. If you want to run PagePark from a folder different from the one that contains the app, set the *pageparkFolderPath* environment variable to point to that folder. 



9. There are three special endpoints on all domains: /version, /now and /status that return the version of PagePark that's running, the time on the server and the stats and prefs. 



### File extensions



The extension of a file determines how PagePark serves it.



.txt



The text in the file is returned, the type is text/plain.



.xml



The text in the file is returned, the type is text/xml.



davetwitterfeed.xml



.json



The text in the file is returned, the type is application/json.



wodemo.json



.png



The contents of the file is returned with type image/png.



tree.png



.opml



The outline is rendered as an expandable outline, the type returned is text/html.



states.opml



.md



The text in the file is passed through a Markdown processor and that text is returned. The type returned is text/html.



test.md



.js



We run the script, and the return value is returned to the caller, with type of text/html. Here's the source code for the script in the demo below.



butt.js



### Port 1339



The first time you run PagePark it will open on port 1339. You can change this by editing prefs.json in the prefs folder. 



This means if you want to access a page on your site, the URL will be of the form:



http://myserver.com:1339/somepage.html



The normal port for HTTP is 80. That would have been the natural default, however a lot of Unix servers require the app to be running in supervisor mode in order for it to open on port 80. You can do this by launching PagePark this way:



sudo node pagepark.js



I made the default 1339 because I wanted it to work "out of the box" for first-time users.



### Mapping port 80 to 1339



Here's a magic incantation that works on Ubuntu that maps requests for port 80 to port 1339.



sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 1339



sudo iptables -t nat -A OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 1339



If you have questions about how and why this is necessary, see this thread on the PagePark repo.



### Example pages



http://morningcoffeenotes.com/ -- simple home page



http://lucky.wtf/ -- images



http://lucky.wtf/test.md -- markdown page



http://lucky.wtf/badass/ -- index file in a sub-directory



http://lucky.wtf/nosuchfile.html -- file not found



http://lucky.wtf/version -- the version of PagePark that's running on the server



http://lucky.wtf/now -- the time on the server



http://lucky.wtf/transcend.opml -- a page written in OPML



http://lucky.wtf/transcend.opml?format=opml -- the OPML source of the page



### config.json files in domains folders



On every request, PagePark looks at the top level of the domain folder for a file named config.json. If it finds it, it reads it and the values in the file control how the request is handled. 



Here's a docs page that lists the values and what they control.



### JavaScript sample code



I've iterated over the code to try to make it good sample code for JavaScript projects. 



I wanted to make code that could be used for people who are just getting started with Node, to help make the process easier.



There will always be more work to do here. ;-)



### Updates



#### v0.8.27 9/22/22 by DW



There's a new version of the marked package that breaks pagepark. 



Changed package.json to use the last version that worked. 



#### v0.8.26 11/12/21 by DW



Added new call for apps running on the same machine as PagePark that makes it possible to find out if a particular domain his being served by PP.



The feature was discussed in this RFC thread. 



#### v0.8.25 8/25/21 by DW



urlGlossary in config.json.



#### v0.8.24 8/20/21 by DW



Templates for individual websites.



#### v0.8.23 7/28/21 by DW



config.flServeConfigJson -- a new config value -- defaults false. Possible breakage.



Updated the persistent apps docs. It previously said there was no way to launch a new app once PagePark has booted. This is no longer true, there is a command-line command that does a rescan of the folders, and launches any newly qualified apps. 



Updated the command line tool docs. A few commands were added since the docs were written. 



#### v0.8.22 7/25/21 by DW



We allow you to specify config.defaultExtension with a leading dot, for example ".opml" or ".md".



It's a common thing to do so why not make sense of it?



#### v0.8.21 7/25/21 by DW



New defaultExtension value in config.json.



#### v0.8.20 7/5/21 by DW



If a request comes in for a host that doesn't have a corresponding domains folder, and there is a folder called default, we serve the request from the default folder.



A wildcard folder, if it exists, takes precedence, for examle if a request comes in for yo.bullmancuso.com, and there is no domains folder with that name, and there is a wildcard folder, *.bullmancuso.com and a default folder, the request goes to the wildcard folder. 



You can put a config.json file in the default folder. 



#### v0.8.18 6/22/21 by DW



In previous versions, if you used the mirrors feature in config.json, we would read a file and return exactly what was in it. 



In this version, we read the file, and process it as if it were a local file. 



For example, if the file has a .md extension, we run it through the template for Markdown files.



If the file has a .opml extension, it is rendered as an outline. 



Here's a config.json you can put into a folder to test the feature. I have uploaded the files to the scripting.com publicfolder sub-folder. 



#### v0.8.10 5/10/20 by DW



Moved the script-running code into a new package, which I'm eventually going to move all the core functionality of PagePark into. The top level will just call into the package. Will allow code to be reused in other projects. Having a full-featured web server as a replacement for davehttp which is very bare-bones. 



Instead of baking my own package-calling code, I'm now using the require-from-string package. 



Added JS code that runs every second, minute, hour and overnight in the prefs/scripts folder. Example code is included. 



Changed the way plugins work. You no longer have to define a module and export a function named filter. It's just straight-line code. Same model as is used in the scripts code, above.



All this needs documentation wihich I will do. I want to give everything a chance to settle in and flesh out.



#### v0.8.7 4/18/20 by DW



Allow the port to be set by process.env.PORT. This is how Glitch says what port to run on. 



Only show status message once an hour as opposed to once a minute. Too much noise. Shhhh.



#### v0.8.6 4/11/20 by DW



Mirrors. 



A plugin can tell PagePark that it did npt handle the request and that it should go ahead and pass it through the rest of PagePark's serving logic by returning with a value in the httpResponse object for flNotHandled of true. 



#### v0.8.5 3/27/20 by DW



Plugins. 



#### v0.8.4 12/31/19 by DW



When serving a markdown document, look at the first 5 lines in the file for a line that begins with "# ". Assume that's the title of the page. Substitute for [%title%] in the template. A way to set the HTML title of the page served.



#### v0.8.1 12/13/19 by DW



If you make a request for a GitHub hosted page with a nocache=true parameter it won't use the cache to read the page. This allowed me to make the cache timeout pretty long, so if you're making a change to a page, be sure to call it at least once with this param to force it to be reloaded. 



#### v0.7.31 12/10/19 by DW



You can now override most of the values in the top level config.json file in any of the config.json files in individual domains. It would have worked this way in the beginning if site-level config settings were possible in the beginning. 



Updated the default markdown template to use the same CSS file that GitHub uses. 



#### v0.7.31 12/9/19 by DW



Hidden files and folders



If the file or folder name begins with . do not serve it. 



This feature can be turned off with config.flHiddenFilesCheck.



#### v0.7.31 12/4/19 by DW



flProcessScriptFiles defaults false. This is a breaking change. If you have pages that are implemented in JS you will have to switch this on for the site in question. 



Displaying pages of various types with the proper content-type header. 



There was a bug that I fixed, we were calling a local routine named getReturnType with the incorrect parameter. The bug was also present when serving content from S3. 



Display an index file for GitHub directories.



Added a section to the docs for config.json about serving content from GitHub. 



#### v0.7.26 6/8/17 by DW



We look for config.json in the home directory. It contains what used to be in pref/prefs.json. All my other apps work with config.json, I wanted PagePark to be consistent. If we don't find config.json we look for prefs/prefs.json, so that there's no breakage. 



#### v0.7.24 7/6/18 by DW



Added feature that lets you set the default type for files that don't have extensions. 



#### v0.7.21 6/6/18 by DW



Added feature that lets you redirect by modifying the contents of the file.



#### v0.7.19 6/4/18 by DW



Built a bridge to S3 with config.s3ServeFromPath. Now PagePark can serve from S3 locations exactly as if they were in the local file system. I plan to use this as a bridge to publicfolder.io. Updated docs for the three S3 configuration options. 



#### v0.7.18 5/12/18 by DW



Continue to have problems with the mime package. Switch to using the daveutils routine instead, does the same thing. One less dependency.



#### v0.7.17 5/10/18 by DW



At some point serving images broke, not sure when. The fix was to not convert the value returned when reading a file to a string. 



#### v0.7.11 2/17/18 by DW



PagePark was failing in httpRespond when filling out logInfo, saying it's undefined. Not clear how this could happen but I added defensive driving to make sure it can't happen, by initializing logInfo at the very start of the routine. 



#### v0.7.9 11/8/17 by DW



A new top-level config.json option, `flUnicasePaths`. 



The feature is documented. 



A blog post about the feature. 



#### v0.7.8 10/3/17 by DW



New features supports logging over WebSockets. To enable, in config.json set flWebsocketEnabled true, and assign a port to the WS server in websocketPort. On every hit, PagePark will send back a JSON structure containing information about the request. 



Note: There's a new dependency, nodejs-websocket, so when installing this update you have to do an `npm install` before running pagepark.js.



#### v0.7.7 9/26/17 by DW



Added two config.json options and changed the name of another. 



1. Changed the name of s3Path to fargoS3Path. It was only used in serving Fargo websites, on one of my servers. It's a very specific bit of functionality. Far more specific than the name implies.



2. New meaning for s3Path option. If specified it points to an S3 location that PagePark will serve the site from. It should begin with a slash but not end with one. All requests for the domain are mapped to requests at the S3 location. Additionally, it processes the request through the same filters it uses for local files, so files ending with .md are passed through the Markdown processor and files that end with .opml are processed as outlines. It follows all the rules outlined on this page.



3. New option: localPath, points to a directory on the local system where files are served from exactly as in #2 above. 



A caveat on this release, I had trouble upgrading one of my servers with this version because of changes to a couple of packages we use. I was able to fix all the problems by installing the latest stable version of Node, as described on this page. If you have trouble, re-open this issue, and we can look into it. 



#### v0.7.6 9/16/17 by DW



Two changes:



1. The mime package had a breaking change. I fixed it so it no longer breaks PagePark. Thanks to Davis Shaver for the report. 



2. Changed the version number system to use the same x.y.z system used by other Node apps. This is v0.7.6.



#### v0.75 7/17/17 by DW



Check stats every minute instead of every second. This means that servers that take a lot of hits won't be writing the stats file so often. If you run PagePark out of a shared folder this can mean a lot of updating, it's not really protecting against anything critical. 



#### v0.74 7/5/17 by DW



If a request comes in for domain.com/hello and it's a directory, redirect to domain.com/hello/.



#### v0.73 6/17/17 by DW



Replaced daveopml with opmltojs, a new package that builds on the xml2js package.



Rebuilt the OPML rendering code to use opmltojs. This allows the <head> info to be transmitted to the rendered page. And the full OPML structure is embedded in the page, meaning the page can render itself without calling back to the server to get missing bits from the OPML.



Added a new pref, flCacheTemplatesLocally, that allows you to say you don't want the OPML and Markdown templates cached. Add it to your prefs.json file. It defaults true, because that was the previous behavior (no breakage). This probably won't matter to people who aren't iterating over template development.



Made the URLs of the OPML and Markdown templates configurable through prefs.json. So if you want to do something nicer than I have, you can, without having to modify pagepark.js.



#### v0.72 6/7/17 by DW



Factored the local utils and opml modules, instead using the new daveutils and daveopml NPM packages. 



We now look for prefs in config.json in in the same folder as pagepark.js, if it's not present, we look for prefs.json in the prefs folder. All my other server software uses config.json, so PagePark fits in better.



Improved rendering outlines, so that you can put an OPML file anywhere, and refer to it and it will render as an outline. If you add ?format=opml to the URL it will return the XML 



#### v0.71 7/31/16 by DW



The repository now includes mdTemplate.txt in the prefs folder. Previously it would download this file from one of my sites the first time it was used. This is a more modern way to distribute it. 



#### v0.70 12/8/15 by DW



It used to be that a file had to exist in order for you to redirect from it in config.redirects. Now it doesn't have to exist. 



#### v0.68 8/31/15 by DW



Small change in error handling when we delegate a request. The previous method would cause PagePark to crash if the app we're trying to delegate to isn't running. Thanks to Dan MacTough for the help fixing this. ;-)



#### v0.67 7/30/15 by DW



New redirect feature for individual pages. 



In config.json for the domain containing the file you want to redirect, create a struct called redirects. The name of each element is the path to a file in the folder, and the value of each is the URL we will redirect to. On each request for that domain, we look in the redirects table to see if the request should be redirected. 



Here's an example of the config.json for smallpicture.com. It redirects from an old version of the outliner howto to the newer version. 



#### v0.66 7/19/15 by DW



In prefs.json a new value, legalPathChars, defaults to the empty string. In this string you can specify characters that are legal in paths on your server. We are very conservative in what we will allow in paths, but if you need to use one of the characters that we consider illegal, add it to this string. 



For example, I am redirecting urls from discuss.userland.com to a static archive. It was a Manila site, so it uses a $ in the URLs, a character which PagePark by default considers illegal. Here's an example. I set legalPathChars to "$" in prefs.json, and it lets the character through, and it is redirected by the very clever little script that handles redirection for that domain. 



#### v0.65 7/16/15 by DW



In prefs.json a new value, error404File, defaults to prefs/error.html



When there's an error, we read that file and send it back as the text of the 404 response.



The default value is more or less exactly the same text earlier versions of PagePark returned. It's actually in HTML, that's the difference. As opposed to being plain text.



#### v0.64 7/7/15 by DW



I wanted to do a site redirect that was more than just a domain name change. 



If a request came in for: http://archive.scripting.com/2007/12



I wanted it to redirect to: http://scripting.com/2007/12.html



That was accomplished with a new element in config.json: jsSiteRedirect. Its value is a JavaScript expression. You can access the path in the variable parsedUrl.pathname.



This is what config.json in the folder archive.scripting.com looks like. 



{"jsSiteRedirect": "'http://scripting.com' + parsedUrl.pathname + '.html'"}



#### v0.62 6/25/15 by DW



Code cleanup and factoring in opml.js.



#### v0.61 6/24/15 by DW



Files with the extension .opml are now  rendered as outlines. 



There's a template for outlines, it uses the new outlineBrowser toolkit. Here's an example of an OPML file rendered through the new template. 



If you want the raw OPML text from the server, you can do it one of two ways:



1. Add ?format=opml at the end of the url. Example.



2. If you're making the request in a program, you can add an Accept header with the value: text/x-opml, following a convention established by the OPML Editor. 



As with scripts and markdown files, you can turn the feature off in config.json. 



#### v0.60 5/26/15 by DW



When delegating requests, pass redirects back to the client, don't follow them. This was necessary so that that OAuth dance with Twitter in nodeStorage would work. 



There's a new structured listing of the source code of PagePark, linked to from the flat listing (above). This makes it easy to see how the code is organized. It gets pretty deeply nested! The outline view makes that manageable. I use the OPML Editor on a Mac to edit PagePark. 



#### v0.59 5/23/15 by DW



You can now delegate requests to apps running on other ports on your server machine. 



There's a new optional section of the prefs. json file where you specify these mappings. It's called domainMap. It's a set of name-values, where the name is the domain, and the value is the port the requests are mapped to. 



Here's an example of a prefs.json that has a domainMap specified. It maps twitter.radio3.io to the process running on port 5342, twitter.happyfriends.camp to 5338, and any request on judgment.club to the process on port 5351. 



The name matches the end of the HOST header for the request, so a request for judy.judgment.club will map, as will renee.judgment.club and judgment.club.



#### v0.57 5/11/15 by DW



PagePark has pre-defined pages, /now, /version and /status, whose values are returned by PagePark itself. It used to be that they took precedence, so if a site defines pages with those names, the internal ones would be served instead. Now we only serve them if the site didn't define it. 



The urlSiteContents feature now transmits search params. It still will only forward GET calls. This needs to be updated in a future version. 



PagePark now supports wildcards. Suppose you want to serve all the names from mydomain.org with a wildcard. Create a sub-folder of the domains folder with the name *.mydomain.org. If a request comes in for a sub-domain of mydomain.org that doesn't have its own folder, we'll route it through that folder. You can combine this feature with the urlSiteContents feature, or script-implemented pages. 



We also set the X-Forwarded-Host and X-Forwarded-For headers on urlSiteContents requests. 



#### v0.56 5/5/15 by DW



New prefs and config values that allow you to disable processing of scripts and Markdown files. By setting the values in prefs.json, you control all domains on the server. And by adding the values to config.json, in the folder the site is served from, you can turn them off selectively by site. I needed to turn off script processing for .js files served from River4, to make it possible to serve a full river from PagePark. 



#### v0.55 4/26/15 by DW



With this release you can serve domains whose content is stored elsewhere on the web. 



There's a new optional element of config.json, urlSiteContents. Its value is the URL of a directory on the web that stores the content of the domain. 



As an experiment, I mapped the domain my.this.how to point to a folder in my public Dropbox folder. Here's how I did it.



1. First I created a new sub-folder of my public Dropbox folder called pageParkDemo. I put one file in that folder, index.html.



2. I used the web interface of my domain registrar to point my.this.how at the IP address of my PagePark server.



3. On my PagePark server, I created a sub-folder of the domains folder called my.this.how.



4. I created a config.json file in that folder, pointing to the pageParkDemo folder in my Dropbox folder.



To test the setup, I just went to my.this.how, where I saw "this is just a demo".



Just for fun I put a picture in that folder, to see if it works.



Obviously it can get a lot more elaborate, and you can store the content anywhere, not just in a Dropbox folder. The key is that the content be accessible over the web. 



#### v0.54 2/18/15 by DW



A new feature for pages implemented as scripts. If the script returns the value undefined PagePark will not return a value to the HTTP client, it assumes that the script will do this. 



To make it possible for a script page to return a value to the client, there's a new built-in function httpReturn. It takes two parameters, the value, a string, and the type, a MIME type. For example you might have a script that makes an HTTP request and returns a value based on the result to the caller.



Here's an example script that illustrates.



#### v0.51 1/18/15 by DW



Created utils.js in the lib folder, and require it in pagepark.js.



New feature: If there's a file called config.json in a domain folder, we read it on every request, and values in that file can change the behavior of the server. The first feature allows you to do a whole-site redirect. Useful if you want to have several names map to the same content. Here's an example of the config.json file that maps a domain to nodestorage.io.



#### v0.48 1/8/15 by DW



The default port the server boots up on is now 1339. Previously it was 80, which is the standard port for HTTP, but on many OSes this requires PagePark to be running in supervisor mode. I added docs above to explain this. 



Changed package.json so that only *request* and *marked* were listed as dependencies. Apparently the others are included in Node without having to list them. 



Instead of keeping our own MIME type table, we use the Node *mime* package, which is also included as a dependency in the package.json file.



#### v0.47 1/7/15 by DW



Added a package.json file to the repository.



Fixed first-time startup problem creating prefs.json and stats.json. 



Also, we now make sure the *domains* folder exists at startup.



Fixed a problem in handling requests if you specified a different folder for PagePark to serve from. 



### Questions, comments?



Please post a note on the Server Snacks mail list or post an issue here.