https://github.com/opifexm/page-loader
A command-line utility that downloads pages from the Internet and saves them on a computer
https://github.com/opifexm/page-loader
axios cheerio commander jest listr nock node-js
Last synced: 4 months ago
JSON representation
A command-line utility that downloads pages from the Internet and saves them on a computer
- Host: GitHub
- URL: https://github.com/opifexm/page-loader
- Owner: opifexM
- Created: 2023-07-03T13:57:35.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2025-04-29T13:03:48.000Z (5 months ago)
- Last Synced: 2025-05-30T13:48:34.852Z (4 months ago)
- Topics: axios, cheerio, commander, jest, listr, nock, node-js
- Language: JavaScript
- Homepage:
- Size: 1.11 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
[](https://sonarcloud.io/summary/new_code?id=opifexM_Page-Loader)
[](https://sonarcloud.io/summary/new_code?id=opifexM_Page-Loader)
[](https://sonarcloud.io/summary/new_code?id=opifexM_Page-Loader)
[](https://github.com/opifexM/fullstack-javascript-project-4/actions)
[](https://github.com/opifexM/Page-Loader/actions/workflows/nodejs.yml)
# PageLoader
**PageLoader** is a command-line utility that downloads web pages from the internet and saves them locally, along with all external resources such as images, stylesheets, and JavaScript files. This allows users to view the pages offline, similar to how browsers save pages for offline use.
## Description
PageLoader is a practical project designed to provide hands-on experience with asynchronous programming in JavaScript using Node.js. The project aims to build a deep understanding of asynchronous operations including promises, `async/await`, error handling, and filesystem interaction.
This utility not only demonstrates how to perform HTTP requests and file downloads asynchronously, but also teaches how to structure robust, testable, and maintainable asynchronous code. Users will work with tools like `axios` for HTTP, `cheerio` for DOM parsing and manipulation, and `nock` for HTTP request mocking in tests.
The page loader extracts all resource links from the downloaded HTML using DOM selectors and replaces them with local paths after downloading the assets. This provides a seamless offline experience when opening the saved page.
## Features
- Downloads HTML pages and saves them to a local directory.
- Parses and downloads all linked resources (CSS, JS, images).
- Rewrites resource links to point to local files.
- Supports async/await for efficient asynchronous operations.
- Logs the execution process for better traceability.
- Provides clear error handling for I/O and HTTP operations.
## Usage
Command Format:
`pageloader [output-path]`
Example:
`pageloader https://example.com ./downloaded`
## Technologies Used
### Core
- **Node.js** — JavaScript runtime environment for executing code outside the browser.
- **Axios** — Promise-based HTTP client for making network requests.
- **Cheerio** — Server-side jQuery-like library for parsing and manipulating HTML.
- **Commander** — CLI helper for parsing arguments and creating commands.
- **Listr2** — Task list library for organizing asynchronous CLI operations.
- **Debug** — Lightweight debugging utility.
- **axios-debug-log** — Middleware for detailed Axios request logging.
### Testing
- **Jest** — Testing framework for JavaScript with support for asynchronous testing.
- **Nock** — HTTP server mocking library, used for testing axios requests without actual network calls.
## License
PageLoader is licensed under the ISC license.