{"id":19176977,"url":"https://github.com/potahtml/mpa-archive","last_synced_at":"2025-04-07T22:11:40.785Z","repository":{"id":241368287,"uuid":"806724710","full_name":"potahtml/mpa-archive","owner":"potahtml","description":"Crawls a Multi-Page Application to a zip file, serve the Multi-Page Application from the zip file. A MPA archiver. Could be used as a Site Generator. Written for https://pota.quack.uy/","archived":false,"fork":false,"pushed_at":"2024-09-30T03:33:03.000Z","size":78,"stargazers_count":475,"open_issues_count":3,"forks_count":11,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-31T21:49:57.160Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://pota.quack.uy/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/potahtml.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-27T19:09:43.000Z","updated_at":"2025-03-21T15:36:49.000Z","dependencies_parsed_at":"2024-05-28T02:06:29.551Z","dependency_job_id":"8b38be55-93d8-4daf-aebc-aa10956bcb22","html_url":"https://github.com/potahtml/mpa-archive","commit_stats":{"total_commits":57,"total_committers":1,"mean_commits":57.0,"dds":0.0,"last_synced_commit":"4f55758b1f35b8a1be262da85d41c0c9bea5c536"},"previous_names":["potahtml/mpa-archive"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/potahtml%2Fmpa-archive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/potahtml%2Fmpa-archive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/potahtml%2Fmpa-archive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/potahtml%2Fmpa-archive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/potahtml","download_url":"https://codeload.github.com/potahtml/mpa-archive/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247737788,"owners_count":20987721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T10:31:26.382Z","updated_at":"2025-04-07T22:11:40.763Z","avatar_url":"https://github.com/potahtml.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Multi-Page Application Archive\n\nCrawls a Multi-Page Application into a zip file. Serve the Multi-Page\nApplication from the zip file. A MPA archiver. Could be used as a Site\nGenerator.\n\n## Installation\n\n`npm install -g mpa-archive`\n\n## Usage\n\n### Crawling\n\n`mpa http://example.net`\n\nWill crawl the url recursively and save it in `example.net.zip`. Once\ndone, it will display a report and can serve the files from the zip.\n\n#### SPA mode\n\nThe original idea is to save the HTML generated by JavaScript, to\nallow search engines index the content of a website that uses\nJavaScript. This has the undesired result that some applications,\nspecially SPA may not work. To save the original HTML instead of the\nrendered HTML you can use the `--spa` option, which will save the\noriginal HTML and avoid re-writing links.\n\n`mpa https://example.net --spa`\n\n### Serving\n\n`mpa`\n\nWill create a server for each zip file on the current directory. binds\nto `0.0.0.0` (it can be opened in `localhost`) the `port` is random\nbut seeded to the zip file name, so it remains the same.\n\n## Features\n\n- It uses headless puppeteer\n- Crawls `http://example.net` with `cpu count / 2` threads\n- Progress is displayed in the console\n- Fetches `urls.txt` , `sitemap.txt` and `sitemap.xml` as a seed point\n- It serializes Custom Elements too\n- Reports HTTP status codes different than 200, 304, 204, 206\n- Crawls on site urls only but will `fetch` external resources\n- Intercepts site resources and saves that too\n- Generates `mpa/sitemap.txt` and `mpa/sitemap.xml`\n- Saves site sourcemaps\n- Can resume if process exit, save checkpoint every 250 urls\n- When serving what has been crawled, if an url is not found it will\n  fetch it from source and update the zip\n- domain blacklist\n  https://github.com/potahtml/mpa-archive/blob/master/src/lib/blacklist.js\n- downloads are saved via `fetch` requests\n\n## Legends\n\n- 🍳 a `url` has been opened in a tab for crawling\n- 🧽 `a` link on page has been focused (for when js modules are\n  preloaded/loaded on focus)\n- ⚠ response headers contains a status code differently than 200,\n  304, 204, 206\n- 🧭 the `document.documentElement.outerHTML` has been saved\n- ✔ the response of a `fetch` request has been saved, or the response\n  body of a request made by the tab has been saved\n- 🔗 a `fetch` request has been fired\n- 🛑🍳 the opened tab gave an error (it retries with via a `fetch`\n  request)\n- 🛑🔗 a fetch request gave en error\n\n## To Consider\n\n- save it in an incremental compression format, that doesnt require\n  re-compressing the whole file when it changes, maybe already does\n  that?\n- urls to externals resources are not re-written to be local\n  resources, if this is done then stuff loaded from the root will\n  break\n- it should maybe crawl the site by clicking the links instead of\n  opening a full tab\n- crawl updates\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpotahtml%2Fmpa-archive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpotahtml%2Fmpa-archive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpotahtml%2Fmpa-archive/lists"}