{"id":13941800,"url":"https://github.com/lgraubner/sitemap-generator","last_synced_at":"2025-05-15T17:04:20.487Z","repository":{"id":8452949,"uuid":"58463074","full_name":"lgraubner/sitemap-generator","owner":"lgraubner","description":"Easily create XML sitemaps for your website.","archived":false,"fork":false,"pushed_at":"2024-06-28T09:50:56.000Z","size":849,"stargazers_count":432,"open_issues_count":33,"forks_count":134,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-05-14T15:28:06.877Z","etag":null,"topics":["crawler","google","seo","sitemap","sitemap-generator","xml-sitemap"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lgraubner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-10T13:21:34.000Z","updated_at":"2025-05-09T13:13:12.000Z","dependencies_parsed_at":"2024-07-30T22:18:05.550Z","dependency_job_id":null,"html_url":"https://github.com/lgraubner/sitemap-generator","commit_stats":null,"previous_names":[],"tags_count":42,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lgraubner","download_url":"https://codeload.github.com/lgraubner/sitemap-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254384987,"owners_count":22062422,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","google","seo","sitemap","sitemap-generator","xml-sitemap"],"created_at":"2024-08-08T02:01:29.311Z","updated_at":"2025-05-15T17:04:20.467Z","avatar_url":"https://github.com/lgraubner.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Sitemap Generator\n\n[![Travis](https://img.shields.io/travis/lgraubner/sitemap-generator.svg)](https://travis-ci.org/lgraubner/sitemap-generator) [![David](https://img.shields.io/david/lgraubner/sitemap-generator.svg)](https://david-dm.org/lgraubner/sitemap-generator) [![npm](https://img.shields.io/npm/v/sitemap-generator.svg)](https://www.npmjs.com/package/sitemap-generator)\n\n\u003e Easily create XML sitemaps for your website.\n\nGenerates a sitemap by crawling your site. Uses streams to efficiently write the sitemap to your drive and runs asynchronously to avoid blocking the thread. Is cappable of creating multiple sitemaps if threshold is reached. Respects robots.txt and meta tags.\n\nThis package is not meant to be used in a production code base directly, but rather on the deployed product. This means you develop your app/website as usual, deploy it and create the sitemap with this tool _afterwards_. The simplest way is to use the [CLI](https://github.com/lgraubner/sitemap-generator-cli) (this is a different package!) to create the sitemap on the command line. If you have a more advanced usecase or want to adjust the crawler behavior you should use the programmtic version (this package). Create the crawler as needed and simply run it via `node mycrawler.js`.\n\n## Table of contents\n\n- [Install](#install)\n- [Usage](#usage)\n- [API](#api)\n- [Options](#options)\n- [Events](#events)\n- [FAQ](#faq)\n- [License](#license)\n\n## Install\n\nThis module is available on [npm](https://www.npmjs.com/).\n\n```\n$ npm install -S sitemap-generator\n```\n\nThis module is running only with Node.js and is not meant to be used in the browser.\n\n## Usage\n\n```JavaScript\nconst SitemapGenerator = require('sitemap-generator');\n\n// create generator\nconst generator = SitemapGenerator('http://example.com', {\n  stripQuerystring: false\n});\n\n// register event listeners\ngenerator.on('done', () =\u003e {\n  // sitemaps created\n});\n\n// start the crawler\ngenerator.start();\n```\n\nThe crawler will fetch all folder URL pages and file types [parsed by Google](https://support.google.com/webmasters/answer/35287?hl=en). If present the `robots.txt` will be taken into account and possible rules are applied for each URL to consider if it should be added to the sitemap. Also the crawler will not fetch URL's from a page if the robots meta tag with the value `nofollow` is present and ignore them completely if `noindex` rule is present. The crawler is able to apply the `base` value to found links.\n\n## API\n\nThe generator offers straightforward methods to start and stop it. You can also add URL's manually.\n\n### start()\n\nStarts crawler asynchronously and writes sitemap to disk.\n\n### stop()\n\nStops the running crawler and halts the sitemap generation.\n\n### getCrawler()\n\nReturns the crawler instance. For more information about the crawler check the [simplecrawler docs](https://github.com/simplecrawler/simplecrawler#readme).\n\nThis can be useful to ignore certain sites and don't add them to the sitemap.\n\n```JavaScript\nconst crawler = generator.getCrawler();\ncrawler.addFetchCondition((queueItem, referrerQueueItem, callback) =\u003e {\n  callback(null, !queueItem.path.match(/myregex/));\n});\n```\n\n### getSitemap()\n\nReturns the sitemap instance (`SitemapRotator`).\n\nThis can be useful to add static URLs to the sitemap:\n\n```JavaScript\nconst crawler = generator.getCrawler()\nconst sitemap = generator.getSitemap()\n\n// Add static URL on crawl init.\ncrawler.on('crawlstart', () =\u003e {\n  sitemap.addURL('/my/static/url')\n})\n````\n\n### queueURL(url)\n\nAdd a URL to crawler's queue. Useful to help crawler fetch pages it can't find itself.\n\n## Options\n\nThere are a couple of options to adjust the sitemap output. In addition to the options beneath the options of the used crawler can be changed. For a complete list please check it's [official documentation](https://github.com/simplecrawler/simplecrawler#configuration).\n\n```JavaScript\nvar generator = SitemapGenerator('http://example.com', {\n  maxDepth: 0,\n  filepath: './sitemap.xml',\n  maxEntriesPerFile: 50000,\n  stripQuerystring: true\n});\n```\n\n### changeFreq\n\nType: `string`  \nDefault: `undefined`\n\nIf defined, adds a `\u003cchangefreq\u003e` line to each URL in the sitemap. Possible values are `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, `never`. All other values are ignored.\n\n### filepath\n\nType: `string`  \nDefault: `./sitemap.xml`\n\nFilepath for the new sitemap. If multiple sitemaps are created \"part\\_$index\" is appended to each filename. If you don't want to write a file at all you can pass `null` as filepath.\n\n### httpAgent\n\nType: `HTTPAgent`  \nDefault: `http.globalAgent`\n\nControls what HTTP agent to use. This is useful if you want configure HTTP connection through a HTTP/HTTPS proxy (see [http-proxy-agent](https://www.npmjs.com/package/http-proxy-agent)).\n\n### httpsAgent\n\nType: `HTTPAgent`  \nDefault: `https.globalAgent`\n\nControls what HTTPS agent to use. This is useful if you want configure HTTPS connection through a HTTP/HTTPS proxy (see [https-proxy-agent](https://www.npmjs.com/package/https-proxy-agent)).\n\nExample:\n\n```JavaScript\n// don't forget to:\n// npm i http-proxy-agent https-proxy-agent\nconst HttpProxyAgent = require(\"http-proxy-agent\");\nconst HttpsProxyAgent = require(\"https-proxy-agent\");\nconst proxyAddress = 'http://localhost:1234';\nconst httpProxyAgent = new HttpProxyAgent(proxyAddress);\nconst httpsProxyAgent = new HttpsProxyAgent(proxyAddress);\noptions.httpAgent = httpProxyAgent;\noptions.httpsAgent = httpsProxyAgent;\n```\n\n### ignore(url)\n\nApply a test condition to a URL before it's added to the sitemap.\n\nType: `function`  \nDefault: `null`\n\nExample:\n\n```JavaScript\nconst generator = SitemapGenerator(url, {\n  ignore: url =\u003e {\n    // Prevent URLs from being added that contain `\u003cpattern\u003e`.\n    return /\u003cpattern\u003e/g.test(url)\n  }\n})\n```\n\n### ignoreAMP\n\nType: `boolean`  \nDefault: `true`\n\nIndicates whether [Google AMP pages](https://www.ampproject.org/) should be ignored and not be added to the sitemap.\n\n### lastMod\n\nType: `boolean`  \nDefault: `false`\n\nWhether to add a `\u003clastmod\u003e` line to each URL in the sitemap. If present the responses `Last-Modified` header will be used. Otherwise todays date is added.\n\n### maxEntriesPerFile\n\nType: `number`  \nDefault: `50000`\n\nGoogle limits the maximum number of URLs in one sitemap to 50000. If this limit is reached the sitemap-generator creates another sitemap. A sitemap index file will be created as well.\n\n### priorityMap\n\nType: `array`  \nDefault: `[]`\n\nIf provided, adds a `\u003cpriority\u003e` line to each URL in the sitemap. Each value in priorityMap array corresponds with the depth of the URL being added. For example, the priority value given to a URL equals `priorityMap[depth - 1]`. If a URL's depth is greater than the length of the priorityMap array, the last value in the array will be used. Valid values are between `1.0` and `0.0`.\n\nExample:\n\n```javascript\n[1.0, 0.8, 0.6, 0.4, 0.2, 0]\n```\n\n### userAgent\n\nType: `string`  \nDefault: `Node/SitemapGenerator`\n\nChange the default crawler user agent.\n\n## Events\n\nThe Sitemap Generator emits several events which can be listened to.\n\n### `add`\n\nTriggered when the crawler successfully added a resource to the sitemap. Passes the url as argument.\n\n```JavaScript\ngenerator.on('add', (url) =\u003e {\n  // log url\n});\n```\n\n### `done`\n\nTriggered when the crawler finished and the sitemap is created.\n\n```JavaScript\ngenerator.on('done', () =\u003e {\n  // sitemaps created\n});\n```\n\n### `error`\n\nThrown if there was an error while fetching an URL. Passes an object with the http status code, a message and the url as argument.\n\n```JavaScript\ngenerator.on('error', (error) =\u003e {\n  console.log(error);\n  // =\u003e { code: 404, message: 'Not found.', url: 'http://example.com/foo' }\n});\n```\n\n### `ignore`\n\nIf an URL matches a disallow rule in the `robots.txt` file or meta robots noindex is present this event is triggered. The URL will not be added to the sitemap. Passes the ignored url as argument.\n\n```JavaScript\ngenerator.on('ignore', (url) =\u003e {\n  // log ignored url\n});\n```\n\n## FAQ\n\n\u003cdetails\u003e\n\u003csummary\u003eDoes this work with React, Angular, ...\u003c/summary\u003e\n\u003cp\u003eThis package don't care what frameworks and technologies you are using under the hood. The only requirement is, that your URL's return valid HTML. Therefore SSR (server side rendering) is required for single page apps as no JavaScript is executed.\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWhere to put this code\u003c/summary\u003e\n\u003cp\u003eThis is basically up to you. You can execute this code manually and upload your sitemap by hand, or you can put this on your server and run this periodically to keep your sitemap up to date.\u003c/p\u003e\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eShould I use this package or the CLI\u003c/summary\u003e\n\u003cp\u003eThe CLI should suffice most of the common use cases. It has several options to tweak in case you want it to behave differently. If your use case is more advanced and you need fine control about what the crawler should fetch, you should use this package and the programmatic API.\u003c/p\u003e\n\u003c/details\u003e\n\n## License\n\n[MIT](https://github.com/lgraubner/sitemap-generator/blob/master/LICENSE) © [Lars Graubner](https://larsgraubner.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgraubner%2Fsitemap-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flgraubner%2Fsitemap-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgraubner%2Fsitemap-generator/lists"}