{"id":13679946,"url":"https://github.com/lgraubner/sitemap-generator-cli","last_synced_at":"2025-04-13T02:15:50.561Z","repository":{"id":32199396,"uuid":"35773034","full_name":"lgraubner/sitemap-generator-cli","owner":"lgraubner","description":"Creates an XML-Sitemap by crawling a given site.","archived":false,"fork":false,"pushed_at":"2023-01-03T23:14:32.000Z","size":1819,"stargazers_count":324,"open_issues_count":27,"forks_count":43,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-13T02:15:37.459Z","etag":null,"topics":["cli","crawler","google","seo","sitemap","xml-sitemap"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lgraubner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-17T16:15:30.000Z","updated_at":"2025-04-02T13:13:28.000Z","dependencies_parsed_at":"2023-01-14T20:44:08.124Z","dependency_job_id":null,"html_url":"https://github.com/lgraubner/sitemap-generator-cli","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lgraubner%2Fsitemap-generator-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lgraubner","download_url":"https://codeload.github.com/lgraubner/sitemap-generator-cli/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248654105,"owners_count":21140237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","crawler","google","seo","sitemap","xml-sitemap"],"created_at":"2024-08-02T13:01:11.306Z","updated_at":"2025-04-13T02:15:50.532Z","avatar_url":"https://github.com/lgraubner.png","language":"JavaScript","funding_links":[],"categories":["JavaScript"],"sub_categories":[],"readme":"# Sitemap Generator CLI\n\n[![Travis](https://img.shields.io/travis/lgraubner/sitemap-generator-cli.svg)](https://travis-ci.org/lgraubner/sitemap-generator-cli) [![David](https://img.shields.io/david/lgraubner/sitemap-generator-cli.svg)](https://david-dm.org/lgraubner/sitemap-generator-cli) [![npm](https://img.shields.io/npm/v/sitemap-generator-cli.svg)](https://www.npmjs.com/package/sitemap-generator-cli)\n\n\u003e Create xml sitemaps from the command line.\n\nGenerates a sitemap by crawling your site. Uses streams to efficiently write the sitemap to your drive. Is cappable of creating multiple sitemaps if threshold is reached. Respects robots.txt and meta tags.\n\n## Table of contents\n\n- [Install](#install)\n- [Usage](#usage)\n- [Options](#options)\n- [License](#license)\n\n## Install\n\nThis module is available on [npm](https://www.npmjs.com/).\n\n```BASH\nnpm install -g sitemap-generator-cli\n# or execute it directly with npx (since npm v5.2)\nnpx sitemap-generator-cli https://example.com\n```\n\n## Usage\n\nThe crawler will fetch all folder URL pages and file types [parsed by Google](https://support.google.com/webmasters/answer/35287?hl=en). If present the `robots.txt` will be taken into account and possible rules are applied for each URL to consider if it should be added to the sitemap. Also the crawler will not fetch URL's from a page if the robots meta tag with the value `nofollow` is present and ignore them completely if `noindex` rule is present. The crawler is able to apply the `base` value to found links.\n\n```BASH\nsitemap-generator [options] \u003curl\u003e\n```\n\nWhen the crawler finished the XML Sitemap will be built and saved to your specified filepath. If the count of fetched pages is greater than 50000 it will be splitted into several sitemap files and create a sitemapindex file. Google does not allow more than 50000 items in one sitemap.\n\nExample:\n\n```BASH\nsitemap-generator http://example.com\n```\n\n## Options\n\n```BASH\nsitemap-generator --help\n\n  Usage: cli [options] \u003curl\u003e\n\n  Options:\n\n    -V, --version                           output the version number\n    -f, --filepath \u003cfilepath\u003e               path to file including filename (default: sitemap.xml)\n    -m, --max-entries \u003cmaxEntries\u003e          limits the maximum number of URLs per sitemap file (default: 50000)\n    -d, --max-depth \u003cmaxDepth\u003e              limits the maximum distance from the original request (default: 0)\n    -q, --query                             consider query string\n    -u, --user-agent \u003cagent\u003e                set custom User Agent\n    -v, --verbose                           print details when crawling\n    -c, --max-concurrency \u003cmaxConcurrency\u003e  maximum number of requests the crawler will run simultaneously (default: 5)\n    -r, --no-respect-robots-txt             controls whether the crawler should respect rules in robots.txt\n    -l, --last-mod                          add Last-Modified header to xml\n    -g, --change-freq \u003cchangeFreq\u003e          adds a \u003cchangefreq\u003e line to each URL in the sitemap.\n    -p, --priority-map \u003cpriorityMap\u003e        priority for each depth url, values between 1.0 and 0.0, example: \"1.0,0.8 0.6,0.4\"\n    -x, --proxy \u003curl\u003e                       Use the passed proxy URL\n    -h, --help                              output usage information\n```\n\n### filepath\n\nPath to file to write including the filename itself. Path can be absolute or relative. Default is `sitemap.xml`.\n\nExamples:\n\n- `sitemap.xml`\n- `mymap.xml`\n- `/var/www/sitemap.xml`\n- `./sitemap.myext`\n\n### maxConcurrency\n\nSets the maximum number of requests the crawler will run simultaneously (default: 5).\n\n### maxEntries\n\nDefine a limit of URLs per sitemap files, useful for site with lots of urls. Defaults to 50000.\n\n### maxDepth\n\nSet a maximum distance from the original request to crawl URLs, useful for generating smaller `sitemap.xml` files. Defaults to 0, which means it will crawl all levels.\n\n### noRespectRobotsTxt\n\nControls whether the crawler should respect rules in robots.txt.\n\n### query\n\nConsider URLs with query strings like `http://www.example.com/?foo=bar` as individual sites and add them to the sitemap.\n\n### user-agent\n\nSet a custom User Agent used for crawling. Default is `Node/SitemapGenerator`.\n\n### verbose\n\nPrint debug messages during crawling process. Also prints out a summery when finished.\n\n### last-mod\n\nadd Last-Modified header to xml\n\n### change-freq\n\nadds a \u003cchangefreq\u003e line to each URL in the sitemap.\n\n### priority-map\n\nadd priority for each depth url, values between 1.0 and 0.0, example: \"1.0,0.8 0.6,0.4\"\n\n## License\n\n[MIT](https://github.com/lgraubner/sitemap-generator/blob/master/LICENSE) © [Lars Graubner](https://larsgraubner.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgraubner%2Fsitemap-generator-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flgraubner%2Fsitemap-generator-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgraubner%2Fsitemap-generator-cli/lists"}