{"id":13564880,"url":"https://github.com/breakroom/sitemapper","last_synced_at":"2025-04-10T03:53:23.566Z","repository":{"id":42189572,"uuid":"215747614","full_name":"breakroom/sitemapper","owner":"breakroom","description":"Fast, stream based XML Sitemap generator in Elixir","archived":false,"fork":false,"pushed_at":"2024-12-30T09:50:46.000Z","size":295,"stargazers_count":46,"open_issues_count":5,"forks_count":14,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T03:53:19.360Z","etag":null,"topics":["elixir","search","seo","sitemap-generator","sitemaps"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/breakroom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-17T08:57:58.000Z","updated_at":"2025-02-28T19:05:54.000Z","dependencies_parsed_at":"2024-01-05T21:59:59.650Z","dependency_job_id":"8aae3b5f-640d-4c8d-ae6b-35474a7eaf35","html_url":"https://github.com/breakroom/sitemapper","commit_stats":{"total_commits":58,"total_committers":5,"mean_commits":11.6,"dds":"0.12068965517241381","last_synced_commit":"7bf3ac50c6912750dc9bb1ddcd5b2ab8337f6d57"},"previous_names":["tomtaylor/sitemapper"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breakroom%2Fsitemapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breakroom%2Fsitemapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breakroom%2Fsitemapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/breakroom%2Fsitemapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/breakroom","download_url":"https://codeload.github.com/breakroom/sitemapper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154998,"owners_count":21056542,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","search","seo","sitemap-generator","sitemaps"],"created_at":"2024-08-01T13:01:37.441Z","updated_at":"2025-04-10T03:53:23.555Z","avatar_url":"https://github.com/breakroom.png","language":"Elixir","funding_links":[],"categories":["Utilities"],"sub_categories":[],"readme":"# Sitemapper\n\n[![Hex pm](http://img.shields.io/hexpm/v/sitemapper.svg?style=flat)](https://hex.pm/packages/sitemapper)\n\nSitemapper is an Elixir library for generating [XML Sitemaps](https://www.sitemaps.org).\n\nIt's designed to generate large sitemaps while maintaining a low memory profile. It can persist sitemaps to Amazon S3, GCP Storage, disk or any other adapter you wish to write.\n\n## Installation\n\n```elixir\ndef deps do\n  [\n    {:sitemapper, \"~\u003e 0.8\"}\n  ]\nend\n```\n\n## Usage\n\n```elixir\n  def generate_sitemap() do\n    config = [\n      store: Sitemapper.S3Store,\n      store_config: [bucket: \"example-bucket\"],\n      sitemap_url: \"http://example-bucket.s3-aws-region.amazonaws.com\"\n    ]\n\n    Stream.concat([1..100_001])\n    |\u003e Stream.map(fn i -\u003e\n      %Sitemapper.URL{\n        loc: \"http://example.com/page-#{i}\",\n        changefreq: :daily,\n        lastmod: Date.utc_today()\n      }\n    end)\n    |\u003e Sitemapper.generate(config)\n    |\u003e Sitemapper.persist(config)\n    |\u003e Stream.run()\n  end\n```\n\n`Sitemapper.generate` receives a `Stream` of URLs. This makes it easy to stream the contents of an Ecto Repo into a sitemap.\n\n```elixir\n  def generate_sitemap() do\n    config = [\n      store: Sitemapper.S3Store,\n      store_config: [bucket: \"example-bucket\"],\n      sitemap_url: \"http://example-bucket.s3-aws-region.amazonaws.com\"\n    ]\n\n    Repo.transaction(fn -\u003e\n      User\n      |\u003e Repo.stream()\n      |\u003e Stream.map(fn %User{username: username, updated_at: updated_at} -\u003e\n        %Sitemapper.URL{\n          loc: \"http://example.com/users/#{username}\",\n          changefreq: :hourly,\n          lastmod: updated_at\n        }\n      end)\n      |\u003e Sitemapper.generate(config)\n      |\u003e Sitemapper.persist(config)\n      |\u003e Stream.run()\n    end)\n  end\n```\n\nTo persist your sitemaps to the local file system, instead of Amazon S3, your config should look like:\n\n```elixir\n[\n  store: Sitemapper.FileStore,\n  store_config: [\n    path: sitemap_folder_path\n  ],\n  sitemap_url: \"http://yourdomain.com\"\n]\n```\n\nNote that you'll need to finish on `Stream.run/1` or `Enum.to_list/1` to execute the stream and return the result.\n\n## Todo\n\n- Support extended Sitemap properties, like images, video, etc.\n\n## Benchmarks\n\n`sitemapper` is about 50% faster than `fast_sitemap`. In the large scenario below (1,000,000 URLs) `sitemapper` uses ~50MB peak memory consumption, while `fast_sitemap` uses ~1000MB.\n\n```shell\n$ MIX_ENV=bench mix run bench/bench.exs\nOperating System: macOS\nCPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz\nNumber of Available Cores: 8\nAvailable memory: 16 GB\nElixir 1.9.1\nErlang 22.0.4\n\nBenchmark suite executing with the following configuration:\nwarmup: 2 s\ntime: 10 s\nmemory time: 0 ns\nparallel: 1\ninputs: large, medium, small\nEstimated total run time: 1.20 min\n\nBenchmarking fast_sitemap - simple with input large...\nBenchmarking fast_sitemap - simple with input medium...\nBenchmarking fast_sitemap - simple with input small...\nBenchmarking sitemapper - simple with input large...\nBenchmarking sitemapper - simple with input medium...\nBenchmarking sitemapper - simple with input small...\n\n##### With input large #####\nName                            ips        average  deviation         median         99th %\nsitemapper - simple          0.0353        28.32 s     ±0.00%        28.32 s        28.32 s\nfast_sitemap - simple        0.0223        44.76 s     ±0.00%        44.76 s        44.76 s\n\nComparison:\nsitemapper - simple          0.0353\nfast_sitemap - simple        0.0223 - 1.58x slower +16.45 s\n\n##### With input medium #####\nName                            ips        average  deviation         median         99th %\nsitemapper - simple            0.35         2.85 s     ±0.57%         2.85 s         2.87 s\nfast_sitemap - simple          0.23         4.28 s     ±0.46%         4.28 s         4.30 s\n\nComparison:\nsitemapper - simple            0.35\nfast_sitemap - simple          0.23 - 1.50x slower +1.43 s\n\n##### With input small #####\nName                            ips        average  deviation         median         99th %\nsitemapper - simple           32.00       31.25 ms     ±8.01%       31.20 ms       37.22 ms\nfast_sitemap - simple         21.55       46.41 ms     ±6.96%       46.26 ms       56.36 ms\n\nComparison:\nsitemapper - simple           32.00\nfast_sitemap - simple         21.55 - 1.49x slower +15.16 ms\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbreakroom%2Fsitemapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbreakroom%2Fsitemapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbreakroom%2Fsitemapper/lists"}