{"id":15009819,"url":"https://github.com/gatenlp/ultimate-sitemap-parser","last_synced_at":"2025-05-15T09:08:17.612Z","repository":{"id":40404408,"uuid":"159314727","full_name":"GateNLP/ultimate-sitemap-parser","owner":"GateNLP","description":"Ultimate Website Sitemap Parser","archived":false,"fork":false,"pushed_at":"2025-04-28T10:00:26.000Z","size":419,"stargazers_count":208,"open_issues_count":2,"forks_count":68,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-05-15T09:08:10.060Z","etag":null,"topics":["python","python-3","python3","robots-txt","sitemap","sitemap-xml","xml-sitemap","xml-sitemap-parser"],"latest_commit_sha":null,"homepage":"https://ultimate-sitemap-parser.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GateNLP.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"docs/contributing.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-11-27T10:05:18.000Z","updated_at":"2025-05-13T04:23:33.000Z","dependencies_parsed_at":"2025-01-22T10:00:51.249Z","dependency_job_id":"4e201e2d-74e2-4b65-9c28-9c519b712666","html_url":"https://github.com/GateNLP/ultimate-sitemap-parser","commit_stats":{"total_commits":75,"total_committers":4,"mean_commits":18.75,"dds":"0.29333333333333333","last_synced_commit":"68d1ccdd573be16cdfadc7f071c088642182bff1"},"previous_names":["berkmancenter/mediacloud-ultimate_sitemap_parser","mediacloud/ultimate-sitemap-parser"],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fultimate-sitemap-parser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fultimate-sitemap-parser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fultimate-sitemap-parser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GateNLP%2Fultimate-sitemap-parser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GateNLP","download_url":"https://codeload.github.com/GateNLP/ultimate-sitemap-parser/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254310520,"owners_count":22049470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","python-3","python3","robots-txt","sitemap","sitemap-xml","xml-sitemap","xml-sitemap-parser"],"created_at":"2024-09-24T19:28:47.195Z","updated_at":"2025-05-15T09:08:12.601Z","avatar_url":"https://github.com/GateNLP.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Ultimate Sitemap Parser\n-----------------------\n\n.. image:: https://img.shields.io/pypi/pyversions/ultimate-sitemap-parser\n   :alt: PyPI - Python Version\n   :target: https://github.com/GateNLP/ultimate-sitemap-parser\n\n.. image:: https://img.shields.io/pypi/v/ultimate-sitemap-parser\n   :alt: PyPI - Version\n   :target: https://pypi.org/project/ultimate-sitemap-parser/\n\n.. image:: https://img.shields.io/conda/vn/conda-forge/ultimate-sitemap-parser\n   :alt: Conda Version\n   :target: https://anaconda.org/conda-forge/ultimate-sitemap-parser\n\n.. image:: https://img.shields.io/pepy/dt/ultimate-sitemap-parser\n   :target: https://pepy.tech/project/ultimate-sitemap-parser\n   :alt: Pepy Total Downloads\n\n\n**Ultimate Sitemap Parser (USP) is a performant and robust Python library for parsing and crawling sitemaps.**\n\n\nFeatures\n========\n\n- Supports all sitemap formats:\n\n  - `XML sitemaps \u003chttps://www.sitemaps.org/protocol.html#xmlTagDefinitions\u003e`_\n  - `Google News sitemaps \u003chttps://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap\u003e`_ and `Image sitemaps \u003chttps://developers.google.com/search/docs/advanced/sitemaps/image-sitemaps\u003e`_\n  - `plain text sitemaps \u003chttps://www.sitemaps.org/protocol.html#otherformats\u003e`_\n  - `RSS 2.0 / Atom 0.3 / Atom 1.0 sitemaps \u003chttps://www.sitemaps.org/protocol.html#otherformats\u003e`_\n  - `Sitemaps linked from robots.txt \u003chttps://developers.google.com/search/reference/robots_txt#sitemap\u003e`_\n\n- Field-tested with ~1 million URLs as part of the `Media Cloud project \u003chttps://mediacloud.org/\u003e`_\n- Error-tolerant with more common sitemap bugs\n- Tries to find sitemaps not listed in ``robots.txt``\n- Uses fast and memory efficient Expat XML parsing\n- Doesn't consume much memory even with massive sitemap hierarchies\n- Provides a generated sitemap tree as easy to use object tree\n- Supports using a custom web client\n- Uses a small number of actively maintained third-party modules\n- Reasonably tested\n\n\nInstallation\n============\n\n.. code:: sh\n\n    pip install ultimate-sitemap-parser\n\nor using Anaconda:\n\n.. code:: sh\n\n    conda install -c conda-forge ultimate-sitemap-parser\n\n\nUsage\n=====\n\n.. code:: python\n\n    from usp.tree import sitemap_tree_for_homepage\n\n    tree = sitemap_tree_for_homepage('https://www.example.org/')\n\n    for page in tree.all_pages():\n        print(page.url)\n\n``sitemap_tree_for_homepage()`` will return a tree of ``AbstractSitemap`` subclass objects that represent the sitemap\nhierarchy found on the website; see a `reference of AbstractSitemap subclasses \u003chttps://ultimate-sitemap-parser.readthedocs.io/en/latest/reference/api/usp.objects.sitemap.html\u003e`_. `AbstractSitemap.all_pages()` returns a generator to efficiently iterate over pages without loading the entire tree into memory.\n\nFor more examples and details, see the `documentation \u003chttps://ultimate-sitemap-parser.readthedocs.io/en/latest/\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fultimate-sitemap-parser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgatenlp%2Fultimate-sitemap-parser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgatenlp%2Fultimate-sitemap-parser/lists"}