{"id":13449372,"url":"https://github.com/earwig/mwparserfromhell","last_synced_at":"2025-05-14T01:09:51.342Z","repository":{"id":3342196,"uuid":"4386648","full_name":"earwig/mwparserfromhell","owner":"earwig","description":"A Python parser for MediaWiki wikicode","archived":false,"fork":false,"pushed_at":"2025-01-13T05:08:37.000Z","size":2207,"stargazers_count":784,"open_issues_count":86,"forks_count":79,"subscribers_count":37,"default_branch":"main","last_synced_at":"2025-04-03T01:41:33.242Z","etag":null,"topics":["mediawiki","parser","python","wikipedia"],"latest_commit_sha":null,"homepage":"https://mwparserfromhell.readthedocs.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/earwig.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-05-20T18:45:54.000Z","updated_at":"2025-04-02T02:39:24.000Z","dependencies_parsed_at":"2024-01-04T05:34:16.379Z","dependency_job_id":"421ac10c-64d5-4875-86bc-dbd827bf9519","html_url":"https://github.com/earwig/mwparserfromhell","commit_stats":{"total_commits":858,"total_committers":24,"mean_commits":35.75,"dds":"0.40442890442890445","last_synced_commit":"0f89f4426bdd9e184ae8c8223672a7a0bf36eb76"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earwig%2Fmwparserfromhell","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earwig%2Fmwparserfromhell/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earwig%2Fmwparserfromhell/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earwig%2Fmwparserfromhell/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/earwig","download_url":"https://codeload.github.com/earwig/mwparserfromhell/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154646,"owners_count":21056541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mediawiki","parser","python","wikipedia"],"created_at":"2024-07-31T06:00:36.611Z","updated_at":"2025-04-10T03:40:08.732Z","avatar_url":"https://github.com/earwig.png","language":"Python","readme":"mwparserfromhell\n================\n\n.. image:: https://img.shields.io/coveralls/earwig/mwparserfromhell/main.svg\n  :alt: Coverage Status\n  :target: https://coveralls.io/r/earwig/mwparserfromhell\n\n**mwparserfromhell** (the *MediaWiki Parser from Hell*) is a Python package\nthat provides an easy-to-use and outrageously powerful parser for MediaWiki_\nwikicode. It supports Python 3.8+.\n\nDeveloped by Earwig_ with contributions from `Σ`_, Legoktm_, and others.\nFull documentation is available on ReadTheDocs_. Development occurs on GitHub_.\n\nInstallation\n------------\n\nThe easiest way to install the parser is through the `Python Package Index`_;\nyou can install the latest release with ``pip install mwparserfromhell``\n(`get pip`_). Make sure your pip is up-to-date first, especially on Windows.\n\nAlternatively, get the latest development version:\n\n.. code-block:: sh\n\n    git clone https://github.com/earwig/mwparserfromhell.git\n    cd mwparserfromhell\n    python setup.py install\n\nThe comprehensive unit testing suite requires `pytest`_ (``pip install pytest``)\nand can be run with ``python -m pytest``.\n\nUsage\n-----\n\nNormal usage is rather straightforward (where ``text`` is page text):\n\n.. code-block:: python\n\n  \u003e\u003e\u003e import mwparserfromhell\n  \u003e\u003e\u003e wikicode = mwparserfromhell.parse(text)\n\n``wikicode`` is a ``mwparserfromhell.Wikicode`` object, which acts like an\nordinary ``str`` object with some extra methods. For example:\n\n.. code-block:: python\n\n  \u003e\u003e\u003e text = \"I has a template! {{foo|bar|baz|eggs=spam}} See it?\"\n  \u003e\u003e\u003e wikicode = mwparserfromhell.parse(text)\n  \u003e\u003e\u003e print(wikicode)\n  I has a template! {{foo|bar|baz|eggs=spam}} See it?\n  \u003e\u003e\u003e templates = wikicode.filter_templates()\n  \u003e\u003e\u003e print(templates)\n  ['{{foo|bar|baz|eggs=spam}}']\n  \u003e\u003e\u003e template = templates[0]\n  \u003e\u003e\u003e print(template.name)\n  foo\n  \u003e\u003e\u003e print(template.params)\n  ['bar', 'baz', 'eggs=spam']\n  \u003e\u003e\u003e print(template.get(1).value)\n  bar\n  \u003e\u003e\u003e print(template.get(\"eggs\").value)\n  spam\n\nSince nodes can contain other nodes, getting nested templates is trivial:\n\n.. code-block:: python\n\n  \u003e\u003e\u003e text = \"{{foo|{{bar}}={{baz|{{spam}}}}}}\"\n  \u003e\u003e\u003e mwparserfromhell.parse(text).filter_templates()\n  ['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']\n\nYou can also pass ``recursive=False`` to ``filter_templates()`` and explore\ntemplates manually. This is possible because nodes can contain additional\n``Wikicode`` objects:\n\n.. code-block:: python\n\n  \u003e\u003e\u003e code = mwparserfromhell.parse(\"{{foo|this {{includes a|template}}}}\")\n  \u003e\u003e\u003e print(code.filter_templates(recursive=False))\n  ['{{foo|this {{includes a|template}}}}']\n  \u003e\u003e\u003e foo = code.filter_templates(recursive=False)[0]\n  \u003e\u003e\u003e print(foo.get(1).value)\n  this {{includes a|template}}\n  \u003e\u003e\u003e print(foo.get(1).value.filter_templates()[0])\n  {{includes a|template}}\n  \u003e\u003e\u003e print(foo.get(1).value.filter_templates()[0].get(1).value)\n  template\n\nTemplates can be easily modified to add, remove, or alter params. ``Wikicode``\nobjects can be treated like lists, with ``append()``, ``insert()``,\n``remove()``, ``replace()``, and more. They also have a ``matches()`` method\nfor comparing page or template names, which takes care of capitalization and\nwhitespace:\n\n.. code-block:: python\n\n  \u003e\u003e\u003e text = \"{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}\"\n  \u003e\u003e\u003e code = mwparserfromhell.parse(text)\n  \u003e\u003e\u003e for template in code.filter_templates():\n  ...     if template.name.matches(\"Cleanup\") and not template.has(\"date\"):\n  ...         template.add(\"date\", \"July 2012\")\n  ...\n  \u003e\u003e\u003e print(code)\n  {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}}\n  \u003e\u003e\u003e code.replace(\"{{uncategorized}}\", \"{{bar-stub}}\")\n  \u003e\u003e\u003e print(code)\n  {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}\n  \u003e\u003e\u003e print(code.filter_templates())\n  ['{{cleanup|date=July 2012}}', '{{bar-stub}}']\n\nYou can then convert ``code`` back into a regular ``str`` object (for\nsaving the page!) by calling ``str()`` on it:\n\n.. code-block:: python\n\n  \u003e\u003e\u003e text = str(code)\n  \u003e\u003e\u003e print(text)\n  {{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}\n  \u003e\u003e\u003e text == code\n  True\n\nLimitations\n-----------\n\nWhile the MediaWiki parser generates HTML and has access to the contents of\ntemplates, among other things, mwparserfromhell acts as a direct interface to\nthe source code only. This has several implications:\n\n* Syntax elements produced by a template transclusion cannot be detected. For\n  example, imagine a hypothetical page ``\"Template:End-bold\"`` that contained\n  the text ``\u003c/b\u003e``. While MediaWiki would correctly understand that\n  ``\u003cb\u003efoobar{{end-bold}}`` translates to ``\u003cb\u003efoobar\u003c/b\u003e``, mwparserfromhell\n  has no way of examining the contents of ``{{end-bold}}``. Instead, it would\n  treat the bold tag as unfinished, possibly extending further down the page.\n\n* Templates adjacent to external links, as in ``http://example.com{{foo}}``,\n  are considered part of the link. In reality, this would depend on the\n  contents of the template.\n\n* When different syntax elements cross over each other, as in\n  ``{{echo|''Hello}}, world!''``, the parser gets confused because this cannot\n  be represented by an ordinary syntax tree. Instead, the parser will treat the\n  first syntax construct as plain text. In this case, only the italic tag would\n  be properly parsed.\n\n  **Workaround:** Since this commonly occurs with text formatting and text\n  formatting is often not of interest to users, you may pass\n  *skip_style_tags=True* to ``mwparserfromhell.parse()``. This treats ``''``\n  and ``'''`` as plain text.\n\n  A future version of mwparserfromhell may include multiple parsing modes to\n  get around this restriction more sensibly.\n\nAdditionally, the parser lacks awareness of certain wiki-specific settings:\n\n* `Word-ending links`_ are not supported, since the linktrail rules are\n  language-specific.\n\n* Localized namespace names aren't recognized, so file links (such as\n  ``[[File:...]]``) are treated as regular wikilinks.\n\n* Anything that looks like an XML tag is treated as a tag, even if it is not a\n  recognized tag name, since the list of valid tags depends on loaded MediaWiki\n  extensions.\n\nIntegration\n-----------\n\n``mwparserfromhell`` is used by and originally developed for EarwigBot_;\n``Page`` objects have a ``parse`` method that essentially calls\n``mwparserfromhell.parse()`` on ``page.get()``.\n\nIf you're using Pywikibot_, your code might look like this:\n\n.. code-block:: python\n\n    import mwparserfromhell\n    import pywikibot\n\n    def parse(title):\n        site = pywikibot.Site()\n        page = pywikibot.Page(site, title)\n        text = page.get()\n        return mwparserfromhell.parse(text)\n\nIf you're not using a library, you can parse any page with the following\nPython 3 code (using the API_ and the requests_ library):\n\n.. code-block:: python\n\n    import mwparserfromhell\n    import requests\n\n    API_URL = \"https://en.wikipedia.org/w/api.php\"\n\n    def parse(title):\n        params = {\n            \"action\": \"query\",\n            \"prop\": \"revisions\",\n            \"rvprop\": \"content\",\n            \"rvslots\": \"main\",\n            \"rvlimit\": 1,\n            \"titles\": title,\n            \"format\": \"json\",\n            \"formatversion\": \"2\",\n        }\n        headers = {\"User-Agent\": \"My-Bot-Name/1.0\"}\n        req = requests.get(API_URL, headers=headers, params=params)\n        res = req.json()\n        revision = res[\"query\"][\"pages\"][0][\"revisions\"][0]\n        text = revision[\"slots\"][\"main\"][\"content\"]\n        return mwparserfromhell.parse(text)\n\n.. _MediaWiki:              https://www.mediawiki.org\n.. _ReadTheDocs:            https://mwparserfromhell.readthedocs.io\n.. _Earwig:                 https://en.wikipedia.org/wiki/User:The_Earwig\n.. _Σ:                      https://en.wikipedia.org/wiki/User:%CE%A3\n.. _Legoktm:                https://en.wikipedia.org/wiki/User:Legoktm\n.. _GitHub:                 https://github.com/earwig/mwparserfromhell\n.. _Python Package Index:   https://pypi.org/\n.. _get pip:                https://pypi.org/project/pip/\n.. _pytest:                 https://docs.pytest.org/\n.. _Word-ending links:      https://www.mediawiki.org/wiki/Help:Links#linktrail\n.. _EarwigBot:              https://github.com/earwig/earwigbot\n.. _Pywikibot:              https://www.mediawiki.org/wiki/Manual:Pywikibot\n.. _API:                    https://www.mediawiki.org/wiki/API:Main_page\n.. _requests:               https://2.python-requests.org\n","funding_links":[],"categories":["Scraping tools","Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearwig%2Fmwparserfromhell","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fearwig%2Fmwparserfromhell","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearwig%2Fmwparserfromhell/lists"}