{"id":21006207,"url":"https://github.com/derlin/phrasal","last_synced_at":"2026-02-10T09:03:56.516Z","repository":{"id":57452380,"uuid":"239772324","full_name":"derlin/phrasal","owner":"derlin","description":"A suite of tools for extracting proper sentences from HTML or text.","archived":false,"fork":false,"pushed_at":"2020-07-17T09:13:28.000Z","size":87,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-07-25T07:16:26.367Z","etag":null,"topics":["nlp"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-11T13:46:06.000Z","updated_at":"2024-06-22T02:46:04.000Z","dependencies_parsed_at":"2022-08-30T01:10:27.163Z","dependency_job_id":null,"html_url":"https://github.com/derlin/phrasal","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/derlin/phrasal","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fphrasal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fphrasal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fphrasal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fphrasal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derlin","download_url":"https://codeload.github.com/derlin/phrasal/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derlin%2Fphrasal/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272084826,"owners_count":24870584,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-25T02:00:12.092Z","response_time":1107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nlp"],"created_at":"2024-11-19T08:49:55.559Z","updated_at":"2026-02-10T09:03:51.473Z","avatar_url":"https://github.com/derlin.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Phrasal\n\n## Forewords \n\n### What is it ?\n\nPhrasal is a library of tools to help gather meaningful, proper sentences from websites. \n\nWell, at least if used together. Each tool has a value of its own. \nFor example, the `Normalizer` (my favorite!) is very useful for NLP, when you have a crappy text corpus you need to clean.\nThe `MocySplitter` is a nice alternative to Moses when you need to cleverly split a stream of text into sentences, one per line. \nEtc.\n\n### Why was it developed ?\n\nI have been working on a project lately, called [SwissText](https://github.com/derlin/swisstext) that gathers Swiss German sentences from scraping the Internet (no kidding, see the LREC 2020 publication on [arXiv](https://arxiv.org/abs/1912.00159)).\nTo do so, I had to build upon existing tools and develop some of my own. \nWhile they were initially for Swiss German, I figured that it would maybe be useful in other contexts, hence this repo which is a stripped-down version of some of the SwissText modules.\n\n### How does it work ?\n\nThis repo contains implementations of four types of tools, which constitute together a pipeline:\n\n1. *converter*: extract (main) text from raw HTML;\n2. *normalizer*: normalize the raw text, including the encoding, quotes, spaces, etc.;\n3. *splitter*: split the text into chunks (potential sentences);\n4. *filterer*: filter chunks to keep only \"proper\" sentences.\n\nFor each step, I propose one or more implementations.\n\n## Tools available\n\n**HtmlConverters**\n\n* `phrasal.BsConverter` \\\nA converter built upon `BeautifulSoup` that exact text found on the HTML. \nText from code blocks, scripts or styles is ignored.\nIt deals cleverly with encodings and always delivers text in UTF-8.\n\n\n* `phrasal.JustextConverter` \\\na converter based on [`justext`](https://pypi.org/project/jusText/), that try to spot and remove boilerplate content.\nBy default, it only keeps \"good\" paragraph, that is text long enough to be a full sentences and with a low link density.\n\n**Normalizers**\n\n* `phrasal.Normalizer`, or simply `phrasal.normalize_text`\\\nNormalize some text (using a serie of homemade regexes), including: normalize spaces, replace combining diacritics by the accented letter codepoints and strip leftovers, normalize dashes and quotemarks, replace non-breakable spaces with regular ones, etc. \\\nIt can also try to fix encoding errors (see [`ftfy`](https://pypi.org/project/ftfy/)) and strip most unicode emoji symbols.\n\n**Splitters**\n\n* `phrasal.MosesSplitter`\\\nMoses' splitter [`split-sentences.perl`](https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ems/support/split-sentences.perl) completely rewritten in Python. It thus perfectly mimics the behavior, while being 5x faster than calling perl from Python (approach taken by [`MosesTokenizer`](https://pypi.org/project/mosestokenizer/) for example).\n* `phrasal.MocySplitter`\\\nAn improvement upon `MosesSplitter`, which: deals more efficiently with lowercase (people are lazy on the Web), try to preserve links, can split on `:` or `;` (optional), etc.\n\n**Filterers**\n\n* `phrasal.PatternSentenceFilter`\\\nA filterer based on a list of simple rules a proper sentence should respect, such as \"*at least five words*\", \"*no S P E L L E D* words\", etc. \\\nWhat is *awesome* ? The rules are expressed in a (homemade) YAML-based syntax and are highly customizable. If you don't like the behavior, have a look at `pattern_sentence_filter.yaml` and try writing your own set of rules !\n\n\n**link_utils**\n\nThe `phrasal.link_utils` module is a simple utility to process href links found on a page. It will resolve relative links\n(given a base URL), remove duplicates, strip anchors and exclude non-HTTP/HTTPs links.\n\nTo get the list of links from a URL (i.e. `href` found on the page main content), use `extract_links`:\n```python\nimport phrasal\n\nall_links = phrasal.extract_links('https://github.com/derlin/phrasal')\n```\n\n## How to use\n\nInstall the library using:\n```bash\n# regular install, one of:\npython setup.py install \npip install .\n\n# for development, one of:\npython setup.py develop\npip install -e .\npip install -e .[showcase] # for streamlit\n```\n\n### As a library\n\n```python\nimport phrasal\n```\nDone.\n\n### From the command line\n\nEach tool contains a command line interface with different arguments. Discover it by typing:\n```bash\npython -m phrasal --help\n```\n```bash\npython -m phrasal --help\nCall one of the tools from the command line. Usage: \n   classname [other arguments specific to classname]|[-h]\n\nAllowed classname arguments:\n - BsConverter\n - JustextConverter\n - PatternSentenceFilter\n - MocySplitter\n - MosesSplitter\n - Normalizer\n```\nHere are some examples:\n```bash\npython -m phrasal JustextConverter -u https://icosys.ch/swisscrawl\n=== from URL https://icosys.ch/swisscrawl\nAs part of the SwissTranslation project, SwissCrawl is a corpus of 500,000+ Swiss German (GSW)  [...]\n[...]\n```\n```bash\npython -m phrasal PatternSentenceFilter -i \u003c(echo 'not-a-sentence\\nYEAH !!!\\nCet outil fonctionne très bien, je l’utilise tous les jours.')\nCet outil fonctionne très bien, je l’utilise tous les jours.\n```\n```bash\npython -m phrasal Normalizer -i raw_text.txt -o clean_text.txt\n```\n\n### I just need one tool...\n\nNo problem, each tool is more or less independent. \nYou may want to simplify the code a bit (e.g. remove the interface inheritance, transform classes into static scripts, I don't know), but I hope the source code is self-explaining. \n\n## Running tests\n\nTests are using `tox` and `pytest`. The easiest way to run them is:\n```bash\npip install tox tox-venv\ntox\n```\n\n## Running the showcase\n\nA showcase using [streamlit](https://www.streamlit.io/) is included. \nIt allows you to test the full pipeline straight from your browser and also play with the different tools and options\nfrom the **Live Customizer**. Once you found what works for you, you can simply copy-paste the code snippet generated.\n\nRun the showcase locally by doing:\n```bash\npip install streamlit\nstreamlit run src/showcase/lit.py\n```\n\n\n## License\n\nThis work is licensed under Apache 2.0, so you can basically do anything with it. \n\n*However*, I would **really enjoy** it if you **credit me** somehow, either by citing my name, send me an email to say hi (I get lonely sometime, may be nice to chat), leave a star on GitHub, or any other way you think may give me strength to keep doing open-source :blush:.\n\n## Related resources\n\n* [get-html](https://pypi.org/project/get-html/) to get raw or renderer HTML (used in this repo)\n* [SwissText](https://github.com/swisstext)\n* [SwissTranslation project page](https://icosys.ch/swisscrawl)\n* :octopus::octopus::octopus::octopus::octopus::octopus::octopus::octopus: (I just love octopuses)\n* [Personal website](https://derlin.ch)\n\n## TODO\n\n* add some usecases, such as finding links, cleaning a text file, etc. add language support information ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Fphrasal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderlin%2Fphrasal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderlin%2Fphrasal/lists"}