https://github.com/siznax/frag2text
Get the plain text of web fragments
https://github.com/siznax/frag2text
Last synced: about 2 months ago
JSON representation
Get the plain text of web fragments
- Host: GitHub
- URL: https://github.com/siznax/frag2text
- Owner: siznax
- License: mit
- Created: 2015-01-07T20:42:36.000Z (over 10 years ago)
- Default Branch: master
- Last Pushed: 2023-05-22T21:33:39.000Z (about 2 years ago)
- Last Synced: 2025-04-12T19:41:17.731Z (3 months ago)
- Language: Python
- Homepage:
- Size: 19.5 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.rst
- Changelog: HISTORY.rst
- License: LICENSE
Awesome Lists containing this project
README
frag2text
=========
.. image:: https://img.shields.io/pypi/v/frag2text.svg
:target: https://pypi.python.org/pypi/frag2text/
Markdown_ gives you HTML from plain text and html2text_ reverses the
process. If you want the plain text version of *a specific section* of
a web page (an HTML fragment), you would normally do the selecting
(parsing) first, then generate the Markdown text to preserve some
formatting.
I made *frag2text* to:
* easily select a web page fragment by CSS selector or XPath
expression
* get the plain text of the fragment with some formatting intact for
later use with Markdown
* not shell out to a another program (like lynx -dump)
* not parse HTML or text directly
* use html5lib for robust parsing
* have a simple python module that is easy to maintain (encapsulating
the tricky business of subclassing parsers, treebuilders and
serializers, by keeping it simple – this shouldn't require
BeautifulSoup)
This is a problem I've tinkered with for some time and solved in many
different ways. It seems trivial but gets ridiculous quickly. If you
have any suggestions or want to share your experiences with other
tools, please let me know.
Install
=======
.. code-block:: shell
$ pip install frag2text
Usage
=====
python
------
.. code-block:: python
>>> from frag2text import frag2text
>>> help(frag2text)
Help on function frag2text in module frag2text:
frag2text(endpoint, stype, selector, clean=False, raw=False, verbose=False)
returns Markdown text of selected fragment.
Args:
endpoint: URL, file, or HTML string
stype: { 'css' | 'xpath' }
selector: CSS selector or XPath expression
Returns:
Markdown text
Options:
clean: cleans fragment (lxml.html.clean defaults)
raw: returns raw HTML fragment
verbose: show http status, encoding, headers
shell
-----
.. code-block:: shell
$ frag2text.py -h
usage: frag2text.py [-h] [-c] [-r] [-v] endpoint {css,xpath} selector
reverse Markdown (html2text) HTML fragments.
positional arguments:
endpoint URL, file, or HTML string
{css,xpath} fragment selector type
selector CSS select statement or XPath expression
optional arguments:
-h, --help show this help message and exit
-c, --clean clean fragment (lxml.html.clean defaults)
-r, --raw output raw fragment
-v, --verbose print status, encoding, headers
Examples
========
python
------
.. code-block:: python
from frag2text import frag2text
info = frag2text('http://wikipedia.org/wiki/Amanita', 'css', '.infobox')
shell
-----
.. code-block:: shell
$ frag2text.py "
hello" xpath //h1
...
# hello
CSS select
----------
.. code-block:: shell
$ frag2text.py http://wikipedia.org/wiki/Amanita css .infobox
_Amanita_
---
_[Amanita muscaria](/wiki/Amanita_muscaria)_
Albin Schmalfuß, 1897
[Scientific classification](/wiki/Biological_classification)
Kingdom: | [Fungi](/wiki/Fungi)
Division: | [Basidiomycota](/wiki/Basidiomycota)
Class: | [Agaricomycetes](/wiki/Agaricomycetes)
Order: | [Agaricales](/wiki/Agaricales)
Family: | [Amanitaceae](/wiki/Amanitaceae)
Genus: | _**Amanita**_
[Pers.](/wiki/Christian_Hendrik_Persoon) (1794)
[Type species](/wiki/Type_species)
_[Amanita muscaria](/wiki/Amanita_muscaria)_
([L.](/wiki/Linnaeus)) [Lam.](/wiki/Lam.) (1783)
[Diversity](/wiki/Biodiversity)
[c.600 species](/wiki/List_of_Amanita_species)
XPath expression
----------------
.. code-block:: shell
$ frag2text.py http://en.wikipedia.org/wiki/Amanita xpath '//p[1]'
The [genus](/wiki/Genus) _**Amanita**_ contains about 600 [species](/wik
i/Species) of [agarics](/wiki/Agarics) including some of the most [toxic
](/wiki/Toxic) known [mushrooms](/wiki/Mushrooms) found worldwide, as we
ll as some well-regarded edible species. This genus is responsible for a
pproximately 95% of the fatalities resulting from [mushroom poisoning](/
wiki/Mushroom_poisoning), with the [death cap](/wiki/Death_cap) accounti
ng for about 50% on its own. The most potent toxin present in these mush
rooms is α[-amanitin](/wiki/%CE%91-amanitin).
.. _Markdown: https://github.com/waylan/Python-Markdown
.. _html2text: https://github.com/Alir3z4/html2text/