{"id":13449032,"url":"https://github.com/bookieio/breadability","last_synced_at":"2025-10-21T20:08:56.293Z","repository":{"id":3177846,"uuid":"4209468","full_name":"bookieio/breadability","owner":"bookieio","description":"Reworked https://www.readability.com/ parsing library (now https://mercury.postlight.com/ is living alternative)","archived":false,"fork":false,"pushed_at":"2024-05-09T16:16:34.000Z","size":618,"stargazers_count":205,"open_issues_count":15,"forks_count":25,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-09-21T11:17:17.610Z","etag":null,"topics":["html-extraction","html-extractor","html-parsing","python","text-extraction","text-mining"],"latest_commit_sha":null,"homepage":"https://bookieio.github.io/breadability/","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bookieio.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE.rst","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.txt","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-05-03T01:37:00.000Z","updated_at":"2025-09-09T14:11:51.000Z","dependencies_parsed_at":"2024-06-18T15:35:31.423Z","dependency_job_id":"f098e504-44bd-41eb-b761-004ce33b71af","html_url":"https://github.com/bookieio/breadability","commit_stats":{"total_commits":219,"total_committers":7,"mean_commits":"31.285714285714285","dds":0.4840182648401826,"last_synced_commit":"95a364c43b00baf6664bea1997a7310827fb1ee9"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/bookieio/breadability","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookieio%2Fbreadability","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookieio%2Fbreadability/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookieio%2Fbreadability/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookieio%2Fbreadability/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bookieio","download_url":"https://codeload.github.com/bookieio/breadability/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bookieio%2Fbreadability/sbom","scorecard":{"id":247795,"data":{"date":"2025-08-11","repo":{"name":"github.com/bookieio/breadability","commit":"2285531110262ba09708c8f2054d365f79915995"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.9,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":2,"reason":"Found 5/21 approved changesets -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.rst:0","Info: FSF or OSI recognized license: BSD 2-Clause \"Simplified\" License: LICENSE.rst:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":4,"reason":"6 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-55x5-fj6c-h6m8","Warn: Project is vulnerable to: PYSEC-2014-9 / GHSA-57qw-cc2g-pv5p","Warn: Project is vulnerable to: PYSEC-2021-19 / GHSA-jq4v-f5q6-mjqq","Warn: Project is vulnerable to: GHSA-pgww-xf46-h92r","Warn: Project is vulnerable to: PYSEC-2022-230 / GHSA-wrxv-2j5q-m38w","Warn: Project is vulnerable to: PYSEC-2018-12 / GHSA-xp26-p53h-6h2p"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 15 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T07:55:25.938Z","repository_id":3177846,"created_at":"2025-08-17T07:55:25.938Z","updated_at":"2025-08-17T07:55:25.938Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280325342,"owners_count":26311426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["html-extraction","html-extractor","html-parsing","python","text-extraction","text-mining"],"created_at":"2024-07-31T06:00:29.077Z","updated_at":"2025-10-21T20:08:56.270Z","avatar_url":"https://github.com/bookieio.png","language":"HTML","funding_links":[],"categories":["HTML"],"sub_categories":[],"readme":"breadability - another readability Python (v2.6-v3.3) port\n==========================================================\n.. image:: https://api.travis-ci.org/bookieio/breadability.png?branch=master\n   :target: https://travis-ci.org/bookieio/breadability\n\nI've tried to work with the various forks of some ancient codebase that ported\n`readability`_ to Python. The lack of tests, unused regex's, and commented out\nsections of code in other Python ports just drove me nuts.\n\nI put forth an effort to bring in several of the better forks into one\ncode base, but they've diverged so much that I just can't work with it.\n\nSo what's any sane person to do? Re-port it with my own repo, add some tests,\ninfrastructure, and try to make this port better. OSS FTW (and yea, NIH FML,\nbut oh well I did try)\n\nThis is a pretty straight port of the JS here:\n\n- http://code.google.com/p/arc90labs-readability/source/browse/trunk/js/readability.js#82\n- http://www.minvolai.com/blog/decruft-arc90s-readability-in-python/\n\n\nAlternatives\n------------\n\n- https://github.com/codelucas/newspaper\n- https://github.com/grangier/python-goose\n- https://github.com/aidanf/BTE\n- http://www.unixuser.org/~euske/python/webstemmer/#extract\n- https://github.com/al3xandru/readability.py\n- https://github.com/rcarmo/soup-strainer\n- https://github.com/bcampbell/decruft\n- https://github.com/gfxmonk/python-readability\n- https://github.com/srid/readability\n- https://github.com/dcramer/decruft\n- https://github.com/reorx/readability\n- https://github.com/mote/python-readability\n- https://github.com/predatell/python-readability-lxml\n- https://github.com/Harshavardhana/boilerpipy\n- https://github.com/raptium/hitomi\n- https://github.com/kingwkb/readability\n\n\nInstallation\n------------\nThis does depend on lxml so you'll need some C headers in order to install\nthings from pip so that it can compile.\n\n.. code-block:: bash\n\n    $ [sudo] apt-get install libxml2-dev libxslt-dev\n    $ [sudo] pip install git+git://github.com/bookieio/breadability.git\n\nTests\n-----\n.. code-block:: bash\n\n    $ pytest tests\n\n\nUsage\n-----\nCommand line\n~~~~~~~~~~~~\n\n.. code-block:: bash\n\n    $ breadability http://wiki.python.org/moin/BeginnersGuide\n\nOptions\n```````\n\n- **b** will write out the parsed content to a temp file and open it in a\n  browser for viewing.\n- **d** will write out debug scoring statements to help track why a node was\n  chosen as the document and why some nodes were removed from the final\n  product.\n- **f** will override the default behaviour of getting an html fragment (\u003cdiv\u003e)\n  and give you back a full \u003chtml\u003e document.\n- **v** will output in verbose debug mode and help let you know why it parsed\n  how it did.\n\n\nPython API\n~~~~~~~~~~\n.. code-block:: python\n\n    from __future__ import print_function\n\n    from breadability.readable import Article\n\n\n    if __name__ == \"__main__\":\n        document = Article(html_as_text, url=source_url)\n        print(document.readable)\n\n\nWork to be done\n---------------\nYep, I've got some catching up to do. I don't do pagination, I've got a lot of\ncustom tweaks I need to get going, there are some articles that fail to parse.\nI also have more tests to write on a lot of the cleaning helpers, but\nhopefully things are setup in a way that those can/will be added.\n\nFortunately, I need this library for my tools:\n\n- https://bmark.us\n- http://r.bmark.us\n\nso I really need this to be an active and improving project.\n\n\nOff the top of my heads TODO list:\n\n- Support metadata from parsed article [url, confidence scores, all\n  candidates we thought about?]\n- More tests, more thorough tests\n- More sample articles we need to test against in the test_articles\n- Tests that run through and check for regressions of the test_articles\n- Tidy'ing the HTML that comes out, might help with regression tests ^^\n- Multiple page articles\n- Performance tuning, we do a lot of looping and re-drop some nodes that\n  should be skipped. We should have a set of regression tests for this so\n  that if we implement a change that blows up performance we know it right\n  away.\n- More docs for things, but sphinx docs and in code comments to help\n  understand wtf we're doing and why. That's the biggest hurdle to some of\n  this stuff.\n\n\nInspiration\n~~~~~~~~~~~\n\n- `python-readability`_\n- `decruft`_\n- `readability`_\n\n\n\n.. _readability: http://code.google.com/p/arc90labs-readability/\n.. _TravisCI: http://travis-ci.org/\n.. _decruft: https://github.com/dcramer/decruft\n.. _python-readability: https://github.com/buriy/python-readability\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbookieio%2Fbreadability","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbookieio%2Fbreadability","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbookieio%2Fbreadability/lists"}