{"id":15687523,"url":"https://github.com/bmoscon/articleparse","last_synced_at":"2025-05-07T18:10:10.072Z","repository":{"id":11724625,"uuid":"14246907","full_name":"bmoscon/ArticleParse","owner":"bmoscon","description":"Heuristic text extraction from news sites in Python3","archived":false,"fork":false,"pushed_at":"2017-12-31T23:44:17.000Z","size":30,"stargazers_count":10,"open_issues_count":3,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-05-07T18:10:03.596Z","etag":null,"topics":["analysis","boilerplate-removal","heuristics","python","text-analysis","text-extraction"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bmoscon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-08T23:07:42.000Z","updated_at":"2025-02-06T04:15:14.000Z","dependencies_parsed_at":"2022-09-02T22:10:40.412Z","dependency_job_id":null,"html_url":"https://github.com/bmoscon/ArticleParse","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmoscon%2FArticleParse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmoscon%2FArticleParse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmoscon%2FArticleParse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bmoscon%2FArticleParse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bmoscon","download_url":"https://codeload.github.com/bmoscon/ArticleParse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252931555,"owners_count":21827112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","boilerplate-removal","heuristics","python","text-analysis","text-extraction"],"created_at":"2024-10-03T17:49:27.688Z","updated_at":"2025-05-07T18:10:10.024Z","avatar_url":"https://github.com/bmoscon.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"ArticleParse\n============\n\n[![License](https://img.shields.io/badge/license-XFree86-blue.svg)](LICENSE)\n\n\nLibrary that strips boilerplate HTML from news articles and performs heuristic analysis to determine the body of the article. Ranks text sections of the website by probability of being news content.\n\n\nCurrently uses for analysis:\n\n* Section Length\n* Section Position\n* Number of Anchors in a Section\n* Anchor Density in a Section\n* Word Count\n* Uppercase Word Count\n* Average Word Length\n* Average Sentence Length\n* Number of Sentences\n\nThis is a work in progress. I have manually tested it on several news websites, but extensive testing still needs to be performed.\n\nSupports Python3\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbmoscon%2Farticleparse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbmoscon%2Farticleparse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbmoscon%2Farticleparse/lists"}