{"id":21008091,"url":"https://github.com/caged/dragnet","last_synced_at":"2026-04-21T22:33:50.965Z","repository":{"id":66029550,"uuid":"287325","full_name":"caged/dragnet","owner":"caged","description":"Attempts to extract readable content and embedded links from HTML markup \u0026 web pages","archived":false,"fork":false,"pushed_at":"2017-09-27T00:12:38.000Z","size":190,"stargazers_count":2,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-01-02T03:39:20.316Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/caged.png","metadata":{"files":{"readme":"README.markdown","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2009-08-25T06:03:31.000Z","updated_at":"2018-08-29T10:06:52.000Z","dependencies_parsed_at":"2023-03-10T23:29:55.722Z","dependency_job_id":null,"html_url":"https://github.com/caged/dragnet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/caged/dragnet","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caged%2Fdragnet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caged%2Fdragnet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caged%2Fdragnet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caged%2Fdragnet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/caged","download_url":"https://codeload.github.com/caged/dragnet/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/caged%2Fdragnet/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32113130,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-21T11:25:29.218Z","status":"ssl_error","status_checked_at":"2026-04-21T11:25:28.499Z","response_time":128,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-19T09:11:29.230Z","updated_at":"2026-04-21T22:33:50.906Z","avatar_url":"https://github.com/caged.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"#dragnet\nThis is still very experimental.  \n\n\n##Extracting readable content from HTML markup\nThis was inspired by the [Readability](http://lab.arc90.com/experiments/readability/) bookmarklet. \nThe goal is to extract meaningful, readable content from HTML.  This will attempt to \nextract content from sources such as blogs articles and publications.  It will also attempt to \nextract links embedded within the readable content.\n\n\n##Approach\nGiven the vast nasty of HTML tag soup regurgitated by some blog engines, it's very \nhard to get fully clean content from a page on any kind of consistent basis.  What works for one \nchunk of HTML, might not work for other chunks of HTML.  We want to attempt to get as clean of \ncontent as we can, but remain as abstract as possible.\n\nA basic overview of what this library does:\n\n1. Try to extract any hEntry microformat items from the page and return those.\n2. Collect all paragraphs (if none found, collect text nodes and/or divs as last resort)\n3. Iterate over the paragraphs, ascending up the hierarchy, scoring the parent based on \n   some common keywords and word count.\n\n  \n####Notable Troublesome URLS\n[x] Readability doesn't parse correctly : [-] Readability parses correctly\n\n* [-] [http://www.politico.com/blogs/glennthrush/0809/Remains_of_the_day_August_24_2009.html](http://www.politico.com/blogs/glennthrush/0809/Remains_of_the_day_August_24_2009.html)\n      Short content, invalid lines.\n* [-] [http://www.tulsaworld.com/news/article.aspx?subjectid=298\u0026articleid=20090824_298_0_TheTul317502\u0026rss_lnk=1](http://www.tulsaworld.com/news/article.aspx?subjectid=298\u0026articleid=20090824_298_0_TheTul317502\u0026rss_lnk=1)\n      No content paragraphs, uses spans and double \u003cbr /\u003e\u003cbr /\u003e to signify paragraphs.\n* [-] [http://article.nationalreview.com/?q=YTMxNGQyNmYyNjljYmE0NDVhZTdlMjlkZTM1Y2NiOTU=](http://article.nationalreview.com/?q=YTMxNGQyNmYyNjljYmE0NDVhZTdlMjlkZTM1Y2NiOTU=)\n      Multiple page article\n* [x] [http://politicalwire.com/archives/2009/08/24/bonus_quote_of_the_day.html](http://politicalwire.com/archives/2009/08/24/bonus_quote_of_the_day.html)\n      Tiny amount of content with no paragraphs and a huge footer with paragraph content.\n* [x] [http://briefingroom.thehill.com/2009/08/24/doctor-to-be-named-in-jackson-homicide-donated-to-republican-party-in-2004/](http://briefingroom.thehill.com/2009/08/24/doctor-to-be-named-in-jackson-homicide-donated-to-republican-party-in-2004/)\n      Paragraphs embedded in spans\n* [-] [http://www.nytimes.com/2009/08/25/us/politics/25detain.html?_r=1\u0026partner=rss\u0026emc=rss](http://www.nytimes.com/2009/08/25/us/politics/25detain.html?_r=1\u0026partner=rss\u0026emc=rss)\n      Not sure what the problem is here.  It doesn't look like anything should be wrong here \n      at first glance.\n* [-] [http://www.weeklystandard.com/weblogs/TWSFP/2009/08/kristol_gratitude_obamastyle.asp](http://www.weeklystandard.com/weblogs/TWSFP/2009/08/kristol_gratitude_obamastyle.asp)\n      Everything looks fine.  Not sure why we're getting a blank block of content back\n* [x] [http://www.cbsnews.com/stories/2009/08/24/entertainment/michaeljackson/main5262822.shtml](http://www.cbsnews.com/stories/2009/08/24/entertainment/michaeljackson/main5262822.shtml)\n      Comments are included as readable content\n* [x] [http://www.whitehouse.gov/blog/Diligence-on-H1N1/](http://www.whitehouse.gov/blog/Diligence-on-H1N1/)\n      Uses divs for content, but contains paragraphs elsewhere.\n* [-] [http://www.msnbc.msn.com/id/32518842/ns/meet_the_press/](http://www.msnbc.msn.com/id/32518842/ns/meet_the_press/)\n      Parsing a bunch of unwanted, non readable content. I assume because it shares the same\n      parent as the other content.\n      \n##TODO\n* Parsing multiple page articles\n* Consider searching for a 'print' link on the page and use this content instead.  This content \n  tends to be a cleaner version of the original and it also tends to bypass the multiple page \n  article issue.\n  \n##Note on Patches/Pull Requests\n* Fork the project.\n* Make your feature addition or bug fix.\n* Add tests for it. This is important so I don't break it in a future version unintentionally.\n* Send me a pull request. Bonus points for topic branches.\n\n\n##Copyright\nCopyright (c) 2009 Justin Palmer. See LICENSE for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaged%2Fdragnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcaged%2Fdragnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcaged%2Fdragnet/lists"}