{"id":25010579,"url":"https://github.com/rogelioolarte/newsextractor","last_synced_at":"2026-04-17T12:32:30.345Z","repository":{"id":212854831,"uuid":"732461797","full_name":"rogelioolarte/NewsExtractor","owner":"rogelioolarte","description":"API designed to extract large amounts of articles from any URL or website supported the use of CSS selectors documented with Swagger (OpenAPI 3).","archived":false,"fork":false,"pushed_at":"2024-05-19T00:19:28.000Z","size":117,"stargazers_count":0,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-30T02:33:32.081Z","etag":null,"topics":["api","api-rest","extract","extractor","java","java17","jpa","jsoup","lombok","lombok-maven","mysql","news","news-extractor","openapi3","spring-boot","springweb","swagger"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rogelioolarte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-16T18:48:21.000Z","updated_at":"2024-05-19T00:21:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"ba99b27a-e61b-4dac-9c81-7920d3526484","html_url":"https://github.com/rogelioolarte/NewsExtractor","commit_stats":null,"previous_names":["rogelioolarte/newsextractor"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/rogelioolarte/NewsExtractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogelioolarte%2FNewsExtractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogelioolarte%2FNewsExtractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogelioolarte%2FNewsExtractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogelioolarte%2FNewsExtractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rogelioolarte","download_url":"https://codeload.github.com/rogelioolarte/NewsExtractor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rogelioolarte%2FNewsExtractor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31929602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-17T10:35:34.458Z","status":"ssl_error","status_checked_at":"2026-04-17T10:35:09.472Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","api-rest","extract","extractor","java","java17","jpa","jsoup","lombok","lombok-maven","mysql","news","news-extractor","openapi3","spring-boot","springweb","swagger"],"created_at":"2025-02-05T04:53:46.717Z","updated_at":"2026-04-17T12:32:30.326Z","avatar_url":"https://github.com/rogelioolarte.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# NewsExtractor v 1.0.0\n\nAPI dedicated to extract news  using CSS query selectors documented with OpenAPI 3.\n\n## About\n\nThe API uses a pattern using [CSS query selectors](https://jsoup.org/cookbook/extracting-data/selector-syntax) to extract news using a maximum of three phases, which can be reduced as needed.\n\nThis API used the following dependencies:\n\n* [SpringBoot - Maven](https://spring.io/)\n    - Lombox \n    - Validation\n    - Spring Data JPA\n    - Spring Web\n    - MySQL Driver\n* [SpringDoc-OpenAPI](https://springdoc.org/)\n* [Jsoup](https://jsoup.org/)\n\n## Pre-requisites\n* [Java JDK v17.0.9](https://www.oracle.com/java/technologies/downloads/#java17)\n* [Maven 3](https://maven.apache.org/download.cgi)\n\n## Usage\n\n* Important: Check the ApplicationConfig.java, because the database-config use environment variables.\n\n### Install the prerequisites, then in the folder of the project: \n    - mvn clean install\n    - java -cp target/NewsExtractor.jar\n\nNext, you check [This endpoint](http://localhost:8080/swagger-ui/index.html) \n\n\n## Features:\n    - You can use a add specific sections.\n    - You can use a add specific articles or article sources.\n    - You can search for words in common in all newspaper articles.\n    - If the page to be extracted does not have connection problems,\n      it is possible to extract and save up to 0.80 seconds per article.\n\n## Note:\nMinimal use requires a newspaper source and a pattern with at least one specific selector.\n\n## FAQ:\n\n- Does the API work completely?\nDue to the fact that tests have not yet been integrated, some pages without the correct adjustment, it will not be possible to completely extract all the news for now because some pages limit requests or have sources that take a long time to return a response.\n\n- If you have a suggestion or advice, feel free to send me an email.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogelioolarte%2Fnewsextractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frogelioolarte%2Fnewsextractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frogelioolarte%2Fnewsextractor/lists"}