{"id":37115349,"url":"https://github.com/koshqua/scrapio","last_synced_at":"2026-01-14T13:32:49.771Z","repository":{"id":57520614,"uuid":"240511021","full_name":"Koshqua/scrapio","owner":"Koshqua","description":"Simple and easy-to-use scraper and crawler in Go. ","archived":false,"fork":false,"pushed_at":"2020-05-04T23:55:52.000Z","size":9880,"stargazers_count":13,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-03T00:02:18.139Z","etag":null,"topics":["crawler","framework","go","golang","json","scraper","spider"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Koshqua.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-14T13:10:27.000Z","updated_at":"2025-07-23T16:02:10.000Z","dependencies_parsed_at":"2022-09-26T18:01:04.156Z","dependency_job_id":null,"html_url":"https://github.com/Koshqua/scrapio","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Koshqua/scrapio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koshqua%2Fscrapio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koshqua%2Fscrapio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koshqua%2Fscrapio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koshqua%2Fscrapio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Koshqua","download_url":"https://codeload.github.com/Koshqua/scrapio/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Koshqua%2Fscrapio/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28421241,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T13:30:50.153Z","status":"ssl_error","status_checked_at":"2026-01-14T13:29:08.907Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","framework","go","golang","json","scraper","spider"],"created_at":"2026-01-14T13:32:49.208Z","updated_at":"2026-01-14T13:32:49.764Z","avatar_url":"https://github.com/Koshqua.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/66c67645f9fd404bbf47a3f443ecba5c)](https://app.codacy.com/manual/Koshqua/scrapio?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=Koshqua/scrapio\u0026utm_campaign=Badge_Grade_Dashboard)\n[![GoDoc](https://godoc.org/github.com/koshqua/scrapio?status.svg)](https://pkg.go.dev/github.com/koshqua/scrapio)\n[![Go Report Card](https://goreportcard.com/badge/github.com/Koshqua/scrapio)](https://goreportcard.com/report/github.com/Koshqua/scrapio)\n\n## Scrapio \n\n**Scrapio** - is a lightweight and user-friendy web crawling and scraping library. \nThe main goal of creating the project was to make scraping big amounts of similar data from web easy and user-friendly. It might be useful for wide range of applications, like data mining, data processing and archiving. \nAfter some time, I am going to make it a standalone service, which will work as an API.\n\n### Installation \n\n\n\n\n### Features\nAt the moment works as a library which can be used to crawl and scrap data from web. \nWhat it can do:\n- Crawl all pages on host, return all the links. \n- Scrap text, image urls and links from Crawl Result pages. \n- It leaves the choice of data output(csv,json, etc) up to you. \n- It's free and quite powerful. \n- Written in go, concurrent, depending on Network Speed can crawl and scrap up to 2k pages/minute.\n  \n### Installation \n```\ngo get github.com/koshqua/scrapio \n```\n\n### Usage \n**Crawler** is easy to use. You just need to specify a starting URL and it will crawl all the URL on the host.\n\n```go \n    //init a new crawler, give it a start url, it's not necessary should be basic URL\n    cr := \u0026crawler.Crawler{StartURL: \"https://gulfnews.com/\"}\n    //Start crawling func. \n    //After some time im going to implement more configs for this func, like max results, etc.\n    cr.Crawl()\n    //Do something with result, it's up to you\n```\n**Scraper** uses data structure given by crawler. \nBefore initiating a scraper, you need to create a few selectors, to assign them to scraper.\nSelectors are the simple css-like selectors.  \n```go\n    //create some Selectors, which you want to scrap.\n    h2 := scraper.NewSelector(\"h2\", true, true, true)\n    img := scraper.NewSelector(\"img\", true, true, true)\n    p := scraper.NewSelector(\"p:first-of-type\", true, true, true)\n    //Initiate a new scrapper with given selectors\n    //Scraper depends on the crawler from previous code snippet.\n    //It gets pages and creates new structure with selectors and scrap results.\n    sc := scraper.InitScraper(*cr, []scraper.Selector{h2, img, p})\n    //And just start scraping\n\terr := sc.Scrap()\n\tif err != nil {\n\t\tlog.Fatalln(err)\n\t}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoshqua%2Fscrapio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoshqua%2Fscrapio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoshqua%2Fscrapio/lists"}