{"id":39479320,"url":"https://github.com/tech-engine/goscrapy","last_synced_at":"2026-01-18T05:01:36.315Z","repository":{"id":186335554,"uuid":"671366083","full_name":"tech-engine/goscrapy","owner":"tech-engine","description":"GoScrapy: Harnessing Go's power for blazingly fast web scraping, inspired by Python's Scrapy framework.","archived":false,"fork":false,"pushed_at":"2024-12-13T13:26:45.000Z","size":6457,"stargazers_count":89,"open_issues_count":1,"forks_count":2,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-12-13T14:26:30.266Z","etag":null,"topics":["data-extraction","go-scrapy","golang","goscraper","scrapy","spider","web-crawler","webscraper","webscrapping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tech-engine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-27T06:47:50.000Z","updated_at":"2024-12-13T13:26:17.000Z","dependencies_parsed_at":"2023-10-16T19:31:10.816Z","dependency_job_id":"25ff25c6-e093-450c-b489-57ae7960ff6b","html_url":"https://github.com/tech-engine/goscrapy","commit_stats":null,"previous_names":["tech-engine/goscrapy"],"tags_count":53,"template":false,"template_full_name":null,"purl":"pkg:github/tech-engine/goscrapy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tech-engine%2Fgoscrapy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tech-engine%2Fgoscrapy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tech-engine%2Fgoscrapy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tech-engine%2Fgoscrapy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tech-engine","download_url":"https://codeload.github.com/tech-engine/goscrapy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tech-engine%2Fgoscrapy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28530621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-extraction","go-scrapy","golang","goscraper","scrapy","spider","web-crawler","webscraper","webscrapping"],"created_at":"2026-01-18T05:01:16.522Z","updated_at":"2026-01-18T05:01:36.195Z","avatar_url":"https://github.com/tech-engine.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GoScrapy: Web Scraping Framework in Go\r\n [![Alt Text](https://goreportcard.com/badge/github.com/tech-engine/goscrapy)](https://github.com/tech-engine/goscrapy)\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg width=\"800\" src=\"./assets/logo.webp\"\u003e\r\n\u003c/p\u003e\r\n\r\n**GoScrapy** aims to be a powerful web scraping framework in Go, inspired by Python's Scrapy framework. It offers an easy-to-use Scrapy-like experience for extracting data from websites, making it an ideal tool for various data collection and analysis tasks, especially for those coming from Python and wanting to try scraping in Golang..\r\n\r\n## Getting Started\r\n\r\nGoscrapy requires **Go version 1.22** or higher to run.\r\n\r\n### 1: Project Initialization\r\n\r\n```sh\r\ngo mod init books_to_scrape\r\n```\r\n\r\n### 2. Install goscrapy cli\r\n\r\n```sh\r\ngo install github.com/tech-engine/goscrapy@latest\r\n```\r\n**Note**: make sure to always keep your goscrapy cli updated.\r\n\r\n### 3. Verify Installation\r\n\r\n```sh\r\ngoscrapy -v\r\n```\r\n### 4. Create a New Project\r\n\r\n```sh\r\ngoscrapy startproject books_to_scrape\r\n```\r\nThis will create a new project directory with all the files necessary to begin working with **GoScrapy**.\r\n\r\n```sh\r\n\\iyuioy\\go\\go-test-scrapy\u003e goscrapy startproject books_to_scrape\r\n\r\n🚀 GoScrapy generating project files. Please wait!\r\n\r\n✔️  books_to_scrape\\constants.go\r\n✔️  books_to_scrape\\errors.go\r\n✔️  books_to_scrape\\job.go\r\n✔️  main.go\r\n✔️  books_to_scrape\\record.go\r\n✔️  books_to_scrape\\spider.go\r\n\r\n✨ Congrates. books_to_scrape created successfully.\r\n```\r\n\r\n### spider.go\r\nIn your __`spider.go`__ file, set up and execute your spider.\r\n\r\nFor detailed code, please refer to the [sample code here](./_examples/scrapejsp_method2/scrapejsp/spider.go).\r\n\r\n```go\r\npackage scrapejsp\r\n\r\nimport (\r\n\t\"context\"\r\n\t\"encoding/json\"\r\n\t\"fmt\"\r\n\t\"log\"\r\n\r\n\t\"github.com/tech-engine/goscrapy/cmd/gos\"\r\n\t\"github.com/tech-engine/goscrapy/pkg/core\"\r\n)\r\n\r\ntype Spider struct {\r\n\tgos.ICoreSpider[*Record]\r\n}\r\n\r\nfunc NewSpider(ctx context.Context) (*Spider, \u003c-chan error) {\r\n\r\n\t// use proxies\r\n\t// proxies := core.WithProxies(\"proxy_url1\", \"proxy_url2\", ...)\r\n\t// core := gos.New[*Record]().WithClient(\r\n\t// \tgos.DefaultClient(proxies),\r\n\t// )\r\n\r\n\tcore := gos.New[*Record]()\r\n\r\n\t// Add middlewares\r\n\tcore.MiddlewareManager.Add(MIDDLEWARES...)\r\n\t// Add pipelines\r\n\tcore.PipelineManager.Add(PIPELINES...)\r\n\r\n\terrCh := make(chan error)\r\n\r\n\tgo func() {\r\n\t\terrCh \u003c- core.Start(ctx)\r\n\t}()\r\n\r\n\treturn \u0026Spider{\r\n\t\tcore,\r\n\t}, errCh\r\n}\r\n\r\n// This is the entrypoint to the spider\r\nfunc (s *Spider) StartRequest(ctx context.Context, job *Job) {\r\n\r\n\treq := s.NewRequest()\r\n\t// req.Meta(\"JOB\", job)\r\n\treq.Url(\"https://jsonplaceholder.typicode.com/todos/1\")\r\n\r\n\ts.Request(req, s.parse)\r\n}\r\n\r\nfunc (s *Spider) Close(ctx context.Context) {\r\n}\r\n\r\nfunc (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {\r\n\tfmt.Printf(\"status: %d\", resp.StatusCode())\r\n\r\n\tvar data Record\r\n\terr := json.Unmarshal(resp.Bytes(), \u0026data)\r\n\tif err != nil {\r\n\t\tlog.Fatalln(err)\r\n\t}\r\n\r\n\t// to push to pipelines\r\n\ts.Yield(\u0026data)\r\n}\r\n```\r\n\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg width=\"600\" src=\"./assets/demo.gif\"\u003e\r\n\u003c/p\u003e\r\n\r\n## Wiki\r\nPlease follow the [wiki](https://github.com/tech-engine/goscrapy/wiki) docs for details.\r\n\r\n### Note\r\n\r\n**GoScrapy** is not stable, so its API may change drastically. Please exercise caution when using it in production.\r\n\r\n## License\r\n\r\n**GoScrapy** is available under the BSL with an additional usage grant that allows free internal use. Please ensure that you agree with the license before contributing to **GoScrapy**, as by contributing to the GoScrapy project, you agree to the terms of the license.\r\n\r\n## Roadmap\r\n\r\n- ~~Cookie management~~\r\n- ~~Builtin \u0026 Custom Middlewares support~~\r\n- ~~Css \u0026 Xpath Selectors~~\r\n- Logging\r\n- Triggers\r\n- Tests(work in progress)\r\n\r\n## Partners\r\n\r\n\u003ca href=\"https://dashboard.mangoproxy.com/signup?promo=v7omc7\"\u003e\r\n\t\u003cimg src=\"https://mangoproxy.com/assetsfile/images/logomango.webp\" width=\"200\"\u003e\r\n\u003c/a\u003e\r\n\r\n## Get in touch\r\n[Discord](https://discord.gg/FPvxETjYPH)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftech-engine%2Fgoscrapy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftech-engine%2Fgoscrapy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftech-engine%2Fgoscrapy/lists"}