{"id":42970890,"url":"https://github.com/gosom/scrapemate","last_synced_at":"2026-01-31T00:52:55.181Z","repository":{"id":144887192,"uuid":"612944773","full_name":"gosom/scrapemate","owner":"gosom","description":"Golang Crawling and scraping framework","archived":false,"fork":false,"pushed_at":"2026-01-06T08:18:28.000Z","size":19382,"stargazers_count":188,"open_issues_count":5,"forks_count":23,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-01-12T16:36:33.799Z","etag":null,"topics":["crawler","go","go-framework","golang","scraper","spider","web-crawler","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gosom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-03-12T13:05:32.000Z","updated_at":"2026-01-06T08:17:27.000Z","dependencies_parsed_at":"2026-01-07T05:06:52.799Z","dependency_job_id":null,"html_url":"https://github.com/gosom/scrapemate","commit_stats":null,"previous_names":[],"tags_count":48,"template":false,"template_full_name":null,"purl":"pkg:github/gosom/scrapemate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gosom%2Fscrapemate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gosom%2Fscrapemate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gosom%2Fscrapemate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gosom%2Fscrapemate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gosom","download_url":"https://codeload.github.com/gosom/scrapemate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gosom%2Fscrapemate/sbom","scorecard":{"id":441787,"data":{"date":"2025-08-11","repo":{"name":"github.com/gosom/scrapemate","commit":"9ea0c8bdd8b3fbd0c09e5c2fee2a3d7d53929097"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Token-Permissions","score":10,"reason":"GitHub workflow tokens follow principle of least privilege","details":["Info: found token with 'none' permissions: .github/workflows/build.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":0,"reason":"1 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 1/22 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/gosom/scrapemate/build.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/build.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/gosom/scrapemate/build.yml/main?enable=pin","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 15 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":2,"reason":"8 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2025-3485 / GHSA-c6gw-w398-hv78","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw","Warn: Project is vulnerable to: GO-2024-3321 / GHSA-v778-237x-gjrc","Warn: Project is vulnerable to: GO-2025-3487 / GHSA-hcg3-q754-cr77","Warn: Project is vulnerable to: GO-2025-3754 / GHSA-2x5j-vhc8-9cwm","Warn: Project is vulnerable to: GO-2025-3787 / GHSA-fv92-fjc5-jj9h"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-19T05:45:18.134Z","repository_id":144887192,"created_at":"2025-08-19T05:45:18.135Z","updated_at":"2025-08-19T05:45:18.135Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28925174,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T22:32:35.345Z","status":"ssl_error","status_checked_at":"2026-01-30T22:32:31.927Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","go","go-framework","golang","scraper","spider","web-crawler","web-scraping"],"created_at":"2026-01-31T00:52:54.532Z","updated_at":"2026-01-31T00:52:55.163Z","avatar_url":"https://github.com/gosom.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# scrapemate\n[![Documentation](https://img.shields.io/badge/Documentation-Read%20Here-blue)](https://gosom.github.io/scrapemate)\n[![GoDoc](https://godoc.org/github.com/gosom/scrapemate?status.svg)](https://godoc.org/github.com/gosom/scrapemate)\n![build](https://github.com/gosom/scrapemate/actions/workflows/build.yml/badge.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/gosom/scrapemate)](https://goreportcard.com/report/github.com/gosom/scrapemate)\n\n[Scrapemate](https://gosom.github.io/scrapemate) is a web crawling and scraping framework written in Golang. It is designed to be simple and easy to use, yet powerful enough to handle complex scraping tasks.\n\n\n## Features\n\n- Low level API \u0026 Easy High Level API\n- Customizable retry and error handling\n- Javascript Rendering with ability to control the browser\n- Screenshots support (when JS rendering is enabled)\n- Capability to write your own result exporter\n- Capability to write results in multiple sinks\n- Default CSV writer\n- Caching (File/LevelDB/Custom)\n- Custom job providers (memory provider included)\n- Headless and Headful support when using JS rendering\n- Automatic cookie and session handling\n- Rotating HTTP/HTTPS/SOCKS5 proxy support\n\n## Browser Engines\n\nScrapemate supports two browser engines for JavaScript rendering: **Playwright** (default) and **Rod**. The browser engine is selected at compile time using Go build tags.\n\n### Playwright (Default)\n\nPlaywright is used by default when no build tags are specified. It requires the Playwright browsers to be installed.\n\n```bash\n# Install playwright browsers\ngo run github.com/playwright-community/playwright-go/cmd/playwright install --with-deps chromium\n```\n\nBuild and run without any special tags:\n\n```bash\ngo build ./...\n```\n\n### Rod\n\nRod is a pure Go solution that uses the Chrome DevTools Protocol directly. To use Rod instead of Playwright, compile with the `rod` build tag.\n\n```bash\n# Build with Rod support\ngo build -tags rod ./...\n\n# Run with Rod support\ngo run -tags rod ./...\n```\n\nRod will automatically download and manage Chrome/Chromium if it's not already available on your system.\n\n### Choosing Between Playwright and Rod\n\n| Feature | Playwright | Rod |\n|---------|-----------|-----|\n| **Dependencies** | Requires browser installation step | Auto-downloads browser |\n| **Browser Support** | Chromium, Firefox, WebKit | Chromium only |\n| **Performance** | Slightly higher overhead | Lower overhead, pure Go |\n| **Docker** | Larger image size | Smaller image size |\n| **API Stability** | Very stable | Stable |\n\n**Recommendation**: Use Playwright if you need multi-browser support or are already familiar with Playwright. Use Rod if you prefer a pure Go solution with automatic browser management and smaller Docker images.\n\n### Example Usage\n\nThe [books-to-scrape-simple](https://github.com/gosom/scrapemate/tree/main/examples/books-to-scrape-simple) example demonstrates how to use both browser engines:\n\n```bash\n# Run with Playwright (default)\n# First install browsers: go run github.com/playwright-community/playwright-go/cmd/playwright install --with-deps chromium\ngo run . -js\n\n# Run with Rod\ngo run -tags rod . -js\n\n# Run with Rod in stealth mode\ngo run -tags rod . -js -stealth\n```\n\n\n## Installation\n\n```\ngo get github.com/gosom/scrapemate\n```\n\n## Quickstart\n\n\n```go\npackage main\n\nimport (\n\t\"context\"\n\t\"encoding/csv\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"os\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"github.com/gosom/scrapemate\"\n\t\"github.com/gosom/scrapemate/adapters/writers/csvwriter\"\n\t\"github.com/gosom/scrapemate/scrapemateapp\"\n)\n\nfunc main() {\n\tcsvWriter := csvwriter.NewCsvWriter(csv.NewWriter(os.Stdout))\n\n\tcfg, err := scrapemateapp.NewConfig(\n\t\t[]scrapemate.ResultWriter{csvWriter},\n\t)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tapp, err := scrapemateapp.NewScrapeMateApp(cfg)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tseedJobs := []scrapemate.IJob{\n\t\t\u0026SimpleCountryJob{\n\t\t\tJob: scrapemate.Job{\n\t\t\t\tID:     \"identity\",\n\t\t\t\tMethod: http.MethodGet,\n\t\t\t\tURL:    \"https://www.scrapethissite.com/pages/simple/\",\n\t\t\t\tHeaders: map[string]string{\n\t\t\t\t\t\"User-Agent\": scrapemate.DefaultUserAgent,\n\t\t\t\t},\n\t\t\t\tTimeout:    10 * time.Second,\n\t\t\t\tMaxRetries: 3,\n\t\t\t},\n\t\t},\n\t}\n\terr = app.Start(context.Background(), seedJobs...)\n\tif err != nil \u0026\u0026 err != scrapemate.ErrorExitSignal {\n\t\tpanic(err)\n\t}\n}\n\ntype SimpleCountryJob struct {\n\tscrapemate.Job\n}\n\nfunc (j *SimpleCountryJob) Process(ctx context.Context, resp *scrapemate.Response) (any, []scrapemate.IJob, error) {\n\tdoc, ok := resp.Document.(*goquery.Document)\n\tif !ok {\n\t\treturn nil, nil, fmt.Errorf(\"failed to cast response document to goquery document\")\n\t}\n\tvar countries []Country\n\tdoc.Find(\"div.col-md-4.country\").Each(func(i int, s *goquery.Selection) {\n\t\tvar country Country\n\t\tcountry.Name = strings.TrimSpace(s.Find(\"h3.country-name\").Text())\n\t\tcountry.Capital = strings.TrimSpace(s.Find(\"div.country-info span.country-capital\").Text())\n\t\tcountry.Population = strings.TrimSpace(s.Find(\"div.country-info span.country-population\").Text())\n\t\tcountry.Area = strings.TrimSpace(s.Find(\"div.country-info span.country-area\").Text())\n\t\tcountries = append(countries, country)\n\t})\n\treturn countries, nil, nil\n}\n\ntype Country struct {\n\tName       string\n\tCapital    string\n\tPopulation string\n\tArea       string\n}\n\nfunc (c Country) CsvHeaders() []string {\n\treturn []string{\"Name\", \"Capital\", \"Population\", \"Area\"}\n}\n\nfunc (c Country) CsvRow() []string {\n\treturn []string{c.Name, c.Capital, c.Population, c.Area}\n}\n\n```\n\n```\ngo mod tidy\ngo run main.go 1\u003ecountries.csv\n```\n\n(hit CTRL-C to exit)\n\n## Migrating from v0.9.x to v1.0.0\n\nVersion 1.0.0 introduces a `BrowserPage` interface abstraction to support multiple browser engines. This is a breaking change for users who use JavaScript rendering with `BrowserActions`.\n\n### Update `BrowserActions` signature\n\n```go\n// Before (v0.9.x)\nfunc (j *MyJob) BrowserActions(ctx context.Context, page playwright.Page) scrapemate.Response {\n    page.Goto(\"https://example.com\", playwright.PageGotoOptions{\n        WaitUntil: playwright.WaitUntilStateNetworkidle,\n    })\n    html, _ := page.Content()\n    return scrapemate.Response{Body: []byte(html)}\n}\n\n// After (v1.0.0)\nfunc (j *MyJob) BrowserActions(ctx context.Context, page scrapemate.BrowserPage) scrapemate.Response {\n    resp, err := page.Goto(\"https://example.com\", scrapemate.WaitUntilNetworkIdle)\n    if err != nil {\n        return scrapemate.Response{Error: err}\n    }\n    return scrapemate.Response{\n        Body:       resp.Body,\n        StatusCode: resp.StatusCode,\n    }\n}\n```\n\n### Accessing the underlying browser page\n\nIf you need browser-specific features, use `Unwrap()`:\n\n```go\n// For Playwright\npwPage := page.Unwrap().(playwright.Page)\n\n// For Rod (when compiled with -tags rod)\nrodPage := page.Unwrap().(*rod.Page)\n```\n\nSee [CHANGELOG.md](CHANGELOG.md) for the full list of changes.\n\n## Documentation\n\nYou can find more documentation [here](https://gosom.github.io/scrapemate)\n\nFor the High Level API see this [example](https://github.com/gosom/scrapemate/tree/main/examples/quotes-to-scrape-app).\n\nRead also [how to use high level api](https://blog.gkomninos.com/golang-web-scraping-using-scrapemate)\n\nFor the Low Level API see [books.toscrape.com](https://github.com/gosom/scrapemate/tree/main/examples/books-to-scrape-simple)\n\nAdditionally, for low level API you can read [the blogpost](https://blog.gkomninos.com/getting-started-with-web-scraping-using-golang-and-scrapemate)\n\n\nSee an example of how you can use `scrapemate` go scrape Google Maps: https://github.com/gosom/google-maps-scraper\n\n## Contributing\n\nContributions are welcome.\n\n## Licence\n\nScrapemate is licensed under the MIT License. See LICENCE file\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgosom%2Fscrapemate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgosom%2Fscrapemate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgosom%2Fscrapemate/lists"}