{"id":37747871,"url":"https://github.com/promptapi/scraper-go","last_synced_at":"2026-01-16T14:18:49.164Z","repository":{"id":57542810,"uuid":"293615911","full_name":"promptapi/scraper-go","owner":"promptapi","description":"Golang wrapper for Prompt API's Scraper API","archived":false,"fork":false,"pushed_at":"2020-10-06T07:33:22.000Z","size":45,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2023-07-27T22:29:33.813Z","etag":null,"topics":["api-marketplace","api-wrapper","css-selector","css-selector-parser","data-extraction","golang","golang-package","image-scraper","image-scraping","promptapi","scraper","scraper-api","web-scraper","web-scraping"],"latest_commit_sha":null,"homepage":"https://promptapi.com/marketplace/description/scraper-api","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/promptapi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-09-07T19:33:57.000Z","updated_at":"2021-04-28T12:19:21.000Z","dependencies_parsed_at":"2022-09-26T18:31:24.095Z","dependency_job_id":null,"html_url":"https://github.com/promptapi/scraper-go","commit_stats":null,"previous_names":[],"tags_count":5,"template":null,"template_full_name":null,"purl":"pkg:github/promptapi/scraper-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptapi%2Fscraper-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptapi%2Fscraper-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptapi%2Fscraper-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptapi%2Fscraper-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/promptapi","download_url":"https://codeload.github.com/promptapi/scraper-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/promptapi%2Fscraper-go/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28479369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T11:59:17.896Z","status":"ssl_error","status_checked_at":"2026-01-16T11:55:55.838Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-marketplace","api-wrapper","css-selector","css-selector-parser","data-extraction","golang","golang-package","image-scraper","image-scraping","promptapi","scraper","scraper-api","web-scraper","web-scraping"],"created_at":"2026-01-16T14:18:49.107Z","updated_at":"2026-01-16T14:18:49.156Z","avatar_url":"https://github.com/promptapi.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Version](https://img.shields.io/badge/version-0.1.4-orange.svg)\n![Go](https://img.shields.io/badge/go-1.15.1-black.svg)\n[![Documentation](https://godoc.org/github.com/promptapi/scraper-go?status.svg)](https://pkg.go.dev/github.com/promptapi/scraper-go)\n[![Go Report Card](https://goreportcard.com/badge/github.com/promptapi/scraper-go)](https://goreportcard.com/report/github.com/promptapi/scraper-go)\n[![Build Status](https://travis-ci.org/promptapi/scraper-go.svg?branch=main)](https://travis-ci.org/promptapi/scraper-go)\n\n# Prompt API - Scraper - Golang Package\n\n`PromptAPI` struct is a simple golang wrapper for [scraper api][scraper-api]\nwith few more extra cream and sugar.\n\n---\n\n## Requirements\n\n1. You need to signup for [Prompt API][promptapi-signup]\n1. You need to subscribe [scraper api][scraper-api], test drive is **free!!!**\n1. You need to set `PROMPTAPI_TOKEN` environment variable after subscription.\n\nthen;\n\n```bash\n$ go get -u github.com/promptapi/scraper-go\n```\n\n---\n\n## Example Basic Usage\n\n```go\n// main.go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\tscraper \"github.com/promptapi/scraper-go\"\n)\n\nfunc main() {\n\ts := new(scraper.PromptAPI)\n\n\tparams := \u0026scraper.Params{\n\t\tURL:      \"https://pypi.org/classifiers/\",\n\t\tCountry:  \"EE\",\n\t}\n\textraHeaders := []*ExtraHeader{} // custom extra headers\n\tresult := new(scraper.Result)\n\n\terr := s.Scrape(params, extraHeaders, result)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Printf(\"Length of incoming data: %d\\n\", len(result.Data))\n\tfmt.Printf(\"Response headers: %v\\n\", result.Headers)\n\tfmt.Printf(\"Content-Length: %v\\n\", result.Headers[\"Content-Length\"])\n\n\tfileSize, err := s.Save(\"/tmp/test.html\", result)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Printf(\"Size of /tmp/test.html -\u003e %d bytes\\n\", fileSize)\n\n}\n```\n\nRun:\n\n```bash\n$ go run main.go \nLength of incoming data: 321322\nResponse headers: map[Accept-Ranges:bytes Content-Length:321322 Content-Security-Policy:base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://api.pwnedpasswords.com https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com Content-Type:text/html; charset=UTF-8 Date:Tue, 08 Sep 2020 19:10:24 GMT ETag:\"1ea9p+Hscl37dEKelacPWw\" Referrer-Policy:origin-when-cross-origin Strict-Transport-Security:max-age=31536000; includeSubDomains; preload Vary:Accept-Encoding, Cookie, Accept-Encoding X-Cache:MISS, HIT X-Cache-Hits:0, 1 X-Content-Type-Options:nosniff X-Frame-Options:deny X-Permitted-Cross-Domain-Policies:none X-Served-By:cache-bwi5127-BWI, cache-hhn4035-HHN X-Timer:S1599592224.395422,VS0,VE247 X-XSS-Protection:1; mode=block]\nContent-Length: 321322\nSize of /tmp/test.html -\u003e 321322 bytes\n```\n\nYou can add url parameters for extra operations. Valid parameters are:\n\n- `AuthPassword`: for HTTP Realm auth password\n- `AuthUsername`: for HTTP Realm auth username\n- `Cookie`: URL Encoded cookie header.\n- `Country`: 2 character country code. If you wish to scrape from an IP address of a specific country.\n- `Referer`: HTTP referer header\n- `Selector`: CSS style selector path such as `a.btn div li`. If `Selector` is\n  enabled, returning result will be collection of data and saved file will be\n  in `.json` format.\n\nExample with `Selector`:\n\n```go\n// main.go\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\n\tscraper \"github.com/promptapi/scraper-go\"\n)\n\nfunc main() {\n\ts := new(scraper.PromptAPI)\n\n\tparams := \u0026scraper.Params{\n\t\tURL:      \"https://pypi.org/classifiers/\",\n\t\tCountry:  \"EE\",\n\t\tSelector: \"ul li button[data-clipboard-text]\",\n\t}\n\t// add extra request headers\n\textraHeaders := []*ExtraHeader{\n\t\t\u0026ExtraHeader{\n\t\t\tname:  \"X-Referer\",\n\t\t\tvalue: \"https://www.google.com\",\n\t\t},\n\t}\n\tresult := new(scraper.Result)\n\n\terr := s.Scrape(params, extraHeaders, result)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Printf(\"Length of incoming data: %d\\n\", len(result.Data))\n\tfmt.Printf(\"Length of extracted data: %d\\n\", len(result.DataSelector))\n\tfmt.Printf(\"Response headers: %v\\n\", result.Headers)\n\tfmt.Printf(\"Content-Length: %v\\n\", result.Headers[\"Content-Length\"])\n\n\tfileSize, err := s.Save(\"/tmp/test.json\", result)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\tfmt.Printf(\"Size of /tmp/test.json -\u003e %d bytes\\n\", fileSize)\n\n}\n```\n\nRun:\n\n```bash\n$ go run main.go \nLength of incoming data: 0\nLength of extracted data: 734\nResponse headers: map[Accept-Ranges:bytes Content-Length:321322 Content-Security-Policy:base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://api.pwnedpasswords.com https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com Content-Type:text/html; charset=UTF-8 Date:Tue, 08 Sep 2020 19:17:22 GMT ETag:\"1ea9p+Hscl37dEKelacPWw\" Referrer-Policy:origin-when-cross-origin Strict-Transport-Security:max-age=31536000; includeSubDomains; preload Vary:Accept-Encoding, Cookie, Accept-Encoding X-Cache:HIT, HIT X-Cache-Hits:1, 1 X-Content-Type-Options:nosniff X-Frame-Options:deny X-Permitted-Cross-Domain-Policies:none X-Served-By:cache-bwi5137-BWI, cache-bma1621-BMA X-Timer:S1599592641.178639,VS0,VE1512 X-XSS-Protection:1; mode=block]\nContent-Length: 321322\nSize of /tmp/test.json -\u003e 173717 bytes\n```\n\nLet’s see `/tmp/test.json` file:\n\n```json\n[\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 1 - Planning\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 2 - Pre-Alpha\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 3 - Alpha\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 4 - Beta\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 5 - Production/Stable\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 6 - Mature\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  \"\u003cbutton class=\\\"button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w\\\" data-clipboard-text=\\\"Development Status :: 7 - Inactive\\\" data-tooltip-label=\\\"Copy to clipboard\\\" type=\\\"button\\\"\u003e\\n Copy\\n\u003c/button\u003e\\n\",\n  ,\n  ,\n  ,\n  ,\n  ,\n]\n```\n\n---\n\n## Development\n\nAvailable rake tasks:\n\n```bash\n$ rake -T\nrake default                    # Default task, show avaliable tasks\nrake release:check              # Do release check\nrake release:publish[revision]  # Publish project with revision: major,minor,patch, default: patch\nrake serve_doc[port]            # Run doc server\nrake test[verbose]              # Run tests\n```\n\n- Run tests: `rake test` or `rake test[-v]`\n- Run doc server: `rake serve_doc` or `rake serve_doc[9000]`\n\nRelease package (*if you have write access*):\n\n1. Commit your changes\n1. Run `rake release:check`\n1. If all goes ok, run `rake release:publish`\n\n---\n\n## License\n\nThis project is licensed under MIT\n\n---\n\n## Contributer(s)\n\n* [Prompt API](https://github.com/promptapi) - Creator, maintainer\n\n---\n\n## Contribute\n\nAll PR’s are welcome!\n\n1. `fork` (https://github.com/promptapi/scraper-go/fork)\n1. Create your `branch` (`git checkout -b my-feature`)\n1. `commit` yours (`git commit -am 'Add awesome features...'`)\n1. `push` your `branch` (`git push origin my-feature`)\n1. Than create a new **Pull Request**!\n\nThis project is intended to be a safe,\nwelcoming space for collaboration, and contributors are expected to adhere to\nthe [code of conduct][coc].\n\n---\n\n[scraper-api]:      https://promptapi.com/marketplace/description/scraper-api\n[promptapi-signup]: https://promptapi.com/#signup-form\n[coc]:              https://github.com/promptapi/scraper-go/blob/main/CODE_OF_CONDUCT.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptapi%2Fscraper-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpromptapi%2Fscraper-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpromptapi%2Fscraper-go/lists"}