{"id":35179903,"url":"https://github.com/scrape-badger/scrapebadger-node","last_synced_at":"2026-05-31T01:02:32.432Z","repository":{"id":330900794,"uuid":"1123824096","full_name":"scrape-badger/scrapebadger-node","owner":"scrape-badger","description":"Official Node.js SDK for ScrapeBadger - Async web scraping APIs for Twitter and more","archived":false,"fork":false,"pushed_at":"2026-03-30T15:22:55.000Z","size":214,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-30T15:31:47.095Z","etag":null,"topics":["scrapebadger-sdk","scraping","scraping-api","sdk","twitter","twitter-api","twitter-scraper","x"],"latest_commit_sha":null,"homepage":"https://scrapebadger.com","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scrape-badger.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-27T17:46:52.000Z","updated_at":"2026-03-30T15:22:59.000Z","dependencies_parsed_at":"2025-12-31T23:00:21.286Z","dependency_job_id":null,"html_url":"https://github.com/scrape-badger/scrapebadger-node","commit_stats":null,"previous_names":["scrape-badger/scrapebadger-node"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/scrape-badger/scrapebadger-node","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrape-badger%2Fscrapebadger-node","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrape-badger%2Fscrapebadger-node/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrape-badger%2Fscrapebadger-node/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrape-badger%2Fscrapebadger-node/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scrape-badger","download_url":"https://codeload.github.com/scrape-badger/scrapebadger-node/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scrape-badger%2Fscrapebadger-node/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31554110,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T10:21:54.569Z","status":"ssl_error","status_checked_at":"2026-04-08T10:21:38.171Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scrapebadger-sdk","scraping","scraping-api","sdk","twitter","twitter-api","twitter-scraper","x"],"created_at":"2025-12-29T00:57:57.202Z","updated_at":"2026-05-31T01:02:32.406Z","avatar_url":"https://github.com/scrape-badger.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://scrapebadger.com/logo-dark.png\" alt=\"ScrapeBadger\" width=\"400\"\u003e\n\u003c/p\u003e\n\n\u003ch1 align=\"center\"\u003eScrapeBadger Node.js SDK\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.npmjs.com/package/scrapebadger\"\u003e\u003cimg src=\"https://img.shields.io/npm/v/scrapebadger.svg\" alt=\"npm version\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.npmjs.com/package/scrapebadger\"\u003e\u003cimg src=\"https://img.shields.io/npm/dm/scrapebadger.svg\" alt=\"npm downloads\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/scrape-badger/scrapebadger-node/actions/workflows/test.yml\"\u003e\u003cimg src=\"https://github.com/scrape-badger/scrapebadger-node/actions/workflows/test.yml/badge.svg\" alt=\"Tests\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://www.typescriptlang.org/\"\u003e\u003cimg src=\"https://img.shields.io/badge/TypeScript-5.0+-blue.svg\" alt=\"TypeScript\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-yellow.svg\" alt=\"License: MIT\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nThe official Node.js/TypeScript client library for the [ScrapeBadger](https://scrapebadger.com) API — Twitter, Google, Vinted, and general web scraping.\n\n## Features\n\n- **Full TypeScript Support** - Complete type definitions for all API endpoints\n- **Modern ESM \u0026 CommonJS** - Works with both module systems\n- **Async Iterators** - Automatic pagination with `for await...of` syntax\n- **Smart Rate Limiting** - Reads API headers and throttles pagination automatically\n- **Resilient Retries** - Exponential backoff with colored console warnings\n- **Typed Exceptions** - Distinct error classes for every failure scenario\n- **37+ Twitter endpoints** - Tweets, users, lists, communities, trends, geo, real-time streams\n- **19 Google product APIs** - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, **Local Pack**, **Shorts**, **Flights**, Products\n- **Vinted scraping** - Search items, item details, user profiles, brands, colors, markets\n- **Web scraping** - Anti-bot bypass, JS rendering, and AI data extraction\n\n## Installation\n\n```bash\nnpm install scrapebadger\n```\n\n```bash\nyarn add scrapebadger\n```\n\n```bash\npnpm add scrapebadger\n```\n\n## Quick Start\n\n```typescript\nimport { ScrapeBadger } from \"scrapebadger\";\n\nconst client = new ScrapeBadger({ apiKey: \"your-api-key\" });\n\n// Get a tweet\nconst tweet = await client.twitter.tweets.getById(\"1234567890\");\nconsole.log(`@${tweet.username}: ${tweet.text}`);\n\n// Scrape a website\nconst result = await client.web.scrape(\"https://scrapebadger.com\", { format: \"markdown\" });\nconsole.log(result.content);\n\n// Get a user profile\nconst user = await client.twitter.users.getByUsername(\"elonmusk\");\nconsole.log(`${user.name} has ${user.followers_count.toLocaleString()} followers`);\n```\n\n## Authentication\n\n```typescript\n// Pass API key directly\nconst client = new ScrapeBadger({ apiKey: \"sb_live_xxxxxxxxxxxxx\" });\n\n// Or use environment variable SCRAPEBADGER_API_KEY\nconst client = new ScrapeBadger();\n```\n\n## Available APIs\n\n| API | Description | Documentation |\n|-----|-------------|---------------|\n| **Web Scraping** | Scrape any website with JS rendering, anti-bot bypass, and AI extraction | [Web Scraping Guide](docs/web-scraping.md) |\n| **Twitter** | 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams | [Twitter Guide](docs/twitter.md) |\n| **Google** | 19 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products | [Google Guide](docs/google.md) |\n| **Vinted** | Search items, get details, user profiles, and reference data across all Vinted markets | [Vinted Guide](docs/vinted.md) |\n| **Reddit** | Search posts/subreddits/users, subreddit details, post comments, user profiles, wiki pages, trophies | [Reddit Guide](docs/reddit.md) |\n\n## Error Handling\n\n```typescript\nimport {\n  ScrapeBadger,\n  AuthenticationError,\n  RateLimitError,\n  NotFoundError,\n  InsufficientCreditsError,\n} from \"scrapebadger\";\n\nconst client = new ScrapeBadger({ apiKey: \"your-api-key\" });\n\ntry {\n  const tweet = await client.twitter.tweets.getById(\"1234567890\");\n} catch (error) {\n  if (error instanceof AuthenticationError) {\n    console.error(\"Invalid API key\");\n  } else if (error instanceof RateLimitError) {\n    console.error(`Rate limited. Retry after: ${error.retryAfter}`);\n  } else if (error instanceof NotFoundError) {\n    console.error(\"Tweet not found\");\n  } else if (error instanceof InsufficientCreditsError) {\n    console.error(\"Out of credits\");\n  } else {\n    throw error;\n  }\n}\n```\n\n## Configuration\n\n```typescript\nconst client = new ScrapeBadger({\n  // Required: Your API key (or use SCRAPEBADGER_API_KEY env var)\n  apiKey: \"your-api-key\",\n\n  // Optional: Custom base URL (default: https://scrapebadger.com)\n  baseUrl: \"https://scrapebadger.com\",\n\n  // Optional: Request timeout in milliseconds (default: 30000)\n  timeout: 30000,\n\n  // Optional: Maximum retry attempts (default: 10)\n  maxRetries: 10,\n\n  // Optional: Initial retry delay in milliseconds (default: 1000)\n  retryDelay: 1000,\n});\n```\n\n### Retry Behavior\n\nThe SDK automatically retries requests that fail with server errors (5xx) or rate\nlimits (429) using exponential backoff (1s, 2s, 4s, 8s, ...). Each retry prints a\ncolored warning:\n\n```\n⚠ ScrapeBadger: 503 Service Unavailable — retrying in 4s (attempt 3/10)\n```\n\n### Rate Limit Aware Pagination\n\nWhen using `*All` pagination methods (e.g. `searchAll`, `getFollowersAll`), the SDK\nreads `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers from each response.\nWhen remaining requests drop below 20% of your tier's limit, pagination automatically\nslows down to spread requests across the remaining window — preventing 429 errors:\n\n```\n⚠ ScrapeBadger: Rate limit: 25/300 remaining (resets in 42s), throttling pagination\n```\n\nThis works transparently with all tier levels (Free: 60/min, Basic: 300/min,\nPro: 1000/min, Enterprise: 5000/min).\n\n## Exceptions\n\n- `ScrapeBadgerError` - Base exception class\n- `AuthenticationError` - Invalid or missing API key\n- `RateLimitError` - Rate limit exceeded\n- `NotFoundError` - Resource not found\n- `ValidationError` - Invalid request\n- `ServerError` - Server error\n- `TimeoutError` - Request timeout\n- `InsufficientCreditsError` - Out of credits\n- `AccountRestrictedError` - Account restricted\n- `WebSocketStreamError` - WebSocket stream failure (auth, limit, or network)\n\n## Requirements\n\n- Node.js 18+ (for native `fetch` support)\n- TypeScript 5.0+ (for best type inference)\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Links\n\n- [Documentation](https://docs.scrapebadger.com)\n- [Discord](https://discord.com/invite/3WvwTyWVCx)\n- [GitHub](https://github.com/scrape-badger/scrapebadger-node)\n- [npm](https://www.npmjs.com/package/scrapebadger)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrape-badger%2Fscrapebadger-node","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscrape-badger%2Fscrapebadger-node","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscrape-badger%2Fscrapebadger-node/lists"}