https://github.com/scrape-badger/scrapebadger-node
Official Node.js SDK for ScrapeBadger - Async web scraping APIs for Twitter and more
https://github.com/scrape-badger/scrapebadger-node
scrapebadger-sdk scraping scraping-api sdk twitter twitter-api twitter-scraper x
Last synced: about 1 month ago
JSON representation
Official Node.js SDK for ScrapeBadger - Async web scraping APIs for Twitter and more
- Host: GitHub
- URL: https://github.com/scrape-badger/scrapebadger-node
- Owner: scrape-badger
- License: mit
- Created: 2025-12-27T17:46:52.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2026-03-30T15:22:55.000Z (3 months ago)
- Last Synced: 2026-03-30T15:31:47.095Z (3 months ago)
- Topics: scrapebadger-sdk, scraping, scraping-api, sdk, twitter, twitter-api, twitter-scraper, x
- Language: TypeScript
- Homepage: https://scrapebadger.com
- Size: 209 KB
- Stars: 2
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
ScrapeBadger Node.js SDK
The official Node.js/TypeScript client library for the [ScrapeBadger](https://scrapebadger.com) API — Twitter, Google, Vinted, and general web scraping.
## Features
- **Full TypeScript Support** - Complete type definitions for all API endpoints
- **Modern ESM & CommonJS** - Works with both module systems
- **Async Iterators** - Automatic pagination with `for await...of` syntax
- **Smart Rate Limiting** - Reads API headers and throttles pagination automatically
- **Resilient Retries** - Exponential backoff with colored console warnings
- **Typed Exceptions** - Distinct error classes for every failure scenario
- **37+ Twitter endpoints** - Tweets, users, lists, communities, trends, geo, real-time streams
- **19 Google product APIs** - Search (with optional deferred AI Overview follow-up), Maps, News, Hotels, Trends (incl. topic autocomplete), Jobs, Shopping (+ merchant URL enrichment), Patents, Scholar (search + profiles + author + author citation + cite formats), Images, Videos, Finance, AI Mode, Lens, **Local Pack**, **Shorts**, **Flights**, Products
- **Vinted scraping** - Search items, item details, user profiles, brands, colors, markets
- **Web scraping** - Anti-bot bypass, JS rendering, and AI data extraction
## Installation
```bash
npm install scrapebadger
```
```bash
yarn add scrapebadger
```
```bash
pnpm add scrapebadger
```
## Quick Start
```typescript
import { ScrapeBadger } from "scrapebadger";
const client = new ScrapeBadger({ apiKey: "your-api-key" });
// Get a tweet
const tweet = await client.twitter.tweets.getById("1234567890");
console.log(`@${tweet.username}: ${tweet.text}`);
// Scrape a website
const result = await client.web.scrape("https://scrapebadger.com", { format: "markdown" });
console.log(result.content);
// Get a user profile
const user = await client.twitter.users.getByUsername("elonmusk");
console.log(`${user.name} has ${user.followers_count.toLocaleString()} followers`);
```
## Authentication
```typescript
// Pass API key directly
const client = new ScrapeBadger({ apiKey: "sb_live_xxxxxxxxxxxxx" });
// Or use environment variable SCRAPEBADGER_API_KEY
const client = new ScrapeBadger();
```
## Available APIs
| API | Description | Documentation |
|-----|-------------|---------------|
| **Web Scraping** | Scrape any website with JS rendering, anti-bot bypass, and AI extraction | [Web Scraping Guide](docs/web-scraping.md) |
| **Twitter** | 37+ endpoints for tweets, users, lists, communities, trends, and real-time streams | [Twitter Guide](docs/twitter.md) |
| **Google** | 19 products — Search, Maps, News, Hotels, Trends, Jobs, Shopping, Patents, Scholar, Images, Videos, Finance, AI Mode, Lens, Autocomplete, Local, Shorts, Flights, Products | [Google Guide](docs/google.md) |
| **Vinted** | Search items, get details, user profiles, and reference data across all Vinted markets | [Vinted Guide](docs/vinted.md) |
| **Reddit** | Search posts/subreddits/users, subreddit details, post comments, user profiles, wiki pages, trophies | [Reddit Guide](docs/reddit.md) |
## Error Handling
```typescript
import {
ScrapeBadger,
AuthenticationError,
RateLimitError,
NotFoundError,
InsufficientCreditsError,
} from "scrapebadger";
const client = new ScrapeBadger({ apiKey: "your-api-key" });
try {
const tweet = await client.twitter.tweets.getById("1234567890");
} catch (error) {
if (error instanceof AuthenticationError) {
console.error("Invalid API key");
} else if (error instanceof RateLimitError) {
console.error(`Rate limited. Retry after: ${error.retryAfter}`);
} else if (error instanceof NotFoundError) {
console.error("Tweet not found");
} else if (error instanceof InsufficientCreditsError) {
console.error("Out of credits");
} else {
throw error;
}
}
```
## Configuration
```typescript
const client = new ScrapeBadger({
// Required: Your API key (or use SCRAPEBADGER_API_KEY env var)
apiKey: "your-api-key",
// Optional: Custom base URL (default: https://scrapebadger.com)
baseUrl: "https://scrapebadger.com",
// Optional: Request timeout in milliseconds (default: 30000)
timeout: 30000,
// Optional: Maximum retry attempts (default: 10)
maxRetries: 10,
// Optional: Initial retry delay in milliseconds (default: 1000)
retryDelay: 1000,
});
```
### Retry Behavior
The SDK automatically retries requests that fail with server errors (5xx) or rate
limits (429) using exponential backoff (1s, 2s, 4s, 8s, ...). Each retry prints a
colored warning:
```
⚠ ScrapeBadger: 503 Service Unavailable — retrying in 4s (attempt 3/10)
```
### Rate Limit Aware Pagination
When using `*All` pagination methods (e.g. `searchAll`, `getFollowersAll`), the SDK
reads `X-RateLimit-Remaining` and `X-RateLimit-Reset` headers from each response.
When remaining requests drop below 20% of your tier's limit, pagination automatically
slows down to spread requests across the remaining window — preventing 429 errors:
```
⚠ ScrapeBadger: Rate limit: 25/300 remaining (resets in 42s), throttling pagination
```
This works transparently with all tier levels (Free: 60/min, Basic: 300/min,
Pro: 1000/min, Enterprise: 5000/min).
## Exceptions
- `ScrapeBadgerError` - Base exception class
- `AuthenticationError` - Invalid or missing API key
- `RateLimitError` - Rate limit exceeded
- `NotFoundError` - Resource not found
- `ValidationError` - Invalid request
- `ServerError` - Server error
- `TimeoutError` - Request timeout
- `InsufficientCreditsError` - Out of credits
- `AccountRestrictedError` - Account restricted
- `WebSocketStreamError` - WebSocket stream failure (auth, limit, or network)
## Requirements
- Node.js 18+ (for native `fetch` support)
- TypeScript 5.0+ (for best type inference)
## License
MIT License - see [LICENSE](LICENSE) for details.
## Links
- [Documentation](https://docs.scrapebadger.com)
- [Discord](https://discord.com/invite/3WvwTyWVCx)
- [GitHub](https://github.com/scrape-badger/scrapebadger-node)
- [npm](https://www.npmjs.com/package/scrapebadger)