https://github.com/miiguellssantos/x-api-research
A Node.js tool to bulk-collect tweets via Twitter API v2, handle rate limits and safe interruption, and export results incrementally to CSV.
https://github.com/miiguellssantos/x-api-research
javascript x-api
Last synced: 12 days ago
JSON representation
A Node.js tool to bulk-collect tweets via Twitter API v2, handle rate limits and safe interruption, and export results incrementally to CSV.
- Host: GitHub
- URL: https://github.com/miiguellssantos/x-api-research
- Owner: miiguellssantos
- Created: 2025-05-06T19:49:31.000Z (13 days ago)
- Default Branch: main
- Last Pushed: 2025-05-06T20:03:16.000Z (13 days ago)
- Last Synced: 2025-05-06T21:20:38.494Z (13 days ago)
- Topics: javascript, x-api
- Language: JavaScript
- Homepage:
- Size: 616 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Twitter Data Collector
A Node.js script that uses the Twitter API v2 to search and collect tweets in bulk, save progress incrementally to CSV, and handle rate limits and safe interruption (Ctrl+C).
## Features
- **Authentication** with Twitter v2 Bearer Token
- **Incremental CSV export** with headers
- **Safe interruption** (SIGINT) and progress saving
- **Rate-limit handling** with automatic wait until reset
- **Configurable query**, date range, and output folder
## Prerequisites
- Node.js v14+
- A Twitter Developer account with Elevated or Academic Research access
- A valid Bearer Token
## Installation
1. Clone this repository
```bash
git clone https://github.com/miiguellssantos/X-API-Research.git
cd X-API-Research
```
2. Install dependencies
```bash
npm install twitter-api-v2 luxon node-notifier
```
3. Add your Bearer Token
- Open `twitterAPI.js`
- Replace the placeholder in the `BEARER_TOKEN` constant with your actual token
## Configuration
- **Queries**
Edit the `queries` array in `twitterAPI.js` to define one or more search queries.
- **Date range**
Adjust `startDate` and `endDate` (ISO 8601 format) in `main()` to control the period.
- **Output path**
Change the `path.resolve(...)` in `main()` to point to your desired CSV directory.
## Usage
```bash
node twitterAPI.js
```
- The script logs authentication status, each batch progress, and any rate-limit waits.
- Press **Ctrl+C** to safely stop the run; progress will be saved up to the last batch.
## CSV Output Format
| Geo | Created At | Likes | Retweets | Tweet ID | Username | Text | User Location | Followers Count |
|--------|-----------------------|-------|----------|-------------------|----------|----------------|---------------|-----------------|
| JSON | `YYYY-MM-DDTHH:MM:SSZ`| int | int | string | string | string | string | int |
## Extending
- Add more tweet or user fields by modifying `params` in `getTweetDataWithStopSupport()`.
- Re-enable desktop notifications (using `node-notifier`) by uncommenting the `notifier.notify(...)` block.