https://github.com/jongan69/fb-marketplace-api
demo api
https://github.com/jongan69/fb-marketplace-api
Last synced: 5 months ago
JSON representation
demo api
- Host: GitHub
- URL: https://github.com/jongan69/fb-marketplace-api
- Owner: jongan69
- Created: 2025-11-13T04:29:18.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-11-13T05:59:23.000Z (8 months ago)
- Last Synced: 2025-11-13T06:16:51.825Z (8 months ago)
- Language: Python
- Size: 98.6 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Facebook Marketplace API
An easy-to-use Facebook Marketplace API that functions without the need to log into a Facebook account. Wraps the Facebook GraphQL API, allowing for quick and easy retrieval of Facebook Marketplace listings and other relevant Marketplace data.
## Features
- 🔍 Search Facebook Marketplace listings by location and keywords
- 📍 Get location coordinates for any city/location
- 🚀 Production-ready with Railway deployment support
- 🛡️ Built-in rate limiting and retry logic
- 📊 Comprehensive logging for debugging
- 🔄 Automatic Brotli and Gzip decompression support
## Quick Start
### Installation
```bash
pip install -r requirements.txt
```
### Running Locally
1. **Optional**: Configure proxy (recommended to avoid IP blocking)
```bash
cp .env.example .env
# Edit .env with your proxy credentials
```
2. **Start the API:**
```bash
python3 app.py
```
The API will start on `http://localhost:5001` (or next available port).
### Production Deployment (Railway)
The API is configured for Railway deployment with:
- `Procfile` - Gunicorn production server
- `railway.json` - Railway-specific configuration
- Automatic port detection from `$PORT` environment variable
**Production URL**: `https://fb-marketplace-api-production.up.railway.app`
### Proxy Configuration (Optional)
To use a proxy service (e.g., Decodo) to avoid IP-based blocking, set these environment variables:
**Local Development (.env file):**
```bash
PROXY_USERNAME=your_username
PROXY_PASSWORD=your_password
PROXY_HOST=gate.decodo.com
PROXY_PORT=7000
```
**Railway (Environment Variables):**
Add these in your Railway project settings → Variables:
- `PROXY_USERNAME` = your proxy username
- `PROXY_PASSWORD` = your proxy password
- `PROXY_HOST` = proxy hostname (e.g., `gate.decodo.com`)
- `PROXY_PORT` = proxy port (e.g., `7000`)
**Testing Proxy:**
```bash
python3 tests/test_proxy.py
```
The proxy will be used automatically for all Facebook API requests when configured. This helps avoid IP-based blocking that occurs with server IPs.
**Supported Proxy Services:**
- Decodo
- Any HTTP/HTTPS proxy service
- Residential proxies
- Datacenter proxies
## API Endpoints
### GET /docs or GET /api-docs
Get comprehensive API documentation in JSON format.
**Example:**
```bash
curl "http://localhost:5001/docs"
```
**Response:**
Returns complete API documentation including:
- All available endpoints
- Request parameters
- Response formats
- Example requests
- Features and configuration
### GET /locations
Get latitude and longitude coordinates for a location.
**Parameters:**
- `locationQuery` (required): Location name (e.g., "Houston", "New York")
**Example:**
```bash
curl "http://localhost:5001/locations?locationQuery=Houston"
```
**Response:**
```json
{
"status": "Success",
"error": {},
"data": {
"locations": [
{
"name": "Houston, Texas",
"latitude": "29.7602",
"longitude": "-95.3694"
}
]
}
}
```
### GET /search
Search for marketplace listings in a specific location.
**Parameters:**
- `locationLatitude` (required): Latitude coordinate
- `locationLongitude` (required): Longitude coordinate
- `listingQuery` (required): Product/search keywords
**Example:**
```bash
curl "http://localhost:5001/search?locationLatitude=29.7602&locationLongitude=-95.3694&listingQuery=couch"
```
**Response:**
```json
{
"status": "Success",
"error": {},
"data": {
"listingPages": [
{
"listings": [
{
"id": "123456789",
"name": "Sectional Couch",
"currentPrice": "$150",
"previousPrice": "",
"saleIsPending": "false",
"primaryPhotoURL": "https://...",
"sellerName": "John Doe",
"sellerLocation": "Houston, Texas",
"sellerType": "User"
}
]
}
]
}
}
```
## Response Format
All endpoints return JSON in this format:
```json
{
"status": "Success" | "Failure",
"error": {
"source": "User" | "Facebook" | "Request",
"message": "Error description"
},
"data": {
// Response data (varies by endpoint)
}
}
```
## Testing
### Quick Test Script
```bash
python3 tests/test_search_simple.py Houston couch
```
### Comprehensive Test Suite
```bash
python3 tests/test_search.py
```
### Production Testing
```bash
python3 tests/test_production.py
```
### Debug Tool
```bash
python3 tests/debug_facebook_response.py
```
## Rate Limiting
The API includes built-in rate limiting protection to work around Facebook's rate limits.
### Critical Finding: IP-Based Blocking
**Important**: If you see rate limiting on the **FIRST request**, this indicates **IP-based blocking**, not request pattern issues. Railway's IP address may be flagged/blacklisted by Facebook.
**Evidence**:
- Rate limit on first request (`Request count: 1`)
- Error code `1675004` (Facebook rate limit error)
- No rate limit headers provided by Facebook
### Why This Happens
1. **Railway IP Reputation**: Railway uses shared IP addresses that may have been used by other scrapers
2. **Server IP Detection**: Facebook detects server IPs vs browser IPs and blocks them aggressively
3. **No Authentication**: Without cookies/auth, Facebook treats requests as suspicious
4. **GraphQL Endpoint**: The `/api/graphql/` endpoint is heavily protected
### Solutions
#### Option 1: Use Residential Proxy (For Scraping)
- Use a residential proxy service
- Rotate IP addresses
- Appear as real browser traffic
- **Note**: May violate Facebook ToS
#### Option 2: Wait for IP Reputation Reset
- Wait 24-48 hours for IP reputation to reset
- May not work if IP is permanently flagged
#### Option 3: Use Different Hosting Provider
- Try deploying to a different provider (AWS, GCP, etc.)
- May have better IP reputation
- Still may get blocked eventually
#### Option 4: Use Facebook Official API (Recommended for Production)
- Use Facebook's official Graph API with proper authentication
- Requires app registration and OAuth
- Has official rate limits but won't be blocked immediately
#### Option 5: Test from Local Machine
- Residential IPs are less likely to be blocked
- Good for testing before deployment
### Implemented Strategies
1. **Request Throttling**: Random delays (10-20 seconds) between requests
2. **Exponential Backoff Retry**: Automatic retry with increasing wait times (120s, 240s, 360s)
3. **User-Agent Rotation**: Randomly selects from multiple modern browser user agents
4. **Enhanced HTTP Headers**: Realistic browser headers (Referer, Origin, Accept, etc.)
5. **Session Management**: Uses `requests.Session()` for connection pooling
6. **Rate Limit Detection**: Detects rate limit errors and automatically retries
7. **Comprehensive Debugging**: Detailed logs to identify rate limit causes
### Configuration
You can adjust these settings at the top of `MarketplaceScraper.py`:
```python
MIN_DELAY_BETWEEN_REQUESTS = 10.0 # Minimum seconds between requests
MAX_DELAY_BETWEEN_REQUESTS = 20.0 # Maximum seconds between requests
RATE_LIMIT_RETRY_DELAY = 120 # Seconds to wait after rate limit error
MAX_RETRIES = 3 # Maximum retries for rate-limited requests
```
### How It Works
1. **Before each request**: Checks time since last request and waits if needed
2. **During request**: Uses random user agent and realistic browser headers
3. **After rate limit detected**: Automatically waits and retries up to 3 times
4. **Debug logging**: Logs detailed information when rate limits occur
### Rate Limit Indicators
Watch for these indicators in logs:
- **Rate limit on first request** = IP blocked (Railway IP flagged)
- **Rate limit after multiple requests** = Pattern detected (request frequency too high)
- **Rate limit headers present** = Temporary limit (can retry with backoff)
- **No rate limit headers** = Permanent block (IP flagged)
### Limitations
⚠️ **Important Notes**:
- Facebook's Terms of Service: Scraping Facebook may violate their ToS. Use responsibly.
- Rate Limits Still Apply: These strategies reduce rate limiting but don't eliminate it entirely.
- IP-based Limits: If your server IP gets flagged, delays won't help - need different IP or wait for reset.
- Railway IP Reputation: Railway's shared IPs may already be flagged by Facebook.
## Compression Support
The API automatically handles:
- **Brotli** compression (used by Facebook on Railway)
- **Gzip** compression
- Automatic decompression and fallback handling
The `requests` library automatically decompresses Brotli when the `brotli` library is available. The code detects already-decompressed content and skips unnecessary decompression attempts.
## Postman Examples
### Base URLs
- **Local**: `http://localhost:5001`
- **Production**: `https://fb-marketplace-api-production.up.railway.app`
### Example Requests
#### Get Locations
```
GET http://localhost:5001/locations?locationQuery=Houston
GET http://localhost:5001/locations?locationQuery=New York
GET http://localhost:5001/locations?locationQuery=Los Angeles
```
#### Search Listings
```
GET http://localhost:5001/search?locationLatitude=29.7602&locationLongitude=-95.3694&listingQuery=couch
GET http://localhost:5001/search?locationLatitude=25.8138&locationLongitude=-80.1335&listingQuery=cars
GET http://localhost:5001/search?locationLatitude=40.7128&locationLongitude=-74.0060&listingQuery=bicycle
```
### Postman Collection Setup
1. **Create New Collection**: "Facebook Marketplace API"
2. **Add Environment Variables** (optional):
- `base_url`: `http://localhost:5001` (or production URL)
- `houston_lat`: `29.7602`
- `houston_lon`: `-95.3694`
3. **Use Variables in URLs**:
```
{{base_url}}/locations?locationQuery=Houston
{{base_url}}/search?locationLatitude={{houston_lat}}&locationLongitude={{houston_lon}}&listingQuery=couch
```
### Complete Search Workflow
1. **Get Location Coordinates**
```
GET {{base_url}}/locations?locationQuery=Houston
```
- Copy `latitude` and `longitude` from response
2. **Search for Products**
```
GET {{base_url}}/search?locationLatitude=29.7602&locationLongitude=-95.3694&listingQuery=couch
```
- Use coordinates from step 1
### Common Test Scenarios
**Popular Cities**: Houston, New York, Los Angeles, Chicago, Miami, San Francisco
**Popular Products**: couch, laptop, car, bicycle, iphone, furniture, tv, bike
**Error Cases**: Missing parameters will return user errors
### Tips
1. **Rate Limiting**: If you get rate limit errors, wait 60+ seconds between requests
2. **Coordinates**: Always get fresh coordinates using `/locations` endpoint
3. **Product Names**: Use simple, common terms (e.g., "couch" not "sofa sectional")
4. **Location Names**: Use city names or neighborhoods (e.g., "Houston" or "Downtown Houston")
## Project Structure
```
marketplace-api/
├── app.py # Flask app entry point (Railway deployment)
├── MarketplaceAPI.py # Flask API routes and endpoints
├── MarketplaceScraper.py # Facebook GraphQL client with rate limiting & proxy
├── requirements.txt # Python dependencies
├── Procfile # Railway start command
├── railway.json # Railway deployment configuration
├── .gitignore # Git ignore rules
├── .env.example # Environment variables template
│
├── tests/ # Test scripts
│ ├── test_search.py # Comprehensive test suite
│ ├── test_search_simple.py # Quick test script
│ ├── test_production.py # Production API tests
│ ├── test_proxy.py # Proxy connection test
│ └── debug_facebook_response.py # Debug tool
│
├── docs/ # Documentation
│ └── filter-params.txt # Advanced filter parameters
│
└── Facebook_Marketplace_API.postman_collection.json # Postman collection
```
### Core Files
- **app.py**: Entry point for Railway, exports Flask app for gunicorn
- **MarketplaceAPI.py**: Flask routes (`/docs`, `/locations`, `/search`)
- **MarketplaceScraper.py**: Core scraping logic with rate limiting and compression handling
- **requirements.txt**: Python dependencies
- **Procfile**: Railway start command
- **railway.json**: Railway-specific configuration
### Available Endpoints
- `GET /docs` or `GET /api-docs` - API documentation
- `GET /locations` - Get location coordinates
- `GET /search` - Search marketplace listings
## Dependencies
- Flask 2.2.5 - Web framework
- requests 2.32.4 - HTTP client
- gunicorn 22.0.0 - Production WSGI server
- brotli 1.1.0 - Brotli decompression support
## Production Deployment
### Pre-Deployment Checklist
- ✅ **Proxy Configuration**: Set proxy environment variables in Railway to avoid IP blocking
- ✅ **Environment Variables**: Configure `PROXY_USERNAME`, `PROXY_PASSWORD`, `PROXY_HOST`, `PROXY_PORT`
- ✅ **Rate Limiting**: Delays are configured (10-20 seconds between requests)
- ✅ **Error Handling**: Automatic retry with exponential backoff
- ✅ **Logging**: Comprehensive debug logging enabled
- ✅ **Compression**: Automatic Brotli/Gzip decompression support
### Railway Deployment Steps
1. **Set Environment Variables** in Railway project settings:
```
PROXY_USERNAME=your_username
PROXY_PASSWORD=your_password
PROXY_HOST=gate.decodo.com
PROXY_PORT=7000
```
2. **Deploy**: Push to connected Git repository (Railway auto-deploys)
3. **Verify**: Check logs for "Proxy configured: ..." and test endpoints
### Verification
After deployment, verify:
- Logs show proxy configuration
- `/locations` endpoint returns data
- `/search` endpoint works without rate limiting
- No "IP blocked" errors in logs
## Notes
⚠️ **Important**:
- Facebook may rate limit requests - use proxy to avoid IP blocking
- Respect Facebook's Terms of Service
- Use responsibly and avoid excessive requests
- Rate limiting is handled automatically but may still occur
- The API automatically handles Brotli/Gzip decompression
- **Proxy is highly recommended** for production deployments to avoid Railway IP blocking
## License
See LICENSE file for details.