{"id":34255102,"url":"https://github.com/jongan69/fb-marketplace-api","last_synced_at":"2026-03-10T22:03:37.223Z","repository":{"id":323972599,"uuid":"1095455448","full_name":"jongan69/fb-marketplace-api","owner":"jongan69","description":"demo api","archived":false,"fork":false,"pushed_at":"2025-11-13T05:59:23.000Z","size":101,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-13T06:16:51.825Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jongan69.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-13T04:29:18.000Z","updated_at":"2025-11-13T05:59:25.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jongan69/fb-marketplace-api","commit_stats":null,"previous_names":["jongan69/fb-marketplace-api"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jongan69/fb-marketplace-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jongan69%2Ffb-marketplace-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jongan69%2Ffb-marketplace-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jongan69%2Ffb-marketplace-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jongan69%2Ffb-marketplace-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jongan69","download_url":"https://codeload.github.com/jongan69/fb-marketplace-api/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jongan69%2Ffb-marketplace-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30357614,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"ssl_error","status_checked_at":"2026-03-10T21:40:59.357Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-16T12:31:51.250Z","updated_at":"2026-03-10T22:03:37.218Z","avatar_url":"https://github.com/jongan69.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Facebook Marketplace API\n\nAn easy-to-use Facebook Marketplace API that functions without the need to log into a Facebook account. Wraps the Facebook GraphQL API, allowing for quick and easy retrieval of Facebook Marketplace listings and other relevant Marketplace data.\n\n## Features\n\n- 🔍 Search Facebook Marketplace listings by location and keywords\n- 📍 Get location coordinates for any city/location\n- 🚀 Production-ready with Railway deployment support\n- 🛡️ Built-in rate limiting and retry logic\n- 📊 Comprehensive logging for debugging\n- 🔄 Automatic Brotli and Gzip decompression support\n\n## Quick Start\n\n### Installation\n\n```bash\npip install -r requirements.txt\n```\n\n### Running Locally\n\n1. **Optional**: Configure proxy (recommended to avoid IP blocking)\n   ```bash\n   cp .env.example .env\n   # Edit .env with your proxy credentials\n   ```\n\n2. **Start the API:**\n   ```bash\n   python3 app.py\n   ```\n\nThe API will start on `http://localhost:5001` (or next available port).\n\n### Production Deployment (Railway)\n\nThe API is configured for Railway deployment with:\n- `Procfile` - Gunicorn production server\n- `railway.json` - Railway-specific configuration\n- Automatic port detection from `$PORT` environment variable\n\n**Production URL**: `https://fb-marketplace-api-production.up.railway.app`\n\n### Proxy Configuration (Optional)\n\nTo use a proxy service (e.g., Decodo) to avoid IP-based blocking, set these environment variables:\n\n**Local Development (.env file):**\n```bash\nPROXY_USERNAME=your_username\nPROXY_PASSWORD=your_password\nPROXY_HOST=gate.decodo.com\nPROXY_PORT=7000\n```\n\n**Railway (Environment Variables):**\nAdd these in your Railway project settings → Variables:\n- `PROXY_USERNAME` = your proxy username\n- `PROXY_PASSWORD` = your proxy password\n- `PROXY_HOST` = proxy hostname (e.g., `gate.decodo.com`)\n- `PROXY_PORT` = proxy port (e.g., `7000`)\n\n**Testing Proxy:**\n```bash\npython3 tests/test_proxy.py\n```\n\nThe proxy will be used automatically for all Facebook API requests when configured. This helps avoid IP-based blocking that occurs with server IPs.\n\n**Supported Proxy Services:**\n- Decodo\n- Any HTTP/HTTPS proxy service\n- Residential proxies\n- Datacenter proxies\n\n## API Endpoints\n\n### GET /docs or GET /api-docs\n\nGet comprehensive API documentation in JSON format.\n\n**Example:**\n```bash\ncurl \"http://localhost:5001/docs\"\n```\n\n**Response:**\nReturns complete API documentation including:\n- All available endpoints\n- Request parameters\n- Response formats\n- Example requests\n- Features and configuration\n\n### GET /locations\n\nGet latitude and longitude coordinates for a location.\n\n**Parameters:**\n- `locationQuery` (required): Location name (e.g., \"Houston\", \"New York\")\n\n**Example:**\n```bash\ncurl \"http://localhost:5001/locations?locationQuery=Houston\"\n```\n\n**Response:**\n```json\n{\n    \"status\": \"Success\",\n    \"error\": {},\n    \"data\": {\n        \"locations\": [\n            {\n                \"name\": \"Houston, Texas\",\n                \"latitude\": \"29.7602\",\n                \"longitude\": \"-95.3694\"\n            }\n        ]\n    }\n}\n```\n\n### GET /search\n\nSearch for marketplace listings in a specific location.\n\n**Parameters:**\n- `locationLatitude` (required): Latitude coordinate\n- `locationLongitude` (required): Longitude coordinate\n- `listingQuery` (required): Product/search keywords\n\n**Example:**\n```bash\ncurl \"http://localhost:5001/search?locationLatitude=29.7602\u0026locationLongitude=-95.3694\u0026listingQuery=couch\"\n```\n\n**Response:**\n```json\n{\n    \"status\": \"Success\",\n    \"error\": {},\n    \"data\": {\n        \"listingPages\": [\n            {\n                \"listings\": [\n                    {\n                        \"id\": \"123456789\",\n                        \"name\": \"Sectional Couch\",\n                        \"currentPrice\": \"$150\",\n                        \"previousPrice\": \"\",\n                        \"saleIsPending\": \"false\",\n                        \"primaryPhotoURL\": \"https://...\",\n                        \"sellerName\": \"John Doe\",\n                        \"sellerLocation\": \"Houston, Texas\",\n                        \"sellerType\": \"User\"\n                    }\n                ]\n            }\n        ]\n    }\n}\n```\n\n## Response Format\n\nAll endpoints return JSON in this format:\n\n```json\n{\n    \"status\": \"Success\" | \"Failure\",\n    \"error\": {\n        \"source\": \"User\" | \"Facebook\" | \"Request\",\n        \"message\": \"Error description\"\n    },\n    \"data\": {\n        // Response data (varies by endpoint)\n    }\n}\n```\n\n## Testing\n\n### Quick Test Script\n\n```bash\npython3 tests/test_search_simple.py Houston couch\n```\n\n### Comprehensive Test Suite\n\n```bash\npython3 tests/test_search.py\n```\n\n### Production Testing\n\n```bash\npython3 tests/test_production.py\n```\n\n### Debug Tool\n\n```bash\npython3 tests/debug_facebook_response.py\n```\n\n## Rate Limiting\n\nThe API includes built-in rate limiting protection to work around Facebook's rate limits.\n\n### Critical Finding: IP-Based Blocking\n\n**Important**: If you see rate limiting on the **FIRST request**, this indicates **IP-based blocking**, not request pattern issues. Railway's IP address may be flagged/blacklisted by Facebook.\n\n**Evidence**:\n- Rate limit on first request (`Request count: 1`)\n- Error code `1675004` (Facebook rate limit error)\n- No rate limit headers provided by Facebook\n\n### Why This Happens\n\n1. **Railway IP Reputation**: Railway uses shared IP addresses that may have been used by other scrapers\n2. **Server IP Detection**: Facebook detects server IPs vs browser IPs and blocks them aggressively\n3. **No Authentication**: Without cookies/auth, Facebook treats requests as suspicious\n4. **GraphQL Endpoint**: The `/api/graphql/` endpoint is heavily protected\n\n### Solutions\n\n#### Option 1: Use Residential Proxy (For Scraping)\n- Use a residential proxy service\n- Rotate IP addresses\n- Appear as real browser traffic\n- **Note**: May violate Facebook ToS\n\n#### Option 2: Wait for IP Reputation Reset\n- Wait 24-48 hours for IP reputation to reset\n- May not work if IP is permanently flagged\n\n#### Option 3: Use Different Hosting Provider\n- Try deploying to a different provider (AWS, GCP, etc.)\n- May have better IP reputation\n- Still may get blocked eventually\n\n#### Option 4: Use Facebook Official API (Recommended for Production)\n- Use Facebook's official Graph API with proper authentication\n- Requires app registration and OAuth\n- Has official rate limits but won't be blocked immediately\n\n#### Option 5: Test from Local Machine\n- Residential IPs are less likely to be blocked\n- Good for testing before deployment\n\n### Implemented Strategies\n\n1. **Request Throttling**: Random delays (10-20 seconds) between requests\n2. **Exponential Backoff Retry**: Automatic retry with increasing wait times (120s, 240s, 360s)\n3. **User-Agent Rotation**: Randomly selects from multiple modern browser user agents\n4. **Enhanced HTTP Headers**: Realistic browser headers (Referer, Origin, Accept, etc.)\n5. **Session Management**: Uses `requests.Session()` for connection pooling\n6. **Rate Limit Detection**: Detects rate limit errors and automatically retries\n7. **Comprehensive Debugging**: Detailed logs to identify rate limit causes\n\n### Configuration\n\nYou can adjust these settings at the top of `MarketplaceScraper.py`:\n\n```python\nMIN_DELAY_BETWEEN_REQUESTS = 10.0  # Minimum seconds between requests\nMAX_DELAY_BETWEEN_REQUESTS = 20.0  # Maximum seconds between requests\nRATE_LIMIT_RETRY_DELAY = 120  # Seconds to wait after rate limit error\nMAX_RETRIES = 3  # Maximum retries for rate-limited requests\n```\n\n### How It Works\n\n1. **Before each request**: Checks time since last request and waits if needed\n2. **During request**: Uses random user agent and realistic browser headers\n3. **After rate limit detected**: Automatically waits and retries up to 3 times\n4. **Debug logging**: Logs detailed information when rate limits occur\n\n### Rate Limit Indicators\n\nWatch for these indicators in logs:\n- **Rate limit on first request** = IP blocked (Railway IP flagged)\n- **Rate limit after multiple requests** = Pattern detected (request frequency too high)\n- **Rate limit headers present** = Temporary limit (can retry with backoff)\n- **No rate limit headers** = Permanent block (IP flagged)\n\n### Limitations\n\n⚠️ **Important Notes**:\n- Facebook's Terms of Service: Scraping Facebook may violate their ToS. Use responsibly.\n- Rate Limits Still Apply: These strategies reduce rate limiting but don't eliminate it entirely.\n- IP-based Limits: If your server IP gets flagged, delays won't help - need different IP or wait for reset.\n- Railway IP Reputation: Railway's shared IPs may already be flagged by Facebook.\n\n## Compression Support\n\nThe API automatically handles:\n- **Brotli** compression (used by Facebook on Railway)\n- **Gzip** compression\n- Automatic decompression and fallback handling\n\nThe `requests` library automatically decompresses Brotli when the `brotli` library is available. The code detects already-decompressed content and skips unnecessary decompression attempts.\n\n## Postman Examples\n\n### Base URLs\n- **Local**: `http://localhost:5001`\n- **Production**: `https://fb-marketplace-api-production.up.railway.app`\n\n### Example Requests\n\n#### Get Locations\n```\nGET http://localhost:5001/locations?locationQuery=Houston\nGET http://localhost:5001/locations?locationQuery=New York\nGET http://localhost:5001/locations?locationQuery=Los Angeles\n```\n\n#### Search Listings\n```\nGET http://localhost:5001/search?locationLatitude=29.7602\u0026locationLongitude=-95.3694\u0026listingQuery=couch\nGET http://localhost:5001/search?locationLatitude=25.8138\u0026locationLongitude=-80.1335\u0026listingQuery=cars\nGET http://localhost:5001/search?locationLatitude=40.7128\u0026locationLongitude=-74.0060\u0026listingQuery=bicycle\n```\n\n### Postman Collection Setup\n\n1. **Create New Collection**: \"Facebook Marketplace API\"\n2. **Add Environment Variables** (optional):\n   - `base_url`: `http://localhost:5001` (or production URL)\n   - `houston_lat`: `29.7602`\n   - `houston_lon`: `-95.3694`\n3. **Use Variables in URLs**:\n   ```\n   {{base_url}}/locations?locationQuery=Houston\n   {{base_url}}/search?locationLatitude={{houston_lat}}\u0026locationLongitude={{houston_lon}}\u0026listingQuery=couch\n   ```\n\n### Complete Search Workflow\n\n1. **Get Location Coordinates**\n   ```\n   GET {{base_url}}/locations?locationQuery=Houston\n   ```\n   - Copy `latitude` and `longitude` from response\n\n2. **Search for Products**\n   ```\n   GET {{base_url}}/search?locationLatitude=29.7602\u0026locationLongitude=-95.3694\u0026listingQuery=couch\n   ```\n   - Use coordinates from step 1\n\n### Common Test Scenarios\n\n**Popular Cities**: Houston, New York, Los Angeles, Chicago, Miami, San Francisco\n\n**Popular Products**: couch, laptop, car, bicycle, iphone, furniture, tv, bike\n\n**Error Cases**: Missing parameters will return user errors\n\n### Tips\n\n1. **Rate Limiting**: If you get rate limit errors, wait 60+ seconds between requests\n2. **Coordinates**: Always get fresh coordinates using `/locations` endpoint\n3. **Product Names**: Use simple, common terms (e.g., \"couch\" not \"sofa sectional\")\n4. **Location Names**: Use city names or neighborhoods (e.g., \"Houston\" or \"Downtown Houston\")\n\n## Project Structure\n\n```\nmarketplace-api/\n├── app.py                          # Flask app entry point (Railway deployment)\n├── MarketplaceAPI.py               # Flask API routes and endpoints\n├── MarketplaceScraper.py           # Facebook GraphQL client with rate limiting \u0026 proxy\n├── requirements.txt                # Python dependencies\n├── Procfile                        # Railway start command\n├── railway.json                    # Railway deployment configuration\n├── .gitignore                      # Git ignore rules\n├── .env.example                    # Environment variables template\n│\n├── tests/                          # Test scripts\n│   ├── test_search.py             # Comprehensive test suite\n│   ├── test_search_simple.py      # Quick test script\n│   ├── test_production.py         # Production API tests\n│   ├── test_proxy.py              # Proxy connection test\n│   └── debug_facebook_response.py # Debug tool\n│\n├── docs/                           # Documentation\n│   └── filter-params.txt          # Advanced filter parameters\n│\n└── Facebook_Marketplace_API.postman_collection.json  # Postman collection\n```\n\n### Core Files\n\n- **app.py**: Entry point for Railway, exports Flask app for gunicorn\n- **MarketplaceAPI.py**: Flask routes (`/docs`, `/locations`, `/search`)\n- **MarketplaceScraper.py**: Core scraping logic with rate limiting and compression handling\n- **requirements.txt**: Python dependencies\n- **Procfile**: Railway start command\n- **railway.json**: Railway-specific configuration\n\n### Available Endpoints\n\n- `GET /docs` or `GET /api-docs` - API documentation\n- `GET /locations` - Get location coordinates\n- `GET /search` - Search marketplace listings\n\n## Dependencies\n\n- Flask 2.2.5 - Web framework\n- requests 2.32.4 - HTTP client\n- gunicorn 22.0.0 - Production WSGI server\n- brotli 1.1.0 - Brotli decompression support\n\n## Production Deployment\n\n### Pre-Deployment Checklist\n\n- ✅ **Proxy Configuration**: Set proxy environment variables in Railway to avoid IP blocking\n- ✅ **Environment Variables**: Configure `PROXY_USERNAME`, `PROXY_PASSWORD`, `PROXY_HOST`, `PROXY_PORT`\n- ✅ **Rate Limiting**: Delays are configured (10-20 seconds between requests)\n- ✅ **Error Handling**: Automatic retry with exponential backoff\n- ✅ **Logging**: Comprehensive debug logging enabled\n- ✅ **Compression**: Automatic Brotli/Gzip decompression support\n\n### Railway Deployment Steps\n\n1. **Set Environment Variables** in Railway project settings:\n   ```\n   PROXY_USERNAME=your_username\n   PROXY_PASSWORD=your_password\n   PROXY_HOST=gate.decodo.com\n   PROXY_PORT=7000\n   ```\n\n2. **Deploy**: Push to connected Git repository (Railway auto-deploys)\n\n3. **Verify**: Check logs for \"Proxy configured: ...\" and test endpoints\n\n### Verification\n\nAfter deployment, verify:\n- Logs show proxy configuration\n- `/locations` endpoint returns data\n- `/search` endpoint works without rate limiting\n- No \"IP blocked\" errors in logs\n\n## Notes\n\n⚠️ **Important**: \n- Facebook may rate limit requests - use proxy to avoid IP blocking\n- Respect Facebook's Terms of Service\n- Use responsibly and avoid excessive requests\n- Rate limiting is handled automatically but may still occur\n- The API automatically handles Brotli/Gzip decompression\n- **Proxy is highly recommended** for production deployments to avoid Railway IP blocking\n\n## License\n\nSee LICENSE file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjongan69%2Ffb-marketplace-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjongan69%2Ffb-marketplace-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjongan69%2Ffb-marketplace-api/lists"}