{"id":24355173,"url":"https://github.com/ssv445/js-rendering-proxy-docker","last_synced_at":"2026-04-25T08:36:49.891Z","repository":{"id":271418910,"uuid":"913382668","full_name":"ssv445/JS-Rendering-Proxy-Docker","owner":"ssv445","description":"JS Rendering Proxy API to Handle JS Website in Your Crawler.","archived":false,"fork":false,"pushed_at":"2025-03-27T07:23:28.000Z","size":589,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T08:29:46.874Z","etag":null,"topics":["crawler","proxy","puppeteer"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssv445.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-07T15:23:36.000Z","updated_at":"2025-03-27T07:23:31.000Z","dependencies_parsed_at":"2025-02-27T13:55:25.245Z","dependency_job_id":"90c4123b-c856-4adb-9278-d0404ad250eb","html_url":"https://github.com/ssv445/JS-Rendering-Proxy-Docker","commit_stats":null,"previous_names":["ssv445/dockerized-pupeteer-scraper-api","ssv445/js-rendering-proxy-docker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ssv445/JS-Rendering-Proxy-Docker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssv445%2FJS-Rendering-Proxy-Docker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssv445%2FJS-Rendering-Proxy-Docker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssv445%2FJS-Rendering-Proxy-Docker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssv445%2FJS-Rendering-Proxy-Docker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssv445","download_url":"https://codeload.github.com/ssv445/JS-Rendering-Proxy-Docker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssv445%2FJS-Rendering-Proxy-Docker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261828809,"owners_count":23216041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","proxy","puppeteer"],"created_at":"2025-01-18T17:29:26.262Z","updated_at":"2026-04-25T08:36:49.850Z","avatar_url":"https://github.com/ssv445.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JS-Rendering-Proxy-Docker\n\nA containerized web scraping Proxy API built with Node.js, Fastify, and Puppeteer.\n\n## Why This Exists\n\n- Traditional PHP crawlers can't process JavaScript, making them ineffective for:\n\n  - React applications\n  - Angular applications\n  - Other JavaScript-heavy sites\n  - Single-page applications (SPAs)\n\n- This proxy service:\n  - Acts as a bridge between PHP crawlers and modern web apps\n  - Returns fully rendered HTML after JavaScript execution\n  - Requires no modifications to existing crawler logic\n  - Handles client-side rendering transparently\n  - Does not follow redirects\n  - Returns all the HTTP errors as it is\n  - Blocks CSS and media.\n\n## Features\n\n- Runs Puppeteer in a Docker container\n- Built on Fastify for high performance\n- Uses slim Node.js base image\n- Includes all required Chromium dependencies\n- Configurable through environment variables\n- Development container support for VS Code\n- Blocks CSS, media and configurable JS resources\n- Request rate limiting and concurrent request control\n- Detailed request logging\n- Process cleanup for zombie/orphaned Chrome processes\n- Graceful timeout and error handling\n- Configurable redirect behavior\n\n## Prerequisites\n\n- Docker\n- Node.js 18+\n- VS Code (optional)\n\n## Quick Start\n\n1. Clone the repository\n\n2. Build and run with Docker:\n\n```\nbash\ndocker build -t puppeteer-scraper .\ndocker run -p 3000:3000 puppeteer-scraper\n```\n\n3. Or use VS Code Dev Containers:\n   - Install \"Remote - Containers\" extension\n   - Open project in VS Code\n   - Click \"Reopen in Container\"\n\n## Environment Variables\n\n- `PUPPETEER_SKIP_CHROMIUM_DOWNLOAD`: Set to `true` (uses system Chromium)\n- `EXECUTABLE_PATH`: Path to Chromium executable (default: `/usr/bin/chromium`)\n- `API_KEY`: Set your secret API key for authentication (optional)\n- `DEBUG`: Enable debug logging (optional)\n- `SERVER_NAME`: Server identifier for logs (optional)\n- `MAX_CONCURRENT_REQUESTS`: Maximum concurrent requests allowed (default: 10)\n\n## Development\n\nThe project includes a devcontainer configuration with:\n\n- ESLint\n- Prettier\n- Git tools\n- NPM utilities\n- Path completion\n- Auto-closing tags\n- Other helpful VS Code extensions\n\n## API Endpoints\n\nThe only endpoint is render, which accepts URL to render and return HTML. API key must be provided in the X-API-Key header.\n\n```\nhttp://localhost:3000/?render_url=[The-URL-To-Render]\nX-API-Key: YOUR_API_KEY\n```\n\nOr you can use it as a proxy.\n\n```javascript\n// Using axios:\nconst axios = require(\"axios\");\n\nconst proxyClient = axios.create({\n  proxy: {\n    host: \"localhost\",\n    port: 3000,\n    protocol: \"http\",\n  },\n  headers: {\n    \"X-API-Key\": \"YOUR_API_KEY\",\n  },\n});\n\n// Make requests through proxy\nawait proxyClient.get(\"https://example.com\");\n```\n\n## API Headers\n\nThe following custom headers can be used to control the page rendering behavior:\n\n| Header                   | Description                                                                                                 | Default                                                                                                                     |\n| ------------------------ | ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |\n| `user-agent`             | User agent string to use for the browser                                                                    | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36` |\n| `x-page-timeout-ms`      | Maximum time to wait for page load in milliseconds                                                          | 60000                                                                                                                       |\n| `x-wait-until-condition` | When to consider navigation successful. Options: `load`, `domcontentloaded`, `networkidle0`, `networkidle2` | `networkidle2`                                                                                                              |\n| `x-block-js`             | Blocks given JS file names (comma separated), using file names                                              | []                                                                                                                          |\n| `x-follow-redirects`     | Whether to follow redirects                                                                                 | false                                                                                                                       |\n| `x-api-key`              | API key for authentication (required if API_KEY env var is set)                                             | null                                                                                                                        |\n\n### CURL Support\n\nThis proxy does not support CURL requests because CURL always uses tunneling for HTTPS URLs. For HTTPS requests, use alternative HTTP clients like:\n\n- axios\n- fetch\n- got\n- node-fetch\n- request\n- superagent\n\nOr disable HTTPS tunneling in CURL (not possible as if now).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssv445%2Fjs-rendering-proxy-docker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssv445%2Fjs-rendering-proxy-docker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssv445%2Fjs-rendering-proxy-docker/lists"}