{"id":21978133,"url":"https://github.com/codeterrayt/scalable-website-screenshot","last_synced_at":"2026-01-27T18:08:22.623Z","repository":{"id":263249438,"uuid":"889795376","full_name":"codeterrayt/Scalable-Website-Screenshot","owner":"codeterrayt","description":"Scalable WebApp that generates full-page screenshots of websites. It leverages a distributed architecture for handling multiple screenshot requests simultaneously, providing real-time updates and efficient task management.","archived":false,"fork":false,"pushed_at":"2024-11-17T15:53:22.000Z","size":11112,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-28T03:43:50.108Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codeterrayt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-17T08:58:07.000Z","updated_at":"2024-12-11T13:02:37.000Z","dependencies_parsed_at":"2024-11-17T16:37:37.512Z","dependency_job_id":null,"html_url":"https://github.com/codeterrayt/Scalable-Website-Screenshot","commit_stats":null,"previous_names":["codeterrayt/scalable-website-screenshot-taker","codeterrayt/scalable-website-screenshot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeterrayt%2FScalable-Website-Screenshot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeterrayt%2FScalable-Website-Screenshot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeterrayt%2FScalable-Website-Screenshot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codeterrayt%2FScalable-Website-Screenshot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codeterrayt","download_url":"https://codeload.github.com/codeterrayt/Scalable-Website-Screenshot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245040220,"owners_count":20551297,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-29T16:18:05.141Z","updated_at":"2026-01-27T18:08:17.587Z","avatar_url":"https://github.com/codeterrayt.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scalable-Website-Screenshot\r\n\r\n**Scalable-Website-Screenshot** is a scalable web application that generates full-page screenshots of websites. It leverages a distributed architecture for handling multiple screenshot requests simultaneously, providing real-time updates and efficient task management.\r\n\r\n\r\n## Demo Video\r\nhttps://github.com/user-attachments/assets/50819078-08a2-470a-b896-03b425b9e3d2\r\n\r\n---\r\n\r\n## Low-Level Design\r\n\u003cp align=\"center\"\u003e\r\n  \u003cimg src=\"https://github.com/user-attachments/assets/48bbbe8d-a8e6-415e-8325-570a3d7a4d7f\" alt=\"Low-Level Design Diagram\"\u003e\r\n\u003c/p\u003e\r\n\r\n\r\n## Tech Stack\r\n\r\n- **Next.js** (Frontend): React-based framework for building the user interface.\r\n- **Redis**: Used for caching the screenshots and real-time communication (Pub/Sub).\r\n- **Fastify**: A fast and lightweight backend framework for handling HTTP requests.\r\n- **WebSocket**: Real-time updates are provided via WebSocket for job completion notifications.\r\n- **RabbitMQ**: Message queue used to handle the distribution of tasks to multiple workers.\r\n\r\n---\r\n\r\n## Installation\r\n\r\n### Clone \r\n```bash\r\ngit clone https://github.com/codeterrayt/Scalable-Website-Screenshot.git\r\ncd Scalable-Website-Screenshot\r\n```\r\n\r\n### Docker \r\nStart Redis and RabbitMQ Services\r\n```bash\r\ndocker-compose up\r\n```\r\n\r\n### 1. **Frontend (Next.js)**\r\n\r\nTo set up the frontend, navigate to the `NextJS/frontend` directory and install the dependencies:\r\n\r\n```bash\r\ncd NextJS/frontend\r\npnpm install \u0026\u0026 npm run dev\r\n```\r\n\r\nThis will start the Next.js server in development mode. You can access the application on `http://localhost:3000`.\r\n\r\n---\r\n\r\n### 2. **Backend \u0026 Worker**\r\n\r\nTo install the backend server and worker dependencies, run the following:\r\n\r\n```bash\r\npnpm install\r\n```\r\n\r\n---\r\n\r\n### 3. **Running the Server**\r\n\r\nTo start the backend server, run:\r\n\r\n```bash\r\nnode index.js\r\n```\r\n\r\nThis will start the Fastify server and make the API endpoints available for screenshot requests.\r\n\r\n---\r\n\r\n### 4. **Running the Worker(s)**\r\n\r\nTo start the worker(s) that process screenshot requests, run:\r\n\r\n```bash\r\nnode worker.js\r\n```\r\n\r\n#### Note:\r\nYou can run **multiple workers** depending on the number of parallel tasks you want to handle. The more workers you run, the more tasks can be processed simultaneously. For example:\r\n\r\n```bash\r\n# Run 5 workers in parallel:\r\nnode worker.js\r\nnode worker.js\r\nnode worker.js\r\nnode worker.js\r\nnode worker.js\r\n```\r\n\r\nEach worker is responsible for fetching a task from RabbitMQ, processing the screenshot, and saving caching it to Redis. Multiple workers will increase the throughput of screenshot generation, allowing the system to scale efficiently with higher load.\r\n\r\n---\r\n\r\n## How It Works\r\n\r\n1. **Frontend (Next.js)**: Users provide a URL via the frontend, and the request is sent to the Fastify backend.\r\n   \r\n2. **Backend (Fastify)**: Fastify checks Redis for a cached screenshot. If it's not found, the request is added to the RabbitMQ queue as a new task.\r\n\r\n3. **RabbitMQ**: The task is picked up by one of the available workers for processing.\r\n\r\n4. **Workers**: Workers use libraries like Puppeteer to generate the screenshot of the given URL. Once the screenshot is captured, it's saved and path cached in Redis for quick retrieval.\r\n\r\n5. **Redis (Pub/Sub)**: After the task is completed, the worker publishes an update to Redis, which the frontend subscribes to using WebSocket to provide real-time status updates to the user.\r\n\r\n6. **WebSocket**: Once the task is completed, the frontend is notified via WebSocket, and the screenshot is served to the user.\r\n\r\n---\r\n\r\n## Notes\r\n\r\n- **Scaling Workers**: You can increase the number of workers to process more tasks in parallel. The system is designed to scale horizontally by adding more worker instances.\r\n- **Redis Caching**: Once a screenshot is generated for a specific URL, it will be cached in Redis, reducing the load for future requests.\r\n  \r\nThe more workers you run, the more tasks can be processed simultaneously, improving throughput and system performance.\r\n\r\n---\r\n\r\n## License\r\n\r\n[MIT License](LICENSE)\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeterrayt%2Fscalable-website-screenshot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodeterrayt%2Fscalable-website-screenshot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodeterrayt%2Fscalable-website-screenshot/lists"}