{"id":51324051,"url":"https://github.com/myselfshravan/third-eye","last_synced_at":"2026-07-01T16:30:46.987Z","repository":{"id":365186271,"uuid":"1270555495","full_name":"myselfshravan/third-eye","owner":"myselfshravan","description":"👁️ Open-source page-intelligence API - screenshot AND extract product images + data (title/brand/price/sizes) from any site, even bot-protected ones. Patchright stealth · Playwright · Fastify. Self-hosted, free.","archived":false,"fork":false,"pushed_at":"2026-06-26T12:19:00.000Z","size":706,"stargazers_count":1,"open_issues_count":12,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-26T12:19:25.142Z","etag":null,"topics":["bot-detection","bullmq","canvaskit","docker","ecommerce","fastify","flutter","headless-chrome","nodejs","patchright","pdf","playwright","product-extraction","screenshot","screenshot-api","self-hosted","typescript","web-scraping","webgl","website-screenshot"],"latest_commit_sha":null,"homepage":"https://github.com/myselfshravan/third-eye#readme","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/myselfshravan.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-15T20:38:33.000Z","updated_at":"2026-06-26T12:19:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/myselfshravan/third-eye","commit_stats":null,"previous_names":["myselfshravan/third-eye"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/myselfshravan/third-eye","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myselfshravan%2Fthird-eye","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myselfshravan%2Fthird-eye/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myselfshravan%2Fthird-eye/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myselfshravan%2Fthird-eye/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/myselfshravan","download_url":"https://codeload.github.com/myselfshravan/third-eye/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/myselfshravan%2Fthird-eye/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35015052,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-01T02:00:05.325Z","response_time":130,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot-detection","bullmq","canvaskit","docker","ecommerce","fastify","flutter","headless-chrome","nodejs","patchright","pdf","playwright","product-extraction","screenshot","screenshot-api","self-hosted","typescript","web-scraping","webgl","website-screenshot"],"created_at":"2026-07-01T16:30:46.167Z","updated_at":"2026-07-01T16:30:46.975Z","avatar_url":"https://github.com/myselfshravan.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 👁️ third-eye\n\n\u003e **The open-source page-intelligence API.** Screenshot *and* extract product\n\u003e images + structured data from any site — even bot-protected ones.\n\u003e Self-hosted, free, fast.\n\n[![CI](https://github.com/myselfshravan/third-eye/actions/workflows/ci.yml/badge.svg)](https://github.com/myselfshravan/third-eye/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Node](https://img.shields.io/badge/node-%E2%89%A522-43853d.svg)](.nvmrc)\n[![TypeScript](https://img.shields.io/badge/TypeScript-strict-3178c6.svg)](tsconfig.json)\n[![Playwright](https://img.shields.io/badge/engine-Patchright%2FChromium-2EAD33.svg)](https://playwright.dev)\n[![PRs welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)\n\nEvery commercial screenshot API charges $9–$79/mo, throttles you, and **still\ngets blocked** by modern bot protection. third-eye is the free, self-hostable\nalternative that goes further: point it at a product page and it returns the\n**actual product images + structured data** (title, brand, price, sizes), not\njust a PNG — purpose-built to feed similar-image search and catalog ingestion.\n\n- 🛡️ **Passes bot detection** — Patchright (patched Chromium) defeats the\n  headless/automation tells that block stock Playwright/Puppeteer. Captures\n  Shopify, Next.js, SPAs, Uniqlo, and more where rivals 403.\n- 🛍️ **Product extraction** — `/v1/extract` pulls product images + data via\n  JSON-LD / OpenGraph / Shopify-JSON / DOM heuristics. *Returns the image from\n  the URL*, with a screenshot fallback.\n- 🧠 **Readiness oracle** — network-idle + `fonts.ready` + lazy-load scroll +\n  animation freeze + **canvas/Flutter first-frame detection** (Flutter/WebGL\n  apps render, not blank).\n- 🖼️ **Screenshots done right** — full-page single-pass (no scroll-stitch\n  seams), element clip, device emulation, dark mode, PNG/JPEG/WebP/PDF.\n- ♻️ **Warm browser pool** — per-request isolation, recycle-after-N, crash\n  self-heal. ⚡ **Sync + async (webhooks) + bulk** via Redis/BullMQ.\n- 🔑 API-key auth, per-plan rate limits, Prometheus metrics, graceful shutdown.\n- 🧩 **Pluggable** storage (`none`/`s3`/`local`), devices, readiness steps,\n  output formats, extractors — see [EXTENDING.md](EXTENDING.md).\n- ☁️ One Docker image → any VPS, published via **Cloudflare Tunnel**. **Not**\n  serverless (cold starts + no GL stack break the warm pool and canvas apps).\n\n## How it compares\n\n| | third-eye | commercial screenshot APIs |\n|---|---|---|\n| **Price** | **Free / self-hosted (MIT)** | $9–$79/mo, then per-shot overage |\n| **Rate limits** | yours to set | 40–100/min typical |\n| **Passes modern bot detection** | ✅ Patchright | ❌ mostly blocked |\n| **Product image + data extraction** | ✅ built-in | ❌ none |\n| **Canvas/Flutter/WebGL rendering** | ✅ SwiftShader + first-frame wait | ⚠️ often blank |\n| **Screenshots / PDF / full-page** | ✅ | ✅ |\n| **Data ownership** | 100% yours | vendor-hosted |\n\n*Compared against allscreenshots, pikwy, site-shot, screenshotapi, microlink,\nscreenshotone, screenshotapi.net, urlbox. Hardest-tier marketplaces behind\nAkamai sensor-data / PerimeterX (e.g. H\u0026M, Zara, Myntra) still require\nresidential proxies — see [Limitations](#limitations).*\n\n## Quick start (local)\n\n```bash\ncp .env.example .env\nnpm install\nnpm run browsers:install        # one-time: Chromium + OS deps\n\n# Engine smoke test — no server/Redis needed:\nnpm run smoke -- https://example.com\nnpm run smoke -- https://flutter.dev --full   # exercises the canvas path\n\n# Full stack (API + worker + Redis) via Docker:\ndocker compose up --build\n```\n\n## API\n\nBase URL `http://localhost:8080`. Auth via `x-api-key` (dev key:\n`te_dev_local`). Add `?response=binary|base64|json` (default `binary`).\n\n### Synchronous\n\n```bash\ncurl -X POST http://localhost:8080/v1/screenshot \\\n  -H 'x-api-key: te_dev_local' -H 'content-type: application/json' \\\n  -d '{\"url\":\"https://example.com\",\"fullPage\":true,\"format\":\"png\"}' \\\n  --output shot.png\n```\n\nConvenience GET (browser-friendly):\n\n```\nGET /v1/screenshot?url=https://example.com\u0026full_page=true\u0026device=iphone-15\n```\n\n### Asynchronous (webhook)\n\n```bash\ncurl -X POST http://localhost:8080/v1/screenshot/async \\\n  -H 'x-api-key: te_dev_local' -H 'content-type: application/json' \\\n  -d '{\"url\":\"https://example.com\",\"webhookUrl\":\"https://you.dev/hook\"}'\n# → { \"jobId\": \"...\", \"status\": \"queued\" }\n\ncurl http://localhost:8080/v1/jobs/\u003cjobId\u003e -H 'x-api-key: te_dev_local'\n```\n\n### Bulk\n\n```bash\ncurl -X POST http://localhost:8080/v1/bulk \\\n  -H 'x-api-key: te_dev_local' -H 'content-type: application/json' \\\n  -d '{\"urls\":[\"https://a.com\",\"https://b.com\"],\"options\":{\"device\":\"desktop-hd\"}}'\n```\n\n### 🛍️ Product extraction (the wedge)\n\nPoint at a product page (PDP); get the product images + structured data back.\n\n```bash\ncurl -X POST http://localhost:8080/v1/extract \\\n  -H 'x-api-key: te_dev_local' -H 'content-type: application/json' \\\n  -d '{\"url\":\"https://bluorng.com/products/flyway-linen-shirt\"}'\n```\n```jsonc\n{\n  \"title\": \"Flyway Linen Shirt\",\n  \"brand\": \"Bluorng\",\n  \"price\": 8200, \"currency\": \"INR\",\n  \"sizes\": [\"XS\",\"S\",\"M\",\"L\",\"XL\",\"XXL\"],\n  \"images\": [ { \"url\": \"https://cdn.shopify.com/.../rvet5refd.jpg\", \"source\": \"shopify\" }, … ],\n  \"primaryImage\": \"https://cdn.shopify.com/.../rvet5refd.jpg\",\n  \"confidence\": \"high\",          // high = JSON-LD/Shopify · medium = OG · low = DOM heuristics\n  \"sources\": [\"shopify\",\"og\"],\n  \"blocked\": false\n}\n```\n\nReturn the **primary product image bytes** directly (screenshot fallback if none):\n\n```bash\ncurl -X POST 'http://localhost:8080/v1/extract?response=image' \\\n  -H 'x-api-key: te_dev_local' -H 'content-type: application/json' \\\n  -d '{\"url\":\"https://www.uniqlo.com/in/en/products/E482443-000/00\"}' --output product.jpg\n```\n\nListing (PLP) → every product card; plus async extraction:\n\n```bash\ncurl -X POST http://localhost:8080/v1/extract/listing -H 'x-api-key: te_dev_local' \\\n  -H 'content-type: application/json' -d '{\"url\":\"https://bluorng.com/collections/all\"}'\ncurl -X POST http://localhost:8080/v1/extract/async   -H 'x-api-key: te_dev_local' \\\n  -H 'content-type: application/json' -d '{\"url\":\"...\",\"webhookUrl\":\"https://you.dev/hook\"}'\n```\n\nExtraction strategy (precedence): **JSON-LD `Product` → OpenGraph → Shopify\n`.json` → DOM gallery heuristics**, normalized to absolute URLs and deduped.\n\n### Key options (see [`src/core/schema.ts`](src/core/schema.ts) for the full contract)\n\n| field | type | notes |\n|---|---|---|\n| `url` | string (required) | target page |\n| `format` | `png` \\| `jpeg` \\| `webp` | default `png` |\n| `pdf` | bool | render PDF instead of an image |\n| `fullPage` | bool | single-pass full-page capture |\n| `selector` / `clip` | string / rect | capture one element or region |\n| `device` | e.g. `iphone-15`, `desktop-hd` | preset viewport + DPR + UA |\n| `viewport` / `deviceScaleFactor` | rect / number | manual surface |\n| `darkMode`, `reducedMotion`, `locale`, `timezone` | | emulation |\n| `waitStrategy` | `auto` \\| `networkidle` \\| `load` \\| `domcontentloaded` | default `auto` |\n| `waitForSelector` / `waitForFunction` / `delayMs` | | extra readiness gates |\n| `blockAds`, `blockCookieBanners`, `hideSelectors`, `removeSelectors` | | clean shots |\n| `injectCss` / `injectJs` / `headers` / `cookies` | | page setup / auth |\n\nExtraction adds `maxImages` and `includeScreenshot`, and inherits the full\ncapture surface (device, stealth, waits, `proxy`).\n\n### ⚡ OG image (ultra-low-latency)\nJust the `og:image` URL, no browser on the happy path — a streamed fetch + parse,\ncached. ~150–500ms (vs seconds for a full render); blocked/no-og sites fall back\nto the browser automatically.\n```bash\ncurl -s \"https://your-host/v1/og?url=https://bluorng.com/products/flyway-linen-shirt\" \\\n  -H \"x-api-key: $KEY\"\n# → {\"image\":\"https://bluorng.com/cdn/shop/files/rvet5refd.jpg?v=…\"}\n```\n\n## Endpoints\n`POST /v1/screenshot` · `GET /v1/screenshot` · `POST /v1/screenshot/async` ·\n`POST /v1/extract` · `POST /v1/extract/async` · `POST /v1/extract/listing` ·\n`GET /v1/og` · `GET /v1/jobs/:id` · `POST /v1/bulk` · `GET /healthz` · `GET /readyz` · `GET /metrics`\n\n## How it works (deep dive)\nThe interesting engineering and the hard cases (Flutter/CanvasKit, when-is-a-page-\nready, full-page stitching, memory) are documented in [CLAUDE.md](CLAUDE.md).\n\n## Deploy\n\nOne Docker image, two roles (`api` + `worker`). Deploys to any Linux box with\nDocker. Recommended: **VPS / EC2 + Cloudflare Tunnel** (TLS + public hostname,\nzero inbound ports). Runbooks: **[DEPLOY.md](DEPLOY.md)** (any VPS) ·\n**[DEPLOY-AWS.md](DEPLOY-AWS.md)** (EC2 + S3, with a one-paste bootstrap script).\n\n```bash\ngit clone https://github.com/myselfshravan/third-eye.git \u0026\u0026 cd third-eye\ncp .env.production.example .env   # set API_KEYS, TUNNEL_TOKEN, pool sizes\ndocker compose -f docker-compose.prod.yml --env-file .env up -d --build\ndocker compose -f docker-compose.prod.yml up -d --scale worker=3   # scale out\n```\n\n- **Storage:** Cloudflare R2 (zero egress) — set `STORAGE_DRIVER=s3` and the\n  `S3_*` vars; point `S3_PUBLIC_BASE_URL` at your R2/CDN domain. `local` and\n  `none` drivers are also built in (see [EXTENDING.md](EXTENDING.md)).\n\n\u003e Not built for serverless (Vercel/Lambda): cold starts kill the warm-pool\n\u003e advantage and there's no GL stack, so Flutter/WebGL pages render blank.\n\n## Testing many URLs at once\nDrop URLs into [`examples/test-urls.json`](examples/test-urls.json) and run the\nbatch runner — captures each straight through the engine and prints a summary:\n\n```bash\nnpm run batch                       # screenshot each URL → captures/\nnpm run batch -- --extract          # extract product data/images → *.product.json\nnpm run batch -- path/to/urls.json  # custom file\n```\n\n## Limitations\n- **Hardest-tier bot walls.** Sites on Akamai *sensor-data* / PerimeterX /\n  DataDome (e.g. H\u0026M, Zara, Myntra) block at the edge on TLS/IP reputation +\n  behavioral signals — *before* JS runs — so stealth alone won't pass them.\n  third-eye detects this and reports `blocked: true` + `httpStatus` honestly\n  rather than returning a fake \"success\". Cracking these needs residential\n  proxies (set `PROXY_URL` / per-request `proxy`) and is a roadmap item. Most\n  D2C/Shopify/Next.js storefronts work out of the box.\n- **No ML (yet).** Extraction is structured-data + heuristics. A vision-model\n  fallback and image embeddings (for direct similar-image search) are planned as\n  a pluggable enrichment step.\n\n## Project status\nPre-1.0 and under active development. Capture, **extraction**, stealth, API,\nworker, and deploy path are working and tested; see the [CHANGELOG](CHANGELOG.md)\nand [open issues](https://github.com/myselfshravan/third-eye/issues).\n\n## Contributing\nContributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and\nguidelines, [EXTENDING.md](EXTENDING.md) for how to add storage backends,\ndevices, readiness steps, or output formats, and [CLAUDE.md](CLAUDE.md) for the\narchitecture. By participating you agree to our\n[Code of Conduct](CODE_OF_CONDUCT.md).\n\n## Security\nFound a vulnerability? Please report it privately — see\n[SECURITY.md](SECURITY.md). Note the **SSRF** hardening guidance there before\nexposing third-eye publicly (it renders arbitrary user-supplied URLs).\n\n## License\n[MIT](LICENSE) © Shravan Revanna\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyselfshravan%2Fthird-eye","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmyselfshravan%2Fthird-eye","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmyselfshravan%2Fthird-eye/lists"}