{"id":29823780,"url":"https://github.com/litongjava/playwright-spider","last_synced_at":"2025-08-30T22:15:38.094Z","repository":{"id":273341068,"uuid":"853924398","full_name":"litongjava/playwright-spider","owner":"litongjava","description":"playwright-spider","archived":false,"fork":false,"pushed_at":"2025-02-18T05:48:03.000Z","size":189,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-29T02:43:29.937Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/litongjava.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-07T23:11:23.000Z","updated_at":"2025-07-19T20:23:53.000Z","dependencies_parsed_at":"2025-01-20T10:35:55.789Z","dependency_job_id":"bf84f1fb-4504-4ba4-84ac-865b8b9400d2","html_url":"https://github.com/litongjava/playwright-spider","commit_stats":null,"previous_names":["litongjava/playwright-server","litongjava/playwright-spider"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/litongjava/playwright-spider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litongjava%2Fplaywright-spider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litongjava%2Fplaywright-spider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litongjava%2Fplaywright-spider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litongjava%2Fplaywright-spider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/litongjava","download_url":"https://codeload.github.com/litongjava/playwright-spider/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/litongjava%2Fplaywright-spider/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272914554,"owners_count":25014443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-29T02:07:06.369Z","updated_at":"2025-08-30T22:15:38.086Z","avatar_url":"https://github.com/litongjava.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Playwright-Server\n\nThis project demonstrates how to integrate Playwright with TioBoot to create a web scraping service that reduces overhead by managing Playwright instances efficiently. The solution initializes the Playwright browser instance at service startup and closes it properly upon service shutdown, ensuring optimal performance for high concurrency and low-latency scenarios.\n\n## Features\n\n- **Efficient Resource Management**: Playwright instance and browser are initialized once on service startup and reused for multiple requests.\n- **Web Scraping**: Provides API endpoints for retrieving webpage content using Playwright.\n- **HTML to Markdown Conversion**: Converts HTML content to Markdown format using `com.vladsch.flexmark`.\n- **Dockerized Deployment**: The project is containerized for easy deployment.\n\n## Prerequisites\n\n- Java 1.8\n- Maven 3.x\n- Docker\n- Playwright dependencies (e.g., Chromium)\n\n## API Endpoints\n\n### Get Web Page Content\n\n**Endpoint**: `/playwright`\n\n**Parameters**: `url` - The URL of the web page to retrieve.\n\n**Example**:\n```bash\ncurl \"http://localhost/playwright?url=https://www.sjsu.edu/registrar/calendar/fall-2024.php\"\n```\n\n### Convert HTML to Markdown\n\n**Endpoint**: `/markdown`\n\n**Parameters**: `url` - The URL of the web page to retrieve and convert.\n\n**Example**:\n```bash\ncurl \"http://localhost/markdown?url=https://www.sjsu.edu/registrar/calendar/fall-2024.php\"\n```\n### a website\n```\ncurl http://127.0.0.1:8007/crawl/hawaii_kapiolani_web_page\n```\n## Build and Run with Docker\n\n### Dockerfile\n\n```Dockerfile\n# 第一阶段：构建阶段\nFROM litongjava/maven:3.8.8-jdk_21_0_6 AS builder\n\n# 设置工作目录\nWORKDIR /src\n\n# 复制pom.xml并下载依赖\nCOPY pom.xml /src/\nCOPY src /src/src\n\n# 运行maven打包命令\nRUN mvn package -DskipTests -Pproduction\n\n# 第二阶段：运行阶段\nFROM litongjava/jdk:21.0.6-chromium\n\n# 设置工作目录\nWORKDIR /app\n\n# 从构建阶段复制生成的jar文件到运行阶段\nCOPY --from=builder /src/target/playwright-server-1.0.0.jar /app/\n\n# 下载Playwright依赖\nRUN java -jar /app/playwright-server-1.0.0.jar --download\n\n# 运行jar文件\nCMD [\"java\",\"-jar\", \"playwright-server-1.0.0.jar\"]\n```\n\n### Build and Run\n\nBuild the Docker image:\n\n```bash\ndocker build -t litongjava/playwright-server:1.0.0 .\n```\n\nRun the Docker container:\n\n```bash\ndocker run -p 8080:80 litongjava/playwright-server:1.0.0\n```\n\n## Conclusion\n\nThis project integrates Playwright with TioBoot to provide a high-performance web scraping solution. By initializing the Playwright instance during service startup and releasing resources on shutdown, we can efficiently handle multiple requests without incurring the overhead of repeatedly starting the browser. The service is also containerized with Docker for easy deployment in any environment.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flitongjava%2Fplaywright-spider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flitongjava%2Fplaywright-spider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flitongjava%2Fplaywright-spider/lists"}