{"id":13486983,"url":"https://github.com/kingschan1204/easyCrawl","last_synced_at":"2025-03-27T21:31:47.796Z","repository":{"id":154629185,"uuid":"623651941","full_name":"kingschan1204/easyCrawl","owner":"kingschan1204","description":"A crawler toolkit implemented in Java","archived":false,"fork":false,"pushed_at":"2025-03-18T08:55:36.000Z","size":359,"stargazers_count":65,"open_issues_count":0,"forks_count":10,"subscribers_count":4,"default_branch":"dev","last_synced_at":"2025-03-24T09:15:28.374Z","etag":null,"topics":["crawling-sites","stock-data"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kingschan1204.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-04T19:59:09.000Z","updated_at":"2025-03-18T08:55:40.000Z","dependencies_parsed_at":"2024-01-14T03:48:51.814Z","dependency_job_id":"30f6390e-bec1-4fb4-a4e1-743df3450d63","html_url":"https://github.com/kingschan1204/easyCrawl","commit_stats":{"total_commits":106,"total_committers":1,"mean_commits":106.0,"dds":0.0,"last_synced_commit":"7910dc0255746ed0fba017be1d8086a82eb529c8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingschan1204%2FeasyCrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingschan1204%2FeasyCrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingschan1204%2FeasyCrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kingschan1204%2FeasyCrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kingschan1204","download_url":"https://codeload.github.com/kingschan1204/easyCrawl/tar.gz/refs/heads/dev","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245927288,"owners_count":20695205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawling-sites","stock-data"],"created_at":"2024-07-31T18:00:54.060Z","updated_at":"2025-03-27T21:31:47.777Z","avatar_url":"https://github.com/kingschan1204.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"\u003e This project is a crawler toolkit implemented based on Java. The original intention is to crawl Internet data simply\n\u003e and efficiently.\n\n\n## Implemented Functions\n\n- HTTP/HTTPS GET, POST, PUT\n- File Download\n- HTTP and SOCKS5 Proxies\n- Convert cURL Commands to HTTP Requests\n- Supports three types of HTTP request engines (`jsoup`, `jdk HttpClient`, `Apache HttpClient5`)\n\n| HTTP request engines | http Proxy | Socket5 Proxy | http compress |\n|----------------------|------------|---------------|---------------|\n| Jsoup                | \u0026#x2705;   | \u0026#x2705;      |               |\n| Jdk HttpClient       | \u0026#x2705;   |               | \u0026#x2705;      |\n| Apache HttpClient5   | \u0026#x2705;   |               | \u0026#x2705;      |\n\n\n## Project Structure\n\u003e The jdk version must be 17 or greater\n- `easycrawl-core   `     Crawler core code\n- `easycrawl-helper   `     General tools\n- `easycrawl-schedule   `   Thread Scheduling Related\n- `easycrawl-sql  `       Automated database operation related tools, currently under development\n- `easycrawl-test  `      Integration test related code\n\n\n\n## Configuration Files\n\n\u003e easy-crawl.yml\n\n```yaml\ndefaultConfig:\n  # HTTP request engines: jsoup jdk httpclient5\n  httpEngine: jsoup\n  # http use-agent\n  useAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36\n  # HTTP request timeout (milliseconds)\n  connectTimeout: 10000\n  # Default file download directory\n  fileFolder: C://temp//\n  # Whether to enable http request compression\n  httpCompress: true\n```\n\n### file download\n\n```\n String curl = \"\"\"\n                curl 'https://www.csindex.com.cn/csindex-home/exportExcel/security-industry-search-excel/CH' \\\\\n                  -H 'Accept: application/json, text/plain, */*' \\\\\n                  -H 'Accept-Language: zh-CN,zh;q=0.9' \\\\\n                  -H 'Cache-Control: no-cache' \\\\\n                  -H 'Connection: keep-alive' \\\\\n                  -H 'Content-Type: application/json;charset=UTF-8' \\\\\n                  -H 'DNT: 1' \\\\\n                  -H 'Origin: https://www.csindex.com.cn' \\\\\n                  -H 'Pragma: no-cache' \\\\\n                  -H 'Referer: https://www.csindex.com.cn/en/indices/index-detail/000300' \\\\\n                  -H 'Sec-Fetch-Dest: empty' \\\\\n                  -H 'Sec-Fetch-Mode: cors' \\\\\n                  -H 'Sec-Fetch-Site: same-origin' \\\\\n                  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' \\\\\n                  -H 'sec-ch-ua: \"Google Chrome\";v=\"131\", \"Chromium\";v=\"131\", \"Not_A Brand\";v=\"24\"' \\\\\n                  -H 'sec-ch-ua-mobile: ?0' \\\\\n                  -H 'sec-ch-ua-platform: \"Windows\"' \\\\\n                  --data-raw '{\"searchInput\":\"\",\"pageNum\":1,\"pageSize\":10,\"sortField\":null,\"sortOrder\":null}'\n                \"\"\";\n\n        File file = new ThinEasyCrawl(curl).folder(\"C:\\\\temp\\\\\").execute().getFile();\n        log.info(\"fileName：{} fileSize：{} kb\", file.getName(), file.length() / 1024);\n```\n\n### HTTP and SOCKS5 Proxies\n\n```\n        String apiUrl = \"https://myip.ipip.net/json\";\n        \n        //socks5\n        ProxyConfig proxy = ProxyConfig.of(Proxy.Type.SOCKS, \"127.0.0.1\", 37890);\n        HttpResult result = new ThinEasyCrawl(apiUrl,WebAgent.Engine.JSOUP).proxy(proxy).execute();\n        System.out.println(result.body());\n        \n        //http\n        ProxyConfig proxy = ProxyConfig.of(Proxy.Type.HTTP, \"127.0.0.1\", 27890);\n        HttpResult result = new ThinEasyCrawl(apiUrl).proxy(proxy).execute();\n        System.out.println(result.body());\n```\n\n### get response cookies\n\n```\n        String curl = \"https://example.com\";\n        Map\u003cString, String\u003e cookies = new ThinEasyCrawl(curl).execute().cookies();\n        System.out.println(cookies);\n```\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingschan1204%2FeasyCrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkingschan1204%2FeasyCrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkingschan1204%2FeasyCrawl/lists"}