{"id":14978760,"url":"https://github.com/yoongikim/autocrawler","last_synced_at":"2025-10-06T18:33:07.710Z","repository":{"id":38419468,"uuid":"158573339","full_name":"YoongiKim/AutoCrawler","owner":"YoongiKim","description":"Google, Naver multiprocess image web crawler (Selenium)","archived":false,"fork":false,"pushed_at":"2024-04-15T10:15:21.000Z","size":175730,"stargazers_count":1660,"open_issues_count":2,"forks_count":422,"subscribers_count":44,"default_branch":"master","last_synced_at":"2025-05-15T12:03:40.744Z","etag":null,"topics":["bigdata","chromedriver","crawler","customizable","deep-learning","google","image-crawler","multiprocess","python","selenium","thread"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YoongiKim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-21T16:01:14.000Z","updated_at":"2025-05-13T05:43:42.000Z","dependencies_parsed_at":"2022-09-02T22:23:21.298Z","dependency_job_id":"0ac64603-4665-4540-b2bb-635874031ea9","html_url":"https://github.com/YoongiKim/AutoCrawler","commit_stats":{"total_commits":87,"total_committers":15,"mean_commits":5.8,"dds":0.6551724137931034,"last_synced_commit":"fd9b594e9d8c0441ea4a2af8bbb1a4cb3bb135b1"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoongiKim%2FAutoCrawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoongiKim%2FAutoCrawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoongiKim%2FAutoCrawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoongiKim%2FAutoCrawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YoongiKim","download_url":"https://codeload.github.com/YoongiKim/AutoCrawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254337612,"owners_count":22054253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","chromedriver","crawler","customizable","deep-learning","google","image-crawler","multiprocess","python","selenium","thread"],"created_at":"2024-09-24T13:58:19.893Z","updated_at":"2025-10-06T18:33:07.616Z","avatar_url":"https://github.com/YoongiKim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AutoCrawler\nGoogle, Naver multiprocess image crawler (High Quality \u0026 Speed \u0026 Customizable)\n\n![](docs/animation.gif)\n\n# How to use\n\n1. Install Chrome\n\n2. pip install -r requirements.txt\n\n3. Write search keywords in keywords.txt\n\n4. **Run \"main.py\"**\n\n5. Files will be downloaded to 'download' directory.\n\n\n# Arguments\nusage:\n```\npython3 main.py [--skip true] [--threads 4] [--google true] [--naver true] [--full false] [--face false] [--no_gui auto] [--limit 0]\n```\n\n```\n--skip true        Skips keyword if downloaded directory already exists. This is needed when re-downloading.\n\n--threads 4        Number of threads to download.\n\n--google true      Download from google.com (boolean)\n\n--naver true       Download from naver.com (boolean)\n\n--full false       Download full resolution image instead of thumbnails (slow)\n\n--face false       Face search mode\n\n--no_gui auto      No GUI mode. (headless mode) Acceleration for full_resolution mode, but unstable on thumbnail mode.\n                   Default: \"auto\" - false if full=false, true if full=true\n                   (can be used for docker linux system)\n                   \n--limit 0          Maximum count of images to download per site. (0: infinite)\n--proxy-list ''    The comma separated proxy list like: \"socks://127.0.0.1:1080,http://127.0.0.1:1081\".\n                   Every thread will randomly choose one from the list.\n```\n\n\n# Full Resolution Mode\n\nYou can download full resolution image of JPG, GIF, PNG files by specifying --full true\n\n![](docs/full.gif)\n\n\n\n# Data Imbalance Detection\n\nDetects data imbalance based on number of files.\n\nWhen crawling ends, the message show you what directory has under 50% of average files.\n\nI recommend you to remove those directories and re-download.\n\n\n# Remote crawling through SSH on your server\n\n```\nsudo apt-get install xvfb \u003c- This is virtual display\n\nsudo apt-get install screen \u003c- This will allow you to close SSH terminal while running.\n\nscreen -S s1\n\nXvfb :99 -ac \u0026 DISPLAY=:99 python3 main.py\n```\n\n# Customize\n\nYou can make your own crawler by changing collect_links.py\n\n# How to fix issues\n\nAs google site consistently changes, you may need to fix ```collect_links.py```\n\n1. Go to google image. [https://www.google.com/search?q=dog\u0026source=lnms\u0026tbm=isch](https://www.google.com/search?q=dog\u0026source=lnms\u0026tbm=isch)\n2. Open devloper tools on Chrome. (CTRL+SHIFT+I, CMD+OPTION+I)\n3. Designate an image to capture.\n![CleanShot 2023-10-24 at 17 59 57@2x](https://github.com/YoongiKim/AutoCrawler/assets/38288705/6488d6df-1f01-4dfd-8691-6c0ac142fc04)\n4. Checkout collect_links.py\n![CleanShot 2023-10-24 at 18 02 35@2x](https://github.com/YoongiKim/AutoCrawler/assets/38288705/097c6c03-dd43-45d4-939e-2f677f595362)\n5. Docs for XPATH usage: [https://www.w3schools.com/xml/xpath_syntax.asp](https://www.w3schools.com/xml/xpath_syntax.asp)\n6. You can test XPATH using CTRL+F on your chrome developer tools.\n![CleanShot 2023-10-24 at 18 05 14@2x](https://github.com/YoongiKim/AutoCrawler/assets/38288705/7ce2601f-9d53-48ff-a1cf-1a2befcc510f)\n7. You need to find logic to crawling to work.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoongikim%2Fautocrawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyoongikim%2Fautocrawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyoongikim%2Fautocrawler/lists"}