{"id":20710014,"url":"https://github.com/oxylabs/scrape-images-from-website","last_synced_at":"2025-04-14T04:10:42.238Z","repository":{"id":134336673,"uuid":"526107282","full_name":"oxylabs/scrape-images-from-website","owner":"oxylabs","description":"Scrape Images From a Website with Python ","archived":false,"fork":false,"pushed_at":"2025-02-11T12:51:48.000Z","size":27,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-27T18:03:18.949Z","etag":null,"topics":["github-python","image-scraper","pyhton-scraper","python-image-scraper","scrape-images","scraper-image-from-website","url-scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-08-18T07:39:49.000Z","updated_at":"2025-02-11T12:51:51.000Z","dependencies_parsed_at":"2024-04-19T12:23:53.278Z","dependency_job_id":"189f60c5-e48d-4a62-b5c5-537fbf99dbf8","html_url":"https://github.com/oxylabs/scrape-images-from-website","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-images-from-website","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-images-from-website/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-images-from-website/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fscrape-images-from-website/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/scrape-images-from-website/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248819404,"owners_count":21166477,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["github-python","image-scraper","pyhton-scraper","python-image-scraper","scrape-images","scraper-image-from-website","url-scraper"],"created_at":"2024-11-17T02:09:33.016Z","updated_at":"2025-04-14T04:10:42.207Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scrape Images From a Website with Python\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7\u0026aff_id=877\u0026url_id=112)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n## Project requirements\n\n```bash\npip install beautifulsoup4 selenium pandas requests Pillow\n```\n\n## Back to square one\n\n```python\nimport pandas as pd\nfrom bs4 import BeautifulSoup\nfrom selenium import webdriver\ndriver = webdriver.Chrome(executable_path='/nix/path/to/webdriver/executable')\ndriver.get('https://your.url/here?yes=brilliant')\nresults = []\ncontent = driver.page_source\nsoup = BeautifulSoup(content)\n```\n\nOur data extraction process begins almost exactly the same (we will import libraries as needed). We assign our preferred webdriver, select the URL from which we will scrape image links and create a list to store them in. As our Chrome driver arrives at the URL, we use the variable ‘content’ to point to the page source and then “soupify” it with BeautifulSoup.\n\nIn the previous tutorial, we performed all actions by using built-in and library defined functions. While we could do another tutorial without defining any functions, it is an extremely useful tool for just about any project:\n\n```python\n# Example on how to define a function and select custom arguments for the\n# code that goes into it.\ndef function_name(arguments):\n    # Function body goes here.\n```\n\nWe’ll move our URL scraper into a defined function. Additionally, we will reuse the same code we used in the [“Python Web Scraping Tutorial: Step-by-Step” article](https://oxylabs.io/blog/python-web-scraping) and repurpose it to scrape full URLs.\n\nBefore\n\n```python\nfor a in soup.findAll(attrs={'class': 'class'}):\n    name = a.find('a')\n    if name not in results:\n        results.append(name.text)\n```\n\nAfter \n\n```python\n#picking a name that represents the functions will be useful later on.\ndef parse_image_urls(classes, location, source):\n    for a in soup.findAll(attrs={'class': classes}):\n        name = a.find(location)\n        if name not in results:\n            results.append(name.get(source))\n```\n\nNote that we now append in a different manner.  Instead of appending the text, we use another function `get()` and add a new parameter ‘source’ to it. We use ‘source’ to indicate the field in the website where image links are stored . They will be nested in a ‘src’, ‘data-src’ or other similar HTML tags.\n\n## Moving forward with defined functions\n\nLet’s assume that our target URL has image links nested in the classes ‘blog-card__link’, ‘img’ and that the URL itself is in the ‘src’ attribute of the element. We would call our newly defined function as such:\n\n```python\nparse_image_urls(\"blog-card__link\", \"img\", \"src\")\n```\n\nOur code should now look something like this:\n\n```python\nimport pandas as pd\nfrom bs4 import BeautifulSoup\nfrom selenium import webdriver\n\ndriver = webdriver.Chrome(executable_path='/nix/path/to/webdriver/executable')\ndriver.get('https://your.url/here?yes=brilliant')\nresults = []\ncontent = driver.page_source\nsoup = BeautifulSoup(content)\n\n\ndef parse_image_urls(classes, location, source):\n    for a in soup.findAll(attrs={'class': classes}):\n        name = a.find(location)\n        if name not in results:\n            results.append(name.get(source))\n\nparse_image_urls(\"blog-card__link\", \"img\", \"src\")\n```\n\nSince we sometimes want to export scraped data and we had already used pandas before, we can check by outputting everything into a “.csv” file. If needed, we can always check for any possible semantic errors this way.\n\n```python\ndf = pd.DataFrame(\"links\": results})\ndf.to_csv('links.csv', index=False, encoding='utf-8')\n```\n\nIf we run our code right now, we should get a `links.csv` file outputted right into the running directory.\n\n## Time to extract images from the website\n\nAssuming that we didn’t run into any issues at the end of the previous section, we can continue to download images from websites.\n\n```python\n#import library requests to send HTTP requests\nimport requests\nfor b in results:\n#add the content of the url to a variable\n    image_content = requests.get(b).content\n```\n\nWe will use the requests library to acquire the content stored in the image URL. Our `for` loop above will iterate over our `results` list.\n\n```python\n#io manages file-related in/out operations\nimport io\n#creates a byte object out of image_content and point the variable image_file to it\nimage_file = io.BytesIO(image_content)\n```\n\nWe are not done yet. So far the “image” we have above is just a Python object.\n\n```python\n#we use Pillow to convert our object to an RGB image\nfrom PIL import Image\nimage = Image.open(image_file).convert('RGB')\n```\n\nWe are still not done as we need to find a place to save our images. Creating a folder “Test” for the purposes of this tutorial would be the easiest option.\n\n```python\n#pathlib let's us point to specific locations. Will be used to save our images.\nimport pathlib\n#hashlib allows us to get hashes. We will be using sha1 to name our images.\nimport hashlib\n#sets a file_path variable which is pointed to \n#our directory and creates a file based on #the sha1 hash of 'image_content' \n#and uses .hexdigest to convert it into a string.\nfile_path = pathlib.Path('nix/path/to/test', hashlib.sha1(image_content).hexdigest()[:10] + '.png')\nimage.save(file_path, \"PNG\", quality=80)\n```\n\n## Putting it all together\n\nLet’s combine all of the previous steps without any comments and see how it works out. Note that pandas are greyed out as we are not extracting data into any tables. We kept it in for the sake of convenience. Use it if you need to see or double-check the outputs.\n\n```python\nimport hashlib\nimport io\nfrom pathlib import Path\nimport pandas as pd\nimport requests\nfrom bs4 import BeautifulSoup\nfrom PIL import Image\nfrom selenium import webdriver\n\ndriver = webdriver.Chrome(executable_path='/nix/path/to/webdriver/executable')\ndriver.get('https://your.url/here?yes=brilliant')\ndriver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\nresults = []\ncontent = driver.page_source\nsoup = BeautifulSoup(content)\n\n\ndef gets_url(classes, location, source):\n   results = []\n   for a in soup.findAll(attrs={'class': classes}):\n       name = a.find(location)\n       if name not in results:\n           results.append(name.get(source))\n   return results\n\n\ndriver.quit()\n\nif __name__ == \"__main__\":\n   returned_results = gets_url(\"blog-card__link\", \"img\", \"src\")\n   for b in returned_results::\n    image_content = requests.get(b).content\n    image_file = io.BytesIO(image_content)\n    image = Image.open(image_file).convert('RGB')\n    file_path = pathlib.Path('nix/path/to/test', hashlib.sha1(image_content).hexdigest()[:10] + '.png')\n    image.save(file_path, \"PNG\", quality=80)\n```\n\nFor efficiency, we quit our webdriver by using “driver.quit()” after retrieving the URL list we need. We no longer need that browser as everything is stored locally.\n\nRunning our application will output one of two results:\n\n1.Images are outputted into the folder we selected by defining the ‘file_path’ variable.\n\n2.Python outputs a `403` Forbidden HTTP error.\n\nObviously, getting the first result means we are finished. We would receive the second outcome if we were to scrape our /blog/ page. Fixing the second outcome will take a little bit of time in most cases, although, at times, there can be more difficult scenarios.\n\nWhenever we use the requests library to send a request to the destination server, a default user-agent “Python-urllib/version.number” is assigned. Some web services might block these user-agents specifically as they are guaranteed to be bots. Fortunately, the requests library allows us to assign any user-agent (or an entire header) we want:\n\n```python\nimage_content = requests.get(b, headers={'User-agent': 'Mozilla/5.0'}).content\n```\n\n## Cleaning up\n\nOur task is finished but the code is still messy.  We can make our application more readable and reusable by putting everything under defined functions:\n\n```python\nimport io\nimport pathlib\nimport hashlib\nimport pandas as pd\nimport requests\nfrom bs4 import BeautifulSoup\nfrom PIL import Image\nfrom selenium import webdriver\n\n\ndef get_content_from_url(url):\n   driver = webdriver.Chrome()  # add \"executable_path=\" if driver not in running directory\n   driver.get(url)\n   driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n   page_content = driver.page_source\n   driver.quit()  # We do not need the browser instance for further steps.\n   return page_content\n\n\ndef parse_image_urls(content, classes, location, source):\n   soup = BeautifulSoup(content)\n   results = []\n   for a in soup.findAll(attrs={\"class\": classes}):\n       name = a.find(location)\n       if name not in results:\n           results.append(name.get(source))\n   return results\n\n\ndef save_urls_to_csv(image_urls):\n   df = pd.DataFrame({\"links\": image_urls})\n   df.to_csv(\"links.csv\", index=False, encoding=\"utf-8\")\n\n\ndef get_and_save_image_to_file(image_url, output_dir):\n   response = requests.get(image_url, headers={\"User-agent\": \"Mozilla/5.0\"})\n   image_content = response.content\n   image_file = io.BytesIO(image_content)\n   image = Image.open(image_file).convert(\"RGB\")\n   filename = hashlib.sha1(image_content).hexdigest()[:10] + \".png\"\n   file_path = output_dir / filename\n   image.save(file_path, \"PNG\", quality=80)\n\n\ndef main():\n   url = \"https://your.url/here?yes=brilliant\"\n   content = get_content_from_url(url)\n   image_urls = parse_image_urls(\n       content=content, classes=\"blog-card__link\", location=\"img\", source=\"src\",\n   )\n   save_urls_to_csv(image_urls)\n\n   for image_url in image_urls:\n       get_and_save_image_to_file(\n           image_url, output_dir=pathlib.Path(\"nix/path/to/test\"),\n       )\n\n\nif __name__ == \"__main__\":  #only executes if imported as main file\n   main()\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscrape-images-from-website","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fscrape-images-from-website","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fscrape-images-from-website/lists"}