Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hhursev/recipe-scrapers

Python package for scraping recipes data
https://github.com/hhursev/recipe-scrapers

Last synced: 3 days ago
JSON representation

Python package for scraping recipes data

Awesome Lists containing this project

README

        

.. image:: https://img.shields.io/github/stars/hhursev/recipe-scrapers?style=social
:target: https://github.com/hhursev/recipe-scrapers/
:alt: Github
.. image:: https://img.shields.io/pypi/v/recipe-scrapers.svg?
:target: https://pypi.org/project/recipe-scrapers/
:alt: Version
.. image:: https://img.shields.io/pypi/pyversions/recipe-scrapers
:target: https://pypi.org/project/recipe-scrapers/
:alt: PyPI - Python Version
.. image:: https://pepy.tech/badge/recipe-scrapers
:target: https://pepy.tech/project/recipe-scrapers
:alt: Downloads
.. image:: https://github.com/hhursev/recipe-scrapers/workflows/unittests/badge.svg?branch=main
:target: https://github.com/hhursev/recipe-scrapers/actions/
:alt: GitHub Actions Unittests
.. image:: https://coveralls.io/repos/hhursev/recipe-scraper/badge.svg?branch=main&service=github
:target: https://coveralls.io/github/hhursev/recipe-scraper?branch=main
:alt: Coveralls
.. image:: https://img.shields.io/github/license/hhursev/recipe-scrapers?
:target: https://github.com/hhursev/recipe-scrapers/blob/main/LICENSE
:alt: License
.. image:: https://app.codacy.com/project/badge/Grade/3ee8da77aaa3475a8085ca22287dea89
:target: https://app.codacy.com/gh/hhursev/recipe-scrapers/dashboard
:alt: Codacy Badge

------

A simple scraping tool for recipe webpages.

Netiquette
----------

If you're using this library to collect large numbers of recipes from the web, please use the software responsibly and try to avoid creating high volumes of network traffic.

Python's standard library provides a ``robots.txt`` `parser `_ that may be helpful to automatically follow common instructions specified by websites for web crawlers.

Getting Started
---------------

Start by using `Python's built-in package installer `_, ``pip``, to install the library:

.. code:: shell

python -m pip install recipe-scrapers

This should produce output about the installation process, with the final line reading: ``Successfully installed recipe-scrapers-``.

To learn what the library can do, you can open a `Python interpreter session `_, and then begin typing -- and/or modifying -- the statements below:

.. code:: python

from urllib.request import urlopen

from recipe_scrapers import scrape_html

url = "https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/"
html = urlopen(url).read().decode("utf-8") # retrieves the recipe webpage HTML
scraper = scrape_html(html, org_url=url)
scraper.title()
scraper.instructions() # etc.
# for a complete list of methods:
# help(scraper)

Some Python HTTP clients that you can use to retrieve HTML include `requests`_, `httpx`_, and the `urllib.request module`_ included in Python's standard library. Please refer to their documentation to find out what options (timeout configuration, proxy support, etc) are available.

.. _requests: https://pypi.org/project/requests/

.. _httpx: https://pypi.org/project/httpx/

.. _urllib.request module: https://docs.python.org/3/library/urllib.request.html

Scrapers available for:
-----------------------

- `https://101cookbooks.com/ `_
- `https://15gram.be `_
- `https://40aprons.com/ `_
- `https://www.750g.com `_
- `https://abeautifulmess.com/ `_
- `https://aberlehome.com/ `_
- `https://abuelascounter.com/ `_
- `https://www.acouplecooks.com `_
- `https://addapinch.com/ `_
- `http://www.afghankitchenrecipes.com/ `_
- `https://aflavorjournal.com/ `_
- `https://ah.nl/ `_
- `https://www.ahealthysliceoflife.com/ `_
- `https://akispetretzikis.com/ `_
- `https://aldi-nord.de/ `_
- `.es `__, `.fr `__, `.lu `__, `.nl `__, `.pl `__, `.pt `__
- `https://aldi-sued.de/ `_
- `.hu `__, `.it `__
- `https://aldi-suisse.ch `_
- `https://aldi.com.au/ `_
- `https://alexandracooks.com/ `_
- `https://alittlebityummy.com/ `_
- `https://allrecipes.com/ `_
- `https://allthehealthythings.com/ `_
- `https://alltommat.se/ `_
- `https://altonbrown.com/ `_
- `https://amazingribs.com/ `_
- `https://ambitiouskitchen.com/ `_
- `https://ameessavorydish.com/ `_
- `https://americastestkitchen.com/ `_ (*)
- `https://archanaskitchen.com/ `_
- `https://www.argiro.gr/ `_
- `https://www.arla.se/ `_
- `https://www.atelierdeschefs.fr/ `_
- `https://averiecooks.com/ `_
- `https://www.bakels.com.au/ `_
- `.co.uk `_
- `https://baking-sense.com/ `_
- `https://bakingmischief.com/ `_
- `https://barefeetinthekitchen.com/ `_
- `https://barefootcontessa.com/ `_
- `https://barefootinthepines.com/ `_
- `https://bbc.com/ `_
- `.co.uk `__
- `https://bbcgoodfood.com/ `_
- `https://bestrecipes.com.au/ `_
- `https://betterfoodguru.com/ `_
- `https://bettybossi.ch/ `_
- `https://bettycrocker.com/ `_
- `https://beyondfrosting.com/ `_
- `https://biancazapatka.com/ `_
- `https://bigoven.com/ `_
- `https://bitsofcarey.com/ `_
- `https://blueapron.com/ `_
- `https://bluejeanchef.com/ `_
- `https://www.bodybuilding.com/ `_
- `https://bonappetit.com/ `_
- `https://bongeats.com/ `_
- `https://books.ottolenghi.co.uk `_ (*)
- `https://bowlofdelicious.com/ `_
- `https://breadtopia.com/ `_
- `https://briceletbaklava.ch/ `_
- `https://brokenovenbaking.com/ `_
- `https://budgetbytes.com/ `_
- `https://cafedelites.com/ `_
- `https://cakemehometonight.com/ `_
- `https://cambreabakes.com/ `_
- `https://carlsbadcravings.com/ `_
- `https://castironketo.net/ `_
- `https://cdkitchen.com/ `_
- `https://celebratingsweets.com/ `_
- `https://chefjeanpierre.com/ `_
- `https://chefkoch.de/ `_
- `https://www.chefnini.com/ `_
- `https://chefsavvy.com/ `_
- `https://claudia.abril.com.br/ `_
- `https://closetcooking.com/ `_
- `https://colleenchristensennutrition.com/ `_
- `https://comidinhasdochef.com/ `_
- `https://cook-talk.com/ `_
- `https://cookeatshare.com/ `_
- `https://cookieandkate.com/ `_
- `https://cookiesandcups.com/ `_
- `https://cooking.nytimes.com/ `_
- `https://cookingcircle.com/ `_
- `https://cookinglight.com/ `_
- `https://cookomix.com/ `_
- `https://cookpad.com/ `_
- `https://cookscountry.com/ `_ (*)
- `https://cooksillustrated.com/ `_ (*)
- `https://cookwell.com/ `_
- `https://copykat.com/ `_
- `https://www.costco.com/ `_
- `https://countryliving.com/ `_
- `https://creativecanning.com/ `_
- `https://cucchiaio.it/ `_
- `https://cuisineaz.com/ `_
- `https://cybercook.com.br/ `_
- `https://damndelicious.net/ `_
- `https://www.davidlebovitz.com/ `_
- `https://delish.com/ `_
- `https://dinneratthezoo.com/ `_
- `https://dinnerthendessert.com/ `_
- `https://dish.co.nz/ `_
- `https://dobruchut.aktuality.sk/ `_
- `https://domesticate-me.com/ `_
- `https://donalskehan.com/ `_
- `https://downshiftology.com/ `_
- `https://www.dr.dk/ `_
- `https://www.eatingbirdfood.com/ `_
- `https://www.eatingwell.com/ `_
- `https://www.eatliverun.com/ `_
- `https://eatsmarter.com/ `_
- `.de `__
- `https://eatthismuch.com/ `_
- `https://eattolerant.de/ `_
- `https://www.eatwell101.com `_
- `https://eatwhattonight.com/ `_
- `https://eggs.ca/ `_
- `https://elavegan.com/ `_
- `https://emmikochteinfach.de/ `_
- `https://en.wikibooks.org/ `_
- `https://epicurious.com/ `_
- `https://www.errenskitchen.com/ `_
- `https://ethanchlebowski.com/ `_
- `https://www.evolvingtable.com/ `_
- `https://www.familyfoodonthetable.com/ `_
- `https://www.farmhouseonboone.com/ `_
- `https://www.fattoincasadabenedetta.it/ `_
- `https://felix.kitchen `_
- `https://fifteenspatulas.com/ `_
- `https://finedininglovers.com/ `_
- `https://fitmencook.com/ `_
- `https://fitslowcookerqueen.com `_
- `https://food.com/ `_
- `https://food52.com/ `_
- `https://foodandwine.com/ `_
- `https://foodfidelity.com/ `_
- `https://foodnetwork.co.uk/ `_
- `.com `__
- `https://foodrepublic.com/ `_
- `https://www.forksoverknives.com/ `_
- `https://forktospoon.com/ `_
- `https://franzoesischkochen.de/ `_
- `https://www.gesund-aktiv.com/ `_
- `https://gimmesomeoven.com/ `_
- `https://glutenfreeonashoestring.com/ `_
- `https://godt.no/ `_
- `https://gonnawantseconds.com/ `_
- `https://goodfooddiscoveries.com/ `_
- `https://goodhousekeeping.com/ `_
- `https://gourmettraveller.com.au/ `_
- `https://www.grandfrais.com/ `_
- `https://greatbritishchefs.com/ `_
- `https://grimgrains.com/ `_
- `http://www.grouprecipes.com/ `_
- `https://halfbakedharvest.com/ `_
- `https://handletheheat.com/ `_
- `https://www.hassanchef.com/ `_
- `https://headbangerskitchen.com/ `_
- `https://healthyeating.nhlbi.nih.gov/ `_
- `https://heatherchristo.com/ `_
- `https://www.heb.com/ `_
- `https://hellofresh.com/ `_
- `.at `__, `.be `__, `.ca `__, `.ch `__, `.co.nz `__, `.co.uk `__, `.com.au `__, `.de `__, `.dk `__, `.es `__, `.fr `__, `.ie `__, `.it `__, `.lu `__, `.nl `__, `.no `__, `.se `__
- `https://www.hersheyland.com/ `_
- `https://hofer.at/ `_
- `.si `__
- `https://www.homechef.com/ `_
- `https://hostthetoast.com/ `_
- `https://hungryhappens.net/ `_
- `https://www.ica.se/ `_
- `https://www.im-worthy.com/ `_
- `https://inbloombakery.com/ `_
- `https://indianhealthyrecipes.com `_
- `https://ingoodflavor.com `_
- `https://www.innit.com/ `_
- `https://insanelygoodrecipes.com `_
- `https://inspiralized.com/ `_
- `https://inspiredtaste.net/ `_
- `https://irishcentral.com/ `_
- `https://izzycooking.com/ `_
- `https://jamieoliver.com/ `_
- `https://jimcooksfoodgood.com/ `_
- `https://www.jocooks.com/ `_
- `https://joshuaweissman.com/ `_
- `https://joyfoodsunshine.com/ `_
- `https://joythebaker.com/ `_
- `https://juliegoodwin.com.au/ `_
- `https://justataste.com/ `_
- `https://justbento.com/ `_
- `https://www.justonecookbook.com/ `_
- `https://kalejunkie.com/ `_
- `https://kennymcgovern.com/ `_
- `https://keukenliefde.nl/ `_
- `https://www.kingarthurbaking.com `_
- `https://kitchenaid.com.au/ `_
- `https://kitchendivas.com `_
- `https://www.kitchendreaming.com `_
- `https://www.kitchensanctuary.com/ `_
- `https://www.kitchenstories.com/ `_
- `https://kochbar.de/ `_
- `https://kochbucher.com/ `_
- `http://koket.se/ `_
- `https://kristineskitchenblog.com/ `_
- `https://krollskorner.com/ `_
- `https://kuchnia-domowa.pl/ `_
- `https://kuchynalidla.sk/ `_
- `https://www.kwestiasmaku.com/ `_
- `https://lanascooking.com/ `_
- `https://www.latelierderoxane.com `_
- `https://leanandgreenrecipes.net `_
- `https://www.lecker.de `_
- `https://lecremedelacrumb.com/ `_
- `https://leitesculinaria.com `_
- `https://lekkerensimpel.com `_
- `https://leukerecepten.nl/ `_
- `https://lifestyleofafoodie.com `_
- `https://littlespicejar.com/ `_
- `https://littlesunnykitchen.com/ `_
- `http://livelytable.com/ `_
- `https://lovingitvegan.com/ `_
- `https://www.maangchi.com `_
- `https://www.madamecuisine.de `_
- `https://madensverden.dk/ `_
- `https://madsvin.com/ `_
- `https://marmiton.org/ `_
- `https://www.marthastewart.com/ `_
- `https://matprat.no/ `_
- `https://www.mccormick.com/ `_
- `https://meganvskitchen.com/ `_
- `https://meljoulwan.com/ `_
- `https://www.melskitchencafe.com/ `_
- `https://www.miljuschka.nl/ `_
- `http://mindmegette.hu/ `_
- `https://minimalistbaker.com/ `_
- `https://ministryofcurry.com/ `_
- `https://misya.info/ `_
- `https://www.mob.co.uk/ `_
- `https://mobkitchen.co.uk/ `_
- `https://www.modernhoney.com/ `_
- `https://www.momontimeout.com/ `_
- `https://momswithcrockpots.com/ `_
- `http://motherthyme.com/ `_
- `https://www.moulinex.fr/ `_
- `https://www.mundodereceitasbimby.com.pt/ `_
- `https://mybakingaddiction.com/ `_
- `https://myjewishlearning.com/ `_
- `https://mykitchen101.com/ `_
- `https://mykitchen101en.com/ `_
- `https://mykoreankitchen.com/ `_
- `https://www.myplate.gov/ `_
- `https://myrecipes.com/ `_
- `https://myvegetarianroots.com/ `_
- `https://www.nhs.uk/healthier-families/ `_
- `https://nibbledish.com/ `_
- `https://norecipes.com/ `_
- `https://nosalty.hu/ `_
- `https://www.notenoughcinnamon.com/ `_
- `https://nourishedbynutrition.com/ `_
- `https://www.nrk.no/ `_
- `https://www.number-2-pencil.com/ `_
- `https://nutritionbynathalie.com/blog `_
- `https://nutritionfacts.org/ `_
- `https://ohsheglows.com/ `_
- `https://omnivorescookbook.com `_
- `https://www.onceuponachef.com `_
- `https://onesweetappetite.com/ `_
- `https://owen-han.com/ `_
- `https://www.paleorunningmomma.com/ `_
- `https://www.panelinha.com.br/ `_
- `https://paninihappy.com/ `_
- `https://www.peelwithzeal.com/ `_
- `https://www.persnicketyplates.com/ `_
- `https://www.pickuplimes.com/ `_
- `https://pinchofyum.com/ `_
- `https://www.pingodoce.pt/ `_
- `https://pinkowlkitchen.com/ `_
- `https://www.platingpixels.com/ `_
- `https://plowingthroughlife.com/ `_
- `https://popsugar.com/ `_
- `https://potatorolls.com/ `_
- `https://practicalselfreliance.com/ `_
- `https://pressureluckcooking.com/ `_
- `https://www.primaledgehealth.com/ `_
- `https://www.projectgezond.nl/ `_
- `https://przepisy.pl/ `_
- `https://purelypope.com/ `_
- `https://purplecarrot.com/ `_
- `https://quitoque.fr/ `_
- `https://rachlmansfield.com/ `_
- `https://rainbowplantlife.com/ `_
- `https://realfood.tesco.com/ `_
- `https://realsimple.com/ `_
- `https://receitas.globo.com/ `_
- `https://receitas.ig.com.br/ `_
- `https://www.receitasnestle.com.br `_
- `https://recept.se/ `_
- `https://receptyprevas.sk/ `_
- `https://recette.plus/ `_
- `https://www.recipegirl.com/ `_
- `https://recipeland.com/ `_
- `https://reciperunner.com/ `_
- `https://recipes.farmhousedelivery.com/ `_
- `https://recipes.timesofindia.com/ `_
- `https://recipetineats.com/ `_
- `https://redhousespice.com/ `_
- `https://reishunger.de/ `_
- `https://rewe.de/ `_
- `https://rezeptwelt.de/ `_
- `https://ricetta.it/ `_
- `https://ricette.giallozafferano.it/ `_
- `https://www.ricetteperbimby.it/ `_
- `https://rosannapansino.com `_
- `https://rutgerbakt.nl/ `_
- `https://www.saboresajinomoto.com.br/ `_
- `https://sallys-blog.de `_
- `https://sallysbakingaddiction.com `_
- `https://saltpepperskillet.com/ `_
- `https://sandwichtribunal.com/ `_
- `https://www.saveur.com/ `_
- `https://www.savorynothings.com/ `_
- `https://schoolofwok.co.uk/ `_
- `https://seriouseats.com/ `_
- `https://shelikesfood.com/ `_
- `https://simple-veganista.com/ `_
- `https://simply-cookit.com/ `_
- `https://simplyquinoa.com/ `_
- `https://simplyrecipes.com/ `_
- `https://simplywhisked.com/ `_
- `https://skinnytaste.com/ `_
- `https://smulweb.nl/ `_
- `https://sobors.hu/ `_
- `https://www.southerncastiron.com/ `_
- `https://southernliving.com/ `_
- `https://spainonafork.com/ `_
- `https://spendwithpennies.com/ `_
- `https://www.springlane.de `_
- `https://www.staysnatched.com/ `_
- `https://steamykitchen.com/ `_
- `https://streetkitchen.hu/ `_
- `https://www.strongrfastr.com `_
- `https://sunbasket.com/ `_
- `https://sundpaabudget.dk/ `_
- `https://www.sunset.com/ `_
- `https://sweetcsdesigns.com/ `_
- `https://sweetpeasandsaffron.com/ `_
- `https://www.taste.com.au/ `_
- `https://www.tasteatlas.com/ `_
- `https://tasteofhome.com `_
- `https://tastesbetterfromscratch.com `_
- `https://tastesoflizzyt.com `_
- `https://tasty.co `_
- `https://tastykitchen.com/ `_
- `https://theclevercarrot.com/ `_
- `https://www.thecookierookie.com/ `_
- `https://thecookingguy.com/ `_
- `https://thefoodietakesflight.com/ `_
- `https://theglutenfreeaustrian.com/ `_
- `https://thehappyfoodie.co.uk/ `_
- `https://thekitchencommunity.org/ `_
- `https://www.thekitchenmagpie.com/ `_
- `https://thekitchn.com/ `_
- `https://theloopywhisk.com/ `_
- `https://www.themagicalslowcooker.com/ `_
- `https://themediterraneandish.com/ `_
- `https://themodernproper.com/ `_
- `https://www.thepalatablelife.com `_
- `https://thepioneerwoman.com/ `_
- `https://therecipecritic.com/ `_
- `https://thesaltymarshmallow.com/ `_
- `https://thespruceeats.com/ `_
- `https://thevintagemixer.com/ `_
- `https://thewoksoflife.com/ `_
- `https://thinlicious.com/ `_
- `https://tidymom.net `_
- `https://tine.no/ `_
- `https://tofoo.co.uk `_
- `https://tudogostoso.com.br/ `_
- `https://twopeasandtheirpod.com/ `_
- `https://uitpaulineskeuken.nl/ `_
- `https://unsophisticook.com/ `_
- `https://usapears.org/ `_
- `https://www.valdemarsro.dk/ `_
- `https://vanillaandbean.com/ `_
- `https://varecha.pravda.sk/ `_
- `https://www.vegetarbloggen.no/ `_
- `https://vegolosi.it/ `_
- `https://vegrecipesofindia.com/ `_
- `https://veroniquecloutier.com `_
- `https://www.waitrose.com/ `_
- `https://watchwhatueat.com/ `_
- `https://wearenotmartha.com/ `_
- `https://wedishitup.com/ `_
- `https://www.weightwatchers.com/ `_ (*)
- `https://www.wellplated.com/ `_
- `https://whatsgabycooking.com/ `_
- `https://whole30.com/ `_
- `https://www.wholefoodsmarket.com/ `_
- `.co.uk `__
- `https://www.williams-sonoma.com/ `_
- `https://womensweeklyfood.com.au/ `_
- `https://woop.co.nz/ `_
- `https://yemek.com/ `_
- `https://yummly.com/ `_ (*)
- `https://www.zaubertopf.de `_
- `https://zeit.de/ (wochenmarkt) `_
- `https://zenbelly.com/ `_

(*) offline saved files only

Contribute
----------

If you spot a design change (or something else) that makes the scraper unable to work for a given site - please fire an issue asap.

If you are programmer PRs with fixes are warmly welcomed and acknowledged with a virtual beer. You can find documentation on how to develop scrapers `here `__.

If you want a scraper for a new site added
------------------------------------------

- Open an `Issue `_ providing us the site name, as well as a recipe link from it.
- You are a developer and want to code the scraper on your own:

- If `Schema is available <#faq>`_ on the site - `you can go like this. `_
- Otherwise, scrape the HTML - `like this `_
- Generating a new scraper class:

.. code:: shell

python generate.py

- **ClassName**: The name of the new scraper class.
- **URL**: The URL of an example recipe from the target site. The content will be stored in ``test_data`` to be used with the test class.

You can find a more detailed guide `here `__.

For Devs / Contribute
---------------------

Assuming you have ``>=python3.9`` installed, navigate to the directory where you want this project to live in and drop these lines

.. code:: shell

git clone [email protected]:hhursev/recipe-scrapers.git &&
cd recipe-scrapers &&
python -m venv .venv &&
source .venv/bin/activate &&
python -m pip install --upgrade pip &&
pip install -e ".[dev]" &&
pip install pre-commit &&
pre-commit install &&
python -m unittest

In case you want to run a single unittest for a newly developed scraper

.. code:: shell

python -m unittest -k

FAQ
---
**What if the recipe site I want to extract information from is not listed above?**

You can give it a try with the ``wild_mode`` option!

If there is Schema/Recipe available it will work just fine.

.. code:: python

url = 'https://www.feastingathome.com/tomato-risotto/'
name = input('What is your name, risotto sampler?\n')
html = requests.get(url, headers={"User-Agent": f"Risotto Sampler {name}"}).content
scraper = scrape_html(html, org_url=url, wild_mode=True)

scraper.host()
scraper.title()
scraper.total_time()
scraper.image()
scraper.ingredients()
scraper.ingredient_groups()
scraper.instructions()
scraper.instructions_list()
scraper.yields()
scraper.to_json()
scraper.links()
scraper.nutrients() # not always available
scraper.canonical_url() # not always available
scraper.equipment() # not always available
scraper.cooking_method() # not always available
scraper.keywords() # not always available
scraper.dietary_restrictions() # not always available

Notes:

- ``scraper.links()`` returns a list of dictionaries containing all of the tag attributes. The attribute names are the dictionary keys.

**How do I know if a website has a Recipe Schema?**

Run in python shell:

.. code:: python

# first ensure you have the required packages:
# pip install "recipe-scrapers[online]"

from recipe_scrapers import scrape_html
scraper = scrape_html(html=None, org_url='', online=True, wild_mode=True)
# if no error is raised - there's schema available:
scraper.title()
scraper.instructions() # etc.

Special thanks to:
------------------

All the `contributors that helped improving `_ the package. You are awesome!

.. image:: https://contrib.rocks/image?repo=hhursev/recipe-scrapers
:target: https://github.com/hhursev/recipe-scrapers/graphs/contributors

Test Data Notice
----------------

All content in ``tests/test_data/`` is used for limited, non-commercial testing purposes and belongs to their respective copyright holders. See the ``tests/test_data/LICENSE.md`` for details. If you're a copyright holder with concerns, you can open an issue or contact us privately via the email in our PyPI page.

Extra:
------
| You want to gather recipes data?
| You have an idea you want to implement?
| Check out `our "Share a project" wall `_ - it may save you time and spark ideas!