{"id":21601760,"url":"https://github.com/bwh1270/allrecipes-scraper","last_synced_at":"2025-03-18T13:20:33.437Z","repository":{"id":65399622,"uuid":"591635795","full_name":"bwh1270/allrecipes-scraper","owner":"bwh1270","description":null,"archived":false,"fork":false,"pushed_at":"2023-01-28T09:16:43.000Z","size":11548,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-24T18:30:06.665Z","etag":null,"topics":["crawler","food-computing","scraper","scraping","scrapy"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bwh1270.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2023-01-21T11:13:33.000Z","updated_at":"2023-01-28T07:05:20.000Z","dependencies_parsed_at":"2023-02-12T10:15:31.804Z","dependency_job_id":null,"html_url":"https://github.com/bwh1270/allrecipes-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwh1270%2Fallrecipes-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwh1270%2Fallrecipes-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwh1270%2Fallrecipes-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bwh1270%2Fallrecipes-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bwh1270","download_url":"https://codeload.github.com/bwh1270/allrecipes-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244227587,"owners_count":20419264,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","food-computing","scraper","scraping","scrapy"],"created_at":"2024-11-24T19:10:38.427Z","updated_at":"2025-03-18T13:20:33.412Z","avatar_url":"https://github.com/bwh1270.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# allrecipes-scraper\nThe recipe information will help the food domain, like food computing.\nThis code cand be classified into two modules. The Fisrt module is crawling which is about finding URLs on the web. The second module is scraping which is about extracting the data of recipe from one or more websites. This code is tested in Linux 16.04 LTS, Window10, and Window11. But, it will work in other environments. \n\n\nJust follow the installation part!\n\n\n## Installation\nYou can clone or download this repository in your folder.\n```\n$ git clone https://github.com/bwh1270/allrecipes-scraper.git {your_folder_name}\n```\n\n\nAnd install the requirement packages.\n```\n$ pip install -r requirements.txt\n```\n\n\nExecute the main.py file. It may take long time depending on your internet speed. In my case, it took 4-5 hours.\n\n\n##### Linux\n```\n$ python3 main.py\n```\n※ Be careful not to lose your internet connection.\n\n\n##### Window\n```\n\u003e python main.py\n```\n※ Be careful not to lose your internet connection.\n\n\n\n\n## Modules\nThere are important three modules in src folder.\n+ crawl_category_url.py\n+ crawl_recipe_url.py\n+ scrape.py\n\n\n\nFirst one, the function to crawling the category urls from top category to the lowest categories. \n\n\nIn the image below, Category url means the things inside the blue box. \n\n\n![image](https://user-images.githubusercontent.com/98958137/213866967-a042f967-68ab-4175-a25f-43ffbe1fb064.png)\n\n\nSecond, the function to crawling the recipe urls in each category urls.\n\n\nLast, the function to scrape the recipe information in each recipe urls using the package of \"/hhursev/recipe-scrapers\"\n\n\n\n\n--- \nThis project is licensed under the terms of the MIT license.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbwh1270%2Fallrecipes-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbwh1270%2Fallrecipes-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbwh1270%2Fallrecipes-scraper/lists"}