{"id":21037219,"url":"https://github.com/kadnan/scrapegen","last_synced_at":"2025-05-15T14:32:26.377Z","repository":{"id":38257130,"uuid":"220537719","full_name":"kadnan/ScrapeGen","owner":"kadnan","description":"A simple python tool that generates a requests/bs4 based web scraper","archived":false,"fork":false,"pushed_at":"2022-06-08T07:28:14.000Z","size":6,"stargazers_count":26,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-03T10:51:20.414Z","etag":null,"topics":["beautiful","bs4","python","requests","scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kadnan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-08T19:52:26.000Z","updated_at":"2025-01-10T17:14:09.000Z","dependencies_parsed_at":"2022-08-31T12:22:00.284Z","dependency_job_id":null,"html_url":"https://github.com/kadnan/ScrapeGen","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kadnan%2FScrapeGen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kadnan%2FScrapeGen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kadnan%2FScrapeGen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kadnan%2FScrapeGen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kadnan","download_url":"https://codeload.github.com/kadnan/ScrapeGen/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254358935,"owners_count":22058017,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautiful","bs4","python","requests","scraper"],"created_at":"2024-11-19T13:24:57.492Z","updated_at":"2025-05-15T14:32:26.030Z","avatar_url":"https://github.com/kadnan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ScrapeGen\n\n_ScrapeGen_ is a simple tool written in python that generates the code of a web scraper based on rules given in a file.\n\n\n## Why was it created?\n \nNo particular reasons other than I was bored and I had come across Simone Giertz's TED talk [Why you should make useless things](https://www.ted.com/talks/simone_giertz_why_you_should_make_useless_things?language=en), hence thought to create something useless.\n\n## How it works?\nThis tool generate a parser that basically rely on Python `requests` and `Beautifulsoup`. If I got bored again then I might add other libraries too, who knows?\n\nAnyways, you will create a YAML file first that will contain all info about the parser. A typical YAML file that generates a parser will look like below:\n\n```\nscript_name: olx_indi_test.py # Name of the scraper file\nmain: # Code under _main_ function\n  entry_url: https://www.olx.com.pk/item/1-kanal-brand-bew-banglow-available-for-sale-in-wapda-town-iid-1009971253 # URL to be parsed\n  entry_function: parse # The function that uses requests library to fetch the data and calling Bs4\n\nrules: # Each Selector will be a separate rule that itself will be a separate method\n    - name: price\n      type: single #Valid types: array,single\n      selector: '#container \u003e main \u003e div \u003e div \u003e div.rui-2SwH7.rui-m4D6f.rui-1nZcN.rui-3CPXI.rui-3E1c2.rui-1JF_2 \u003e div.rui-2ns2W._2r-Wm \u003e div \u003e section \u003e span._2xKfz'\n      extract: #Either an attribute value or just text\n        what: text\n\n    - name: seller\n      type: single #Valid types: array,single\n      selector: '#container \u003e main \u003e div \u003e div \u003e div.rui-2SwH7.rui-m4D6f.rui-1nZcN.rui-3CPXI.rui-3E1c2.rui-1JF_2 \u003e div.rui-2ns2W.YpyR- \u003e div \u003e div \u003e div._1oSdP \u003e div \u003e a \u003e div'\n      extract:\n        what: text\n\n\n```\n\nAssuming you installed all required libs mentioned in `requirements.txt`, all you have to do is to run the command:\n \n `python parse_gen.py indi.yaml` \n \n Where `indi.yaml` is the file that contains the content given above. If it runs successfully, it generates a file with name `olx_indi_test.py` which looks like below:\n \n ```\n import requests\nfrom bs4 import BeautifulSoup\n\n\ndef get_price(soup_object):\n    _price = None\n    price_section = soup_object.select(\n        \"#container \u003e main \u003e div \u003e div \u003e div.rui-2SwH7.rui-m4D6f.rui-1nZcN.rui-3CPXI.rui-3E1c2.rui-1JF_2 \u003e div.rui-2ns2W._2r-Wm \u003e div \u003e section \u003e span._2xKfz\")\n\n    if len(price_section) \u003e 0:\n        _price = price_section[0].text.strip()\n    return _price\n\n\ndef get_seller(soup_object):\n    _seller = None\n    seller_section = soup_object.select(\n        \"#container \u003e main \u003e div \u003e div \u003e div.rui-2SwH7.rui-m4D6f.rui-1nZcN.rui-3CPXI.rui-3E1c2.rui-1JF_2 \u003e div.rui-2ns2W.YpyR- \u003e div \u003e div \u003e div._1oSdP \u003e div \u003e a \u003e div\")\n\n    if len(seller_section) \u003e 0:\n        _seller = seller_section[0].text.strip()\n    return _seller\n\n\ndef parse(_url):\n\n    r = requests.get(_url)\n    if r.status_code == 200:\n        html = r.text.strip()\n        soup = BeautifulSoup(html, 'lxml')\n        price = get_price(soup)\n        seller = get_seller(soup)\n\n\nif __name__ == '__main__':\n    main_url = \"https://www.olx.com.pk/item/1-kanal-brand-bew-banglow-available-for-sale-in-wapda-town-iid-1009971253\"\n    parse(main_url)\n ```\n \n The generated can easily be modified based on your needs.  ","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkadnan%2Fscrapegen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkadnan%2Fscrapegen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkadnan%2Fscrapegen/lists"}