{"id":20710116,"url":"https://github.com/oxylabs/how-to-build-web-scraper","last_synced_at":"2026-05-01T20:31:38.606Z","repository":{"id":134336622,"uuid":"526095604","full_name":"oxylabs/how-to-build-web-scraper","owner":"oxylabs","description":"Step by step guide to building a web scraper with Python","archived":false,"fork":false,"pushed_at":"2025-09-25T07:52:58.000Z","size":80,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-25T09:28:27.510Z","etag":null,"topics":["build-a-web-scraper","github-python","parser","python","python-web-scraper","python-web-scraping","python3","url-scraper","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-08-18T07:03:09.000Z","updated_at":"2025-09-25T07:53:01.000Z","dependencies_parsed_at":null,"dependency_job_id":"b22ee4f5-9abf-4d8f-8038-b4b2b210617a","html_url":"https://github.com/oxylabs/how-to-build-web-scraper","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/how-to-build-web-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-build-web-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-build-web-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-build-web-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-build-web-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/how-to-build-web-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fhow-to-build-web-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32512662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["build-a-web-scraper","github-python","parser","python","python-web-scraper","python-web-scraping","python3","url-scraper","web-scraping"],"created_at":"2024-11-17T02:09:54.618Z","updated_at":"2026-05-01T20:31:38.593Z","avatar_url":"https://github.com/oxylabs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Build a Web Scraper?\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=how-to-build-web-scraper-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n[![](https://dcbadge.limes.pink/api/server/Pds3gBmKMH?style=for-the-badge\u0026theme=discord)](https://discord.gg/Pds3gBmKMH) [![YouTube](https://img.shields.io/badge/YouTube-Oxylabs-red?style=for-the-badge\u0026logo=youtube\u0026logoColor=white)](https://www.youtube.com/@oxylabs)\n\nStep by step guide to building a web scraper with Python and JavaScript\n\n## Building a Web Scraper with Python\n\n### STEP 1. How to get the HTML?\nThe first step to building a web scraper is getting the HTML of a page. We will be using the ```requests``` library to get the HTML. It allows us to send a request and get a response. This can be installed using pip or pip3, depending on your Python installation.\n\n```bash\npip install requests\n```\n\nNow create a new file with extension ```.py``` in your favorite editor and open it. Alternatively, you can also use Jupyter Notebooks or even a Python console. This allows the execution of small code snippets and viewing of the result immediately.\n\nIf you are using Jupyter Notebooks, enter these lines in a cell, and execute the cell. If you are using a code editor, enter these lines, save the file, and execute it with Python.\n\n```python\nimport requests\n\nurl_to_parse = \"https://en.wikipedia.org/wiki/Python_(programming_language)\"\nresponse = requests.get(url_to_parse)\nprint(response)\n```\n\nYou will see the output like this:\n\n```python\n\u003cResponse [200]\u003e\n```\n\nThis means we received a response object with status code 200 (a successful response).\nIf we check the type of response object by calling the ```type(response)```, we will see that it is an instance of ```requests.models.Response```.\nThis has many interesting properties, like ```status_code```, ```encoding```, and the most interesting of all — ```text```.\n\nEdit the code file so that ```response.text``` is printed\n\n```python\nprint(response.text)\n```\n\nYou will see that the output will be the entire HTML of the page. Here is the partial output:\n\n```html\n\u003c!DOCTYPE html\u003e\u003chtml class=\"client-nojs\" lang=\"en\" dir=\"ltr\"\u003e\u003chead\u003e\u003cmeta charset=\"UTF-8\"/\u003e\u003ctitle\u003ePython (programming language) - Wikipedia\u003c/title\u003e…\n```\n\nNow that we have the HTML ready, it’s time to move on to the next step.\n\n### STEP 2. How to parse the HTML?\n\nNow this HTML response, which currently is a string, needs to be parsed into an object. The most important thing here is that we should be able to easily query this object to get the desired data.\n\nWe can use parsing libraries directly. However, we will use another library called ```beautifulsoup4```. This sits on top of the parser. The advantage is that we can easily write selectors so that we can query this HTML markup and look for the data that we need.\n\nTo install this library, run the following on your terminal:\n\n```bash\npip install beautifulsoup4\n```\nOR\n\n```bash\npip install bs4\n```\n\nOnce the installation is complete, add the import statement and create an object of ```BeautifulSoup```. Here is the updated code:\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\nurl_to_parse = \"https://en.wikipedia.org/wiki/Python_(programming_language)\"\nresponse = requests.get(url_to_parse)\nsoup = BeautifulSoup(response.text,'html.parser')\n```\n\nNote that we are specifying the parsers as ```html.parser```. We can, however, use a different parser like ```lxml```.\nNow that we have the parsed object, we can now extract the data we need.\n\n### STEP 3. How to extract data?\n\nBeautifulSoup provides an easy way to navigate the data structure. Here are some examples and the output:\n\n```python\nsoup.title\n# output \u003ctitle\u003ePython (programming language) - Wikipedia\u003c/title\u003e\nsoup.title.name\n# output 'title'\nsoup.title.text\n# output 'Python (programming language) - Wikipedia'\nsoup.title.parent.name\n# output 'head'\n```\n\nIf you’re looking for a specific text, you first need to know where exactly that text is located in the HTML. In this example, we will try to extract the items from the table of contents of this Wikipedia page.\n\nOpen the url https://en.wikipedia.org/wiki/Python_(programming_language) in Chrome or Firefox, right click any item in the table of contents, and click Inspect. This will show that the text that we need is in ```\u003cdiv id=\"toc\" class=\"toc\"\u003e```\n\n![](https://images.prismic.io/oxylabs-sm/OTFkYzc4MTUtZTRmMS00ZGM5LWI5NDgtYTMwMzY5YjE3OGFk_wikipedia_devtools.png?auto=compress,format\u0026rect=0,0,773,333\u0026w=773\u0026h=333\u0026fm=webp\u0026dpr=2\u0026q=50)\n\nOnce we know where the text is located, we have two options:\n\n1. We can use the ```find()``` or ```find_all()``` method. \n2. Alternatively, we can use the ```select()``` method.\n\n#### Using find method with Beautiful Soup\n\nThe only difference between the ```find()``` and ```find_all()``` methods is that the ```find()``` method returns the first match, while ```find_all()``` returns them all.\n\nLet’s look at a few examples.\n\nIf we simply run ```soup.find(\"div\")```, it will return the first ```div``` it finds, which is the same as running ```soup.div```. This needs filtering as we need a specific ```div``` which contains the table of contents.\n\nIn this case, the whole table of contents is in the div that has it’s ```id``` set to ```toc```. This information can be supplied to the ```find()``` method as the second argument.\n\n```python\nsoup.find(\"div\",id=\"toc\")\n```\n\nThis will return everything inside the first ```div``` which has its ```id``` set to ```toc```. It also means that instead of ```div```, this method can accept any tag.\n\nLet’s take another example. In this page, there is a link with markup like this:\n\n```html\n\u003ca href=\"/wiki/End-of-life_(product)\" class=\"mw-redirect\" title=\"End-of-life (product)\"\u003eend-of-life\u003c/a\u003e\n```\n\nThis can be selected using any of these methods:\n\n```python\nsoup.find('a',title=\"End-of-life (product)\")\nsoup.find('a',href=\"/wiki/End-of-life_(product)\")\n```\n\nYou can even use more than one attribute:\n\n```python\nsoup.find('a',title=\"End-of-life (product)\",href=\"/wiki/End-of-life_(product)\")\n```\n\nNOTE. Be careful about ```class``` attributes. ```Class``` is a reserved keyword in Python. It means that you cannot use class in the same fashion:\n\n```python\nsoup.find('a',class=\"mw-redirect\") # SyntaxError: invalid syntax\n```\n\nThe workaround is to suffix class with an underscore:\n\n```python\nsoup.find('a',class_=\"mw-redirect\") # will return first a tag with this class\n```\n\n#### Using CSS selectors with BeautifulSoup\n\nBeautifulSoup also supports use of CSS selectors. This is arguably a better approach, because CSS selectors are generic and not specific to BeautifulSoup. Chances are that you already know how to build CSS selectors. Even if you don’t know CSS selectors, learning CSS selectors would be a good idea as it can help in the future. Even JavaScript scraping packages work well with CSS selectors.\n\nNote that there are two options – ```select()``` and ```select_one()```. The ```select()``` method is similar to ```find_all()```. Both return a list of all the matching occurrences. The ```select_one()``` method is similar to the ```find()``` method, which returns the first matching occurrence.\n\nLet’s look at the same examples. To extract this link:\n\n```html\n\u003ca href=\"/wiki/End-of-life_(product)\" class=\"mw-redirect\" title=\"End-of-life (product)\"\u003eend-of-life\u003c/a\u003e\n```\n\nEither of these methods will work:\n\n```python\nsoup.select_one('a[title=\"End-of-life (product)\"]')\n\nsoup.select_one('a[href=\"/wiki/End-of-life_(product)\"]')\n```\n\nAgain, you can use more than one attribute:\n\n```python\nsoup.select_one('a[title=\"End-of-life (product)\"][href=\"/wiki/End-of-life_(product)\"]')\n```\n\nNOTE. When using more than one attribute, there should not be any space. This is standard CSS syntax and not specific to BeautifulSoup.\n\nWhile using class, the syntax is much cleaner. A class is represented as a period. Similarly, id is presented by #.\n\n```python\nsoup.select_one('a.mw-redirect') # will return the first a tag with mw-redirect\nsoup.select_one('a#mw-redirect') # will return the first a tag with id mw-redirect\n```\n\nIf you want to chain more than one class, write the classes separated with a period, but no space. \n\n```python\nsoup.select_one('a.mw-redirect.external') # will return the first a tag with classes mw-redirect and external.\n```\n\nComing back to the example of Wikipedia Table of contents, the following snippet will return all the span with class ```toctext```.\n\n```python\ntoc = soup.select(\"span.toctext\")\nfor item in toc:\nprint(item)\n# OUTPUT\n# \u003cspan class=\"toctext\"\u003eHistory\u003c/span\u003e\n# \u003cspan class=\"toctext\"\u003eDesign philosophy and features\u003c/span\u003e\n# \u003cspan class=\"toctext\"\u003eSyntax and semantics\u003c/span\u003e\n```\n\nThis code returns all elements.  If we check the type of these elements, it will be ```bs4.element.Tag```. Typically, we would need the text insides these elements.  This is as simple as getting the ```.text``` of the elements.\n\n```python\ntoc = soup.select(\"span.toctext\")\nfor item in toc:\nprint(item.text)\n# OUTPUT\n# History\n# Design philosophy and features\n# Syntax and semantics\n# ...\n```\n\nLet’s get one more piece of information from the table of contents – the toc number. For example, the toc number of “Syntax and semantics” is 3, and the toc number for “Statements and control flow” is 3.1.\n\nTo get both of these, we can go through the parent elements, and again use the select method on the individual elements.\n\n```python\nfor item in soup.select('li.toclevel-1'):\ntoc_number = item.select_one('span.tocnumber').text\nprint(toc_number)\n# OUTPUT\n# 1\n# 2\n# 3\n# …\n```\n\nThe most important point here is that the ```select``` method works with ```beautifulsoup``` objects, as well as the elements extracted by ```select``` methods.\n\nWe can create dictionary inside the for loop and save everything in a list:\n\n```python\n# Create empty list\ndata = []\n# loop over outer elements\nfor item in soup.select('li.toclevel-1'):\n# Get the toc number element and it’s text\ntoc_number = item.select_one('span.tocnumber').text\n# Get the toc text element and it’s text\ntoc_text= item.select_one('span.toctext').text\n# Create a dictionary and add to the list\ndata.append({\n         'TOC Number': toc_number,\n         'TOC Text': toc_text\n})\n```\n\nNow we are ready to save this dictionary to a file or a database. To keep things simple, let’s begin with a file.\n\n### STEP 4. How to export data to CSV?\n\nExporting to CSV doesn’t need any installation. The csv module, which is bundled with Python installation, offers this functionality.\n\nHere is the code snippet with each line explained:\n\n```python\n# Import csv module\nimport csv\n# open a new file in write mode\nwith open('wiki.csv', 'w', newline='') as csvfile:\n# Specify the column names\nfieldnames = ['TOC Number', 'TOC Text']\n# create a dictionary writer object\nwriter = csv.DictWriter(csvfile, fieldnames=fieldnames)\n# write the headers, this will write the fieldnames as column headings\nwriter.writeheaders()\n# run a loop on the data\nfor item in data:\n# Each item in the list is a dictionary\n# this will be written in one row.\nwriter.writerow(item)\n```\n\nThe data is exported to a CSV file. This is the last step of building a web scraper in Python.\n\n### How to build a web scraper in JavaScript\n\nBuilding a web scraper in JavaScript follows the same steps:\n\n1. Get the HTML.\n2. Parse the Response.\n3. Extract desired data.\n4. Save the data.\n\n#### Preparing the Development Environment\n\nThe only software required are node.js and npm. Once you have node.js setup, open terminal and create a new node project:\n\n```bash\nnpm init -y\n```\n\nAfter that, install these three packages:\n\n```bash\nnpm install axios cheerio json2csv\n```\n\nNow let’s move on to the first step.\n\n### STEP 1. How to get the HTML?\n\nThe HTML page can be fetched by the package ```axios```.\n\nCreate a new file and enter these lines:\n\n```javascript\n// load axios\nconst axios = require(\"axios\");\nconst wiki_python = \"https://en.wikipedia.org/wiki/Python_(programming_language)\";\n\n//create an async function\n(async function() {\n// get the response\t\t\nconst response = await axios.get(wiki_python);\n// prints 200 if response is successful\nconsole.log(response.status)\n})();\n```\n\nWhile most of the code is a standard node.js code, the important line in this code is:\n\n```javascript\nconst response = await axios.get(url);\n```\n\nOnce this line executes, the response will contain the HTML that we need.\n\n### STEP 2. How to parse the HTML?\n\nFor parsing, the package that can be used is ```cheerio```.\n\nOpen the same file that we have been working on, and once the response is available, add the following line of code:\n\n```javascript\nconst $ = cheerio.load(response.data);\n```\n\nNote here that the HTML is being accessed using the ```data``` attribute of the ```response``` object created by Axios.\n\nIt’s also important to mention that instead of using a variable name, we are using the $ sign. This simply means that we will be able to write jQuery-like syntax and use CSS selectors.\n\n### STEP 3. How to extract data?\n\nThe desired data can be extracted using CSS selectors. For example, this line will select all the TOC elements:\n\n```javascript\nconst TOC = $(\"li.toclevel-1\"); \n```\n\nNow we can run a loop on all these elements, and select ```toc``` number and ```toc text```. The extract data can then be pushed to a list to create a JSON.\n\n```javascript\nconst toc_data = []\nTOC.each(function () {\n        level = $(this).find(\"span.tocnumber\").first().text();\n        text = $(this).find(\"span.toctext\").first().text();\n        toc_data.push({ level, text });\n    });\n```\n\nNow we are ready to save the data to a CSV.\n\n### STEP 4. How to export data to CSV?\n\nFor exporting data to CSV, we can simply use the package ```json2csv``` because we already have the data in JSON format. This will create the CSV in memory. To write this CSV to disk, we can use the fs package, which does not need to be installed separately.\n\n```javascript\nconst parser = new j2cp();\n    const csv = parser.parse(toc_data);\n    fs.writeFileSync(\"./wiki_toc.csv\", csv);\n```\n\nOnce everything is put together, this is how the entire code file would be:\n\n```javascript\nconst fs = require(\"fs\");\nconst j2cp = require(\"json2csv\").Parser;\nconst axios = require(\"axios\");\nconst cheerio = require(\"cheerio\");\n\nconst wiki_python =  \"https://en.wikipedia.org/wiki/Python_(programming_language)\";\n\nasync function getWikiTOC(url) {\n  try {\n    const response = await axios.get(url);\n    const $ = cheerio.load(response.data);\n\n    const TOC = $(\"li.toclevel-1\");\n    let toc_data = [];\n    TOC.each(function () {\n      level = $(this).find(\"span.tocnumber\").first().text();\n      text = $(this).find(\"span.toctext\").first().text();\n      toc_data.push({ level, text });\n    });\n    const parser = new j2cp();\n    const csv = parser.parse(toc_data);\n    fs.writeFileSync(\"./wiki_toc.csv\", csv);\n  } catch (err) {\n    console.error(err);\n  }\n}\n\ngetWikiTOC(wiki_python);\n```\n\nSave the above code as ```wiki_toc.js```, open the terminal, and run ```node wiki_toc.js```. This will save the extracted data in ```wiki_toc.csv``` file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-build-web-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fhow-to-build-web-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fhow-to-build-web-scraper/lists"}