{"id":20710197,"url":"https://github.com/oxylabs/beautiful-soup-parsing-tutorial","last_synced_at":"2025-08-25T13:17:07.368Z","repository":{"id":134336579,"uuid":"526099515","full_name":"oxylabs/beautiful-soup-parsing-tutorial","owner":"oxylabs","description":"Using Python \u0026 Beautiful Soup to Parse Data","archived":false,"fork":false,"pushed_at":"2025-06-26T08:22:30.000Z","size":19,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-26T09:30:34.318Z","etag":null,"topics":["beautiful-soup-parsing","beautifulsoup","beautifulsoup-html-parser","github-python","how-to-use-beautiful-soup-in-python","parser","python","python-beautiful-soup","python-web-scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-08-18T07:15:40.000Z","updated_at":"2025-06-26T08:22:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"cc78a6b7-9e92-4d4b-9ed3-8840c6f83eaa","html_url":"https://github.com/oxylabs/beautiful-soup-parsing-tutorial","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/oxylabs/beautiful-soup-parsing-tutorial","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbeautiful-soup-parsing-tutorial","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbeautiful-soup-parsing-tutorial/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbeautiful-soup-parsing-tutorial/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbeautiful-soup-parsing-tutorial/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/beautiful-soup-parsing-tutorial/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fbeautiful-soup-parsing-tutorial/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272072196,"owners_count":24868382,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-25T02:00:12.092Z","response_time":1107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautiful-soup-parsing","beautifulsoup","beautifulsoup-html-parser","github-python","how-to-use-beautiful-soup-in-python","parser","python","python-beautiful-soup","python-web-scraper"],"created_at":"2024-11-17T02:10:28.165Z","updated_at":"2025-08-25T13:17:07.360Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using Python and Beautiful Soup to Parse Data: Intro Tutorial\n\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=beautiful-soup-parsing-tutorial-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n## Installing Beautiful Soup\n\n```bash\npip install BeautifulSoup4\n```\n\n## Getting started\n\nA sample HTML file will help demonstrate the main methods of how Beautiful Soup parses data. This file is much more simple than your average modern website, however, it will be sufficient for the scope of this tutorial.\n\n```html\n\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n    \u003chead\u003e\n        \u003ctitle\u003eWhat is a Proxy?\u003c/title\u003e\n        \u003cmeta charset=\"utf-8\"\u003e\n    \u003c/head\u003e\n\n    \u003cbody\u003e\n        \u003ch2\u003eProxy types\u003c/h2\u003e\n\n        \u003cp\u003e\n          There are many different ways to categorize proxies. However, two of   \n the most popular types are residential and data center proxies. Here is a list of the most common types.\n        \u003c/p\u003e\n\n        \u003cul id=\"proxytypes\"\u003e\n            \u003cli\u003eResidential proxies\u003c/li\u003e\n            \u003cli\u003eDatacenter proxies\u003c/li\u003e\n            \u003cli\u003eShared proxies\u003c/li\u003e\n            \u003cli\u003eSemi-dedicated proxies\u003c/li\u003e\n            \u003cli\u003ePrivate proxies\u003c/li\u003e\n        \u003c/ul\u003e\n\n    \u003c/body\u003e\n\u003c/html\u003e\n```\n\n## Traversing for HTML tags\n\nFirst, we can use Beautiful Soup to extract a list of all the tags used in our sample HTML file. For this, we will use the soup.descendants generator.\n\n```python\nfrom bs4 import BeautifulSoup\n\nwith open('index.html', 'r') as f:\n    contents = f.read()\n\n    soup = BeautifulSoup(contents, features=\"html.parser\")\n\n    for child in soup.descendants:\n\n        if child.name:\n            print(child.name)\n```\n\nAfter running this code (right click on code and click “Run”) you should get the below output:\n\n```html\nhtml\nhead\ntitle\nmeta\nbody\nh2\np\nul\nli\nli\nli\nli\nli\n```\n\nWhat just happened? Beautiful Soup traversed our HTML file and printed all the HTML tags that it has found sequentially. Let’s take a quick look at what each line did.\n\n```python\nfrom bs4 import BeautifulSoup\n```\n\nThis tells Python to use the Beautiful Soup library.\n\n```python\nwith open('index.html', 'r') as f:\n    contents = f.read()\n```\n\nAnd this code, as you could probably guess, gives an instruction to open our sample HTML file and read its contents.\n\n```python\n    soup = BeautifulSoup(contents, features=\"html.parser\")\n```\n\nThis line creates a BeautifulSoup object and passes it to Python’s built-in BeautifulSoup HTML parser. Other parsers, such as lxml, might also be used, but it is a separate external library and for the purpose of this tutorial the built-in parser will do just fine.\n\n```python\n    for child in soup.descendants:\n\n        if child.name:\n            print(child.name)\n```\n\nThe final pieces of code, namely the soup.descendants generator, instruct Beautiful Soup to look for HTML tags and print them in the PyCharm console. The results can also easily be exported to a .csv file but we will get to this later.\n\n## Getting the full content of tags\n\nTo get the content of tags, this is what we can do:\n\n```python\nfrom bs4 import BeautifulSoup\n\nwith open('index.html', 'r') as f:\n    contents = f.read()\n\n    soup = BeautifulSoup(contents, features=\"html.parser\")\n\n    print(soup.h2)\n    print(soup.p)\n    print(soup.li)\n```\n\nThis is a simple instruction that outputs the HTML tag with its full content in the specified order. Here’s what the output should look like:\n\n```html\n\u003ch2\u003eProxy types\u003c/h2\u003e\n\u003cp\u003e\n          There are many different ways to categorize proxies.  However, two of the most popular types are residential and data center proxies. Here is a list of the most common types.\n        \u003c/p\u003e\n\u003cli\u003eResidential proxies\u003c/li\u003e\n```\n\nYou could also remove the HTML tags and print text only, by using, for example:\n\n```python\n    print(soup.li.text)\n```\n\nWhich in our case will give the following output:\n\n```html\nResidential proxies\n```\n\nNote that this only prints the first instance of the specified tag. Let’s continue to see how to find elements by ID or using the find_all method to filter elements by specific criteria.\n\n## Using Beautiful Soup to find elements by ID\n\nWe can use two similar ways to find elements by ID:\n\n```python\n    print(soup.find('ul', attrs={'id': 'proxytypes'}))\n```\n\nor\n\n```python\n    print(soup.find('ul', id='proxytypes'))\n```\n\nBoth of these will output the same result in the Python Console:\n\n```html\n\u003cul id=\"proxytypes\"\u003e\n\u003cli\u003eResidential proxies\u003c/li\u003e\n\u003cli\u003eDatacenter proxies\u003c/li\u003e\n\u003cli\u003eShared proxies\u003c/li\u003e\n\u003cli\u003eSemi-dedicated proxies\u003c/li\u003e\n\u003cli\u003ePrivate proxies\u003c/li\u003e\n\u003c/ul\u003e\n```\n\n## Finding all specified tags and extracting text\n\nThe find_all method is a great way to extract specific data from an HTML file. It accepts many criteria that make it a flexible tool allowing us to filter data in convenient ways. Yet for this tutorial we do not need anything more complex. Let’s find all items of our list and print them as text only:\n\n```python\n   for tag in soup.find_all('li'):\n        print(tag.text)\n```\n\nThis is how the full code should look like:\n\n```python\nfrom bs4 import BeautifulSoup\n\nwith open('index.html', 'r') as f:\n    contents = f.read()\n\n    soup = BeautifulSoup(contents, features=\"html.parser\")\n\n    for tag in soup.find_all('li'):\n        print(tag.text)\n```\n\nAnd here’s the output:\n\n```\nResidential proxies\nDatacenter proxies\nShared proxies\nSemi-dedicated proxies\nPrivate proxies\n```\n\n## Exporting data to a .csv file\n\n```bash\npip install pandas\n```\n\nAdd this line to the beginning of your code to import the library:\n\n```python\nimport pandas as pd\n```\n\nGoing further, let’s add some lines that will export the list we extracted earlier to a .csv file. This is how our full code should look like:\n\n```python\nfrom bs4 import BeautifulSoup\nimport pandas as pd\n\nwith open('index.html', 'r') as f:\n    contents = f.read()\n\n    soup = BeautifulSoup(contents, features=\"html.parser\")\n    results = soup.find_all('li')\n\n    df = pd.DataFrame({'Names': results})\n    df.to_csv('names.csv', index=False, encoding='utf-8')\n```\n\nWhat happened here? Let’s take a look:\n\n```python\n    results = soup.find_all('li')\n```\n\nThis line finds all instances of the `\u003cli\u003e` tag and stores it in the results object.\n\n```python\n    df = pd.DataFrame({'Names': results})\n    df.to_csv('names.csv', index=False, encoding='utf-8')\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fbeautiful-soup-parsing-tutorial","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fbeautiful-soup-parsing-tutorial","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fbeautiful-soup-parsing-tutorial/lists"}