{"id":20710201,"url":"https://github.com/oxylabs/automate-competitors-benchmark-analysis","last_synced_at":"2025-09-02T13:11:24.844Z","repository":{"id":134336544,"uuid":"464461104","full_name":"oxylabs/automate-competitors-benchmark-analysis","owner":"oxylabs","description":"A tutorial for automating competitors’ \u0026 benchmark analysis using Python","archived":false,"fork":false,"pushed_at":"2025-02-11T12:44:44.000Z","size":17,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-29T22:21:58.237Z","etag":null,"topics":["analysis","automation","github-python","python","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oxylabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-28T11:48:28.000Z","updated_at":"2025-02-11T12:44:48.000Z","dependencies_parsed_at":null,"dependency_job_id":"fa19fd8e-52d3-4bd6-b80c-19f9b3200c41","html_url":"https://github.com/oxylabs/automate-competitors-benchmark-analysis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fautomate-competitors-benchmark-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fautomate-competitors-benchmark-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fautomate-competitors-benchmark-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oxylabs%2Fautomate-competitors-benchmark-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oxylabs","download_url":"https://codeload.github.com/oxylabs/automate-competitors-benchmark-analysis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250372946,"owners_count":21419722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","automation","github-python","python","web-scraping"],"created_at":"2024-11-17T02:10:29.678Z","updated_at":"2025-09-02T13:11:24.835Z","avatar_url":"https://github.com/oxylabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Automate Competitors’ \u0026 Benchmark Analysis With Python\n\n[![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877\u0026utm_medium=affiliate\u0026groupid=877\u0026utm_content=automate-competitors-benchmark-analysis-github\u0026transaction_id=102f49063ab94276ae8f116d224b67)\n\n[![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/GbxmdGhZjq)\n\n- [Using Oxylabs’ solution to retrieve the SERPs results](#using-oxylabs-solution-to-retrieve-the-serps-results)\n- [Scraping URLs of the top results](#scraping-urls-of-the-top-results)\n- [Obtaining the off-page metrics](#obtaining-the-off-page-metrics)\n- [Obtaining the Page Speed metrics](#obtaining-the-page-speed-metrics)\n- [Converting Python list into a dataframe and exporting it as an Excel file](#converting-python-list-into-a-dataframe-and-exporting-it-as-an-excel-file)\n\nDoing competitors’ or benchmark analysis for SEO can be a burdensome task as it requires taking into account many factors which usually are extracted from different data sources. \n\nThe purpose of this article is to help you automate the data extraction processes as much as possible. After learning how to do this, you can dedicate your time to what matters: the analysis itself and coming up with actionable insights to strategize.\n\nFor a detailed explanation, see our [blog post](https://oxy.yt/erEh).\n\n## Using Oxylabs’ solution to retrieve the SERPs results\n\n```python\nimport requests\n\nkeyword = \"\u003cyour_keyword\u003e\"\n\npayload = {\n    \"source\": \"SEARCH_ENGINE_search\",\n    \"domain\": \"com\",\n    \"query\": keyword,\n    \"parse\": \"true\",\n}\n\nresponse = requests.request(\n    \"POST\",\n    \"https://realtime.oxylabs.io/v1/queries\",\n    auth=(\"\u003cyour_username\u003e\", \"\u003cyour_password\u003e\"),\n    json=payload,\n)\n\nlist_comparison = [\n    [x[\"url\"], x[\"title\"]]\n    for x in response.json()[\"results\"][0][\"content\"][\"results\"][\"organic\"]\n]\n```\n\nViewing the results:\n\n```python\n\u003e\u003e\u003e print(list_comparison)\n[\n    [\"https://example.com/result/example-link\", \"Example Link - Example\"],\n    [\"https://more-examples.net\", \"Homepage - More Examples\"],\n    [\"https://you-searched-for.com/query=your_keyword\", \"You Searched for 'your_keyword'. Analyze your search now!\"],\n]\n```\n\n## Scraping URLs of the top results\n\n```python\nimport requests\nfrom bs4 import BeautifulSoup\n\nfor y in list_comparison:\n    try:\n        print(\"Scraping: \" + y[0])\n        html = requests.request(\"get\", y[0])\n        soup = BeautifulSoup(html.text)\n\n        try:\n            metatitle = (soup.find(\"title\")).get_text()\n        except Exception:\n            metatitle = \"\"\n\n        try:\n            metadescription = soup.find(\"meta\", attrs={\"name\": \"description\"})[\"content\"]\n        except Exception:\n             metadescription = \"\"\n\n        try:\n            h1 = soup.find(\"h1\").get_text()\n        except Exception:\n            h1 = \"\"\n\n        paragraph = [a.get_text() for a in soup.find_all('p')]\n        text_length = sum(len(a) for a in paragraph)\n        text_counter = sum(a.lower().count(keyword) for a in paragraph)\n        metatitle_occurrence = keyword in metatitle.lower()\n        h1_occurrence = keyword in h1.lower()\n        metatitle_equal = metatitle == y[1]        \n        y.extend([metatitle, metatitle_equal, metadescription, h1, paragraph, text_length, text_counter, metatitle_occurrence, h1_occurrence])\n\n    except Exception as e:\n        print(e)\n        y.extend([\"No data\"]*9)\n```\n\n## Obtaining the off-page metrics\n\n```python\nimport time\nfrom mozscape import Mozscape\n\nclient = Mozscape(\"\u003cMOZ username\u003e\", \"\u003cMOZ password\u003e\")\n\nfor y in list_comparison:\n    try:\n        print(\"Getting MOZ results for: \" + y[0])\n        domainAuthority = client.urlMetrics(y[0])\n        y.extend([domainAuthority[\"ueid\"], domainAuthority[\"uid\"], domainAuthority[\"pda\"]])\n    except Exception as e:\n        print(e)\n        time.sleep(10)  # Retry once after 10 seconds.\n        domainAuthority = client.urlMetrics(y[0])\n        y.extend([domainAuthority[\"ueid\"], domainAuthority[\"uid\"], domainAuthority[\"pda\"]])\n```\n\n## Obtaining the Page Speed metrics\n\n```python\nimport json\n\npagespeed_key = \"\u003cyour page speed key\u003e\"\n\n\nfor y in list_comparison:\n    try:\n\n        print(\"Getting results for: \" + y[0])\n        url = \"https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=\" + y[0] + \"\u0026strategy=mobile\u0026locale=en\u0026key=\" + pagespeed_key\n        response = requests.request(\"GET\", url)\n        data = response.json() \n\n        overall_score = data[\"lighthouseResult\"][\"categories\"][\"performance\"][\"score\"] * 100\n        fcp = data[\"loadingExperience\"][\"metrics\"][\"FIRST_CONTENTFUL_PAINT_MS\"][\"percentile\"]/1000\n        fid = data[\"loadingExperience\"][\"metrics\"][\"FIRST_INPUT_DELAY_MS\"][\"percentile\"]/1000\n        lcp = data[\"loadingExperience\"][\"metrics\"][\"LARGEST_CONTENTFUL_PAINT_MS\"][\"percentile\"]\n        cls = data[\"loadingExperience\"][\"metrics\"][\"CUMULATIVE_LAYOUT_SHIFT_SCORE\"][\"percentile\"]/100\n\n\n\n        y.extend([fcp, fid, lcp, cls, overall_score])\n\n    except Exception as e:\n        print(e)\n        y.extend([\"No data\", \"No data\", \"No data\", \"No data\", overall_score])\n```\n\n## Converting Python list into a dataframe and exporting it as an Excel file\n\n```python\nimport pandas as pd\n\ndf = pd.DataFrame(list_comparison)\ndf.columns = [\"URL\",\"Metatitle SERPs\", \"Metatitle Onpage\",\"Metatitle Equal\", \"Metadescription\", \"H1\", \"Paragraphs\", \"Text Length\", \"Keyword Occurrences Paragraph\", \"Metatitle Occurrence\", \"Metadescription Occurrence\", \"Equity Backlinks MOZ\", \"Total Backlinks MOZ\", \"Domain Authority\", \"FCP\", \"FID\",\"LCP\",\"CLS\",\"Overall Score\"]\ndf.to_excel('\u003cfilename\u003e.xlsx', header=True, index=False)\n```\n\nIf you wish to find out more, see our [blog post](https://oxy.yt/erEh).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fautomate-competitors-benchmark-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foxylabs%2Fautomate-competitors-benchmark-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foxylabs%2Fautomate-competitors-benchmark-analysis/lists"}