{"id":21898560,"url":"https://github.com/zzarif/industry-market-cap-analysis","last_synced_at":"2026-04-18T02:32:39.561Z","repository":{"id":232238319,"uuid":"783784549","full_name":"zzarif/Industry-Market-Cap-Analysis","owner":"zzarif","description":"Financial analysis dashboards comparing multiple industries.","archived":false,"fork":false,"pushed_at":"2024-08-07T08:31:33.000Z","size":907,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-14T03:05:33.550Z","etag":null,"topics":["csv","data-analytics","data-science","data-scraping","pandas","selenium","selenium-python","tableau","tableau-dashboards","tableau-public"],"latest_commit_sha":null,"homepage":"https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/IndustryComparisonDashboard","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zzarif.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-04-08T15:06:37.000Z","updated_at":"2025-02-05T13:24:53.000Z","dependencies_parsed_at":"2024-05-05T17:34:57.663Z","dependency_job_id":"c50ea571-a48c-4d4a-839b-b5b1f43b2583","html_url":"https://github.com/zzarif/Industry-Market-Cap-Analysis","commit_stats":null,"previous_names":["zzarif/tableau-industry-insights-dashboard","zzarif/industry-market-cap-analysis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zzarif/Industry-Market-Cap-Analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzarif%2FIndustry-Market-Cap-Analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzarif%2FIndustry-Market-Cap-Analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzarif%2FIndustry-Market-Cap-Analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzarif%2FIndustry-Market-Cap-Analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zzarif","download_url":"https://codeload.github.com/zzarif/Industry-Market-Cap-Analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zzarif%2FIndustry-Market-Cap-Analysis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31953781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T00:39:45.007Z","status":"online","status_checked_at":"2026-04-18T02:00:07.018Z","response_time":103,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","data-analytics","data-science","data-scraping","pandas","selenium","selenium-python","tableau","tableau-dashboards","tableau-public"],"created_at":"2024-11-28T14:33:11.676Z","updated_at":"2026-04-18T02:32:39.541Z","avatar_url":"https://github.com/zzarif.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003c!-- \u003ca href=\"http://www.amitmerchant.com/electron-markdownify\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/amitmerchant1990/electron-markdownify/master/app/img/markdownify.png\" alt=\"Markdownify\" width=\"200\"\u003e\u003c/a\u003e\n  \u003cbr\u003e --\u003e\n  Industry Market Cap Analysis\n  \u003cbr\u003e\n\u003c/h1\u003e\n\n\u003ch4 align=\"center\"\u003eFinancial analysis dashboards comparing 1500 companies across 8 industries.\u003c/h4\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003c!-- \u003ca href=\"https://badge.fury.io/js/electron-markdownify\"\u003e\n    \u003cimg src=\"https://badge.fury.io/js/electron-markdownify.svg\"\n         alt=\"Gitter\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://gitter.im/amitmerchant1990/electron-markdownify\"\u003e\u003cimg src=\"https://badges.gitter.im/amitmerchant1990/electron-markdownify.svg\"\u003e\u003c/a\u003e --\u003e\n  \u003c!-- \u003ca href=\"\"\u003e\n      \u003cimg src=\"https://img.shields.io/badge/website-online-blue.svg\"\u003e\n  \u003c/a\u003e --\u003e\n  \u003ca href=\"https://github.com/zzarif/Industry-Market-Cap-Analysis\"\u003e\n    \u003cimg src=\"https://img.shields.io/github/last-commit/zzarif/Industry-Market-Cap-Analysis\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.kaggle.com/datasets/zibranzarif/industry-market-cap-analysis-dataset\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/dataset-kaggle-blue.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/IndustryComparisonDashboard\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/tableau-public-red.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/license-MIT-yellow.svg\"\u003e\n  \u003c/a\u003e\n\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#-problem-statement\"\u003eProblem Statement\u003c/a\u003e •\n  \u003ca href=\"#findings-and-observations-from-the-dashboards\"\u003eFindings from Dashboards\u003c/a\u003e •\n  \u003ca href=\"#%EF%B8%8F-build-from-source\"\u003eBuild from Source\u003c/a\u003e •\n  \u003ca href=\"#-run-the-selenium-scraper\"\u003eRun the Selenium Scraper\u003c/a\u003e •\n  \u003ca href=\"#-transform-and-clean-the-data\"\u003eTransform and Clean the data\u003c/a\u003e •\n  \u003ca href=\"#%EF%B8%8F-contact\"\u003eContact\u003c/a\u003e\n\u003c/p\u003e\n\n## 📋 Problem Statement\nComprehensive financial analysis [dashboards](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/IndustryComparisonDashboard) comparing the financial performance and key metrics of **1500** companies across **8** different industries. Utilizing the financial information from [companiesmarketcap.com](https://companiesmarketcap.com/) the dashboard offers a wide range of visualizations and metrics, including:\n\n1. Bar charts comparing the average market cap, revenue, and earnings of companies in each industry.\n2. Employee productivity scatter plots presenting Revenue-per-Employee and Earnings-per-Employee color coded by industry.\n3. Geographic distribution of market capital and companies.\n4. Comparative Price-to-Sales and Price-to-Earning Ratio identifying growth-oriented industries.\n5. Box plots revealing the typical range and variability of operating margins of companies in each industry.\n6. Debt-to-Equity and Current Ratio heatmaps showing financial health of companies.\n\nHere are the links to Tableau public dashboards:\n- [Industry Comparison Dashboard](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/IndustryComparisonDashboard)\n- [Geographical Distribution and Operating Margin Dashboard](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/GeographicDistributionandOperatingMarginDashboard)\n- [Financial Health Heatmaps Dashboard](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/FinancialHealthDashboard)\n\n## Findings and Observations from the [Dashboards](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/IndustryComparisonDashboard)\n\n### 📊 [Dashboard 1:](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/IndustryComparisonDashboard) Industry Comparison\n![industry_comparison_dashboard](dashboard/images/industry_comparison_dashboard.png)\nFindings:\n1. **Tehnology** has the highest average market value in general and **Real Estate** is the lowest. However, **Oil \u0026 Gas** is the leading industry in terms of total country-wise average market cap. **Saudi Arabia** is the largest contributor.\n2. **Oil \u0026 Gas** industry dominates the market in terms of Average Revenue and Earnings. **Real Estate** is the lowest.\n3. In terms of Revenue per employee, **Saudi Aramco** (Oil \u0026 Gas) is the highest whereas **Phoenix Group** (Insurance) is the lowest. \n4. In terms of Earnings per employee, again **Saudi Aramco** (Oil \u0026 Gas) is the highest whereas **Walgreens Boots Alliance** (Pharmaceuticals) is the lowest. On average, **Oil \u0026 Gas** is the most employee productive industry.\n\n### 📊 [Dashboard 2:](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/GeographicDistributionandOperatingMarginDashboard) Geographical Distribution and Operating Margin\n![geographical_dist_and_operating_margin_dashboard](dashboard/images/geographical_dist_and_operating_margin_dashboard.png)\nFindings:\n1. **Saudi Arabia** has the largest share of market capital ($248.3T), followed by **USA** ($56.4T), and **Denmark** ($51.6T)\n2. From the 1500 total companies, 591 belong to **USA** alone, followed by **India** (96), **China** (67), and **Hong Kong** (48)\n3. **Real Estate** has the highest median operating margin at around 15.4%, followed by **Oil \u0026 Gas** and **Pharmaceuticals** both at approximately 11.1%. This suggests these industries tend to have healthier operating profitability on average.\n4. The **Pharmaceuticals** industry has the widest spread between its upper and lower whiskers, indicating it has the most variability in operating margins. Companies span a wide range from highly profitable to unprofitable.\n5. Insurance, Oil \u0026 Gas, and Real Estate have their entire boxes above the zero line, meaning **over 75%** of companies have positive margins. In contrast, **Pharmaceuticals** and **Food** have a large portion of their boxes below zero, indicating a significant number of unprofitable companies.\n6. **Technology** companies tend to have both high P/S and high P/E ratios compared to other industries. It is a growth-oriented sector.\n7. **Retail** and **Food** industries generally have lower P/S and P/E ratios. These are slower-growth sectors.\n\n### 📊 [Dashboard 3:](https://public.tableau.com/app/profile/zibran.zarif/viz/IndustryInsightsDashboard/FinancialHealthDashboard) Financial Health Heatmaps Dashboard\n![financial_health_dashboard](dashboard/images/financial_health_dashboard.png)\nFindings:\n1. **Real Estate** companies from the **UK** have the most average Debt. Same goes for **Pharmaceuticals** (Canada), **Retail** (Saudi Arabia), and **Insurance** (Japan).\n2. **Food** companies in **Malaysia** have significantly more total assets relative to its total liabilities. Same goes for **Oil \u0026 Gas** (Russia), and **Retail** (Greece).\n\n## ⚙️ Build from Source\n1. Clone the repo\n```bash\ngit clone https://github.com/zzarif/Industry-Market-Cap-Analysis.git\ncd Industry-Market-Cap-Analysis/\n```\n2. Initialize and activate virtual environment\n```bash\nvirtualenv --no-site-packages venv\nsource venv/Scripts/activate\n```\n3. Install dependencies\n```bash\npip install -r requirements.txt\n```\n*Note: Select virtual environment interpreter from* `Ctrl`+`Shift`+`P`\n## 🌐 Run the Selenium Scraper\n### Traditional Approach\n```bash\npython scraper/main.py\n```\nRun this command and wait for it to finish. When complete, you will get a file named [scraped_company_data.csv](data/scraped_company_data.csv) (this file requires data transformation in the next step).\n\n#### How the scraping is done:\nThe `main.py` file calls `fetch_companies` method 8 times for 8 different industries. Each time `fetch_companies` method fetches 200 companies' data for each industry (100 companies for *Electricity* industry) totalling to 1500 companies' data. `fetch_companies` method uses `selenium` webdriver to scrape [companiesmarketcap.com](https://companiesmarketcap.com/) with necessary `chrome_options` added as arguments. At a time, it will iterate over maximum 2 pages to scrape 200 companies' data (each page has 100 companies listed). During each iteration, the outer `for` loop, fetches 100 companies. The inner `for` loops go over each of those companies and fetch that company specific metrics. Finally, the companies and their respective data is returned to `main.py` file where it is appended to the global `companies` list and converted to [scraped_company_data.csv](data/scraped_company_data.csv) file. This file requires data transformation in the next step.\n\n### Alternative Approach (Scraping Parallelly)\nUsing traditional approach, scraping financial data for 1500 companies one-by-one might take a significant amount of time (several hours) depending on your network bandwidth. A better and faster approach would be to split the task into multiple scraper instances that will scrape data parallely. Each scraper will be assigned to scrape financial data of the companies belonging to a particular industry.\n\nTo do this, you can simply create 8 copies of the `main.py` file (for 8 industries) and modify each `main.py` to fetch companies only for a particular industry. Now, run all the copies parallelly and merge the output CSV files.\n\nAlternatively, you can use Python's `multiprocessing` module to spawn multiple processes to accomplish the same task.\n\nBe sure to rename the final merged CSV file as [scraped_company_data.csv](data/scraped_company_data.csv) (this file requires data transformation in the next step)\n\n## 🔄 Transform and Clean the data\n```bash\npython data_transformation/transform_data.py\n```\nAt this stage, you will get a file named [transformed_company_data.csv](data/transformed_company_data.csv) (you can load this file into Tableau as Text file)\n\n## ✉️ Contact\n\n[![LinkedIn](https://img.shields.io/badge/LinkedIn-0077B5?logo=linkedin\u0026logoColor=white)](https://www.linkedin.com/in/zibran-zarif-amio-b82717263/) [![Mail](https://img.shields.io/badge/Gmail-EA4335?logo=gmail\u0026logoColor=fff)](mailto:zibran.zarif.amio@gmail.com)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzzarif%2Findustry-market-cap-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzzarif%2Findustry-market-cap-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzzarif%2Findustry-market-cap-analysis/lists"}