{"id":27979390,"url":"https://github.com/rayxiang03/indeed-job-scraping","last_synced_at":"2026-04-30T13:33:48.951Z","repository":{"id":291784070,"uuid":"978750107","full_name":"rayxiang03/Indeed-Job-Scraping","owner":"rayxiang03","description":"Python toolkit for scraping Indeed job listings, preprocessing data, and generating visualizations for market analysis.","archived":false,"fork":false,"pushed_at":"2025-05-06T14:00:05.000Z","size":789,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-08T02:52:43.283Z","etag":null,"topics":["cloudscraper","data-visualization","indeed","job-analysis","nlp","pandas","python","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rayxiang03.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-06T13:03:49.000Z","updated_at":"2025-05-06T14:02:53.000Z","dependencies_parsed_at":"2025-05-06T15:05:44.544Z","dependency_job_id":null,"html_url":"https://github.com/rayxiang03/Indeed-Job-Scraping","commit_stats":null,"previous_names":["rayxiang03/indeed-job-scraping"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayxiang03%2FIndeed-Job-Scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayxiang03%2FIndeed-Job-Scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayxiang03%2FIndeed-Job-Scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rayxiang03%2FIndeed-Job-Scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rayxiang03","download_url":"https://codeload.github.com/rayxiang03/Indeed-Job-Scraping/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252989963,"owners_count":21836667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudscraper","data-visualization","indeed","job-analysis","nlp","pandas","python","web-scraping"],"created_at":"2025-05-08T02:52:46.239Z","updated_at":"2026-04-30T13:33:48.944Z","avatar_url":"https://github.com/rayxiang03.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🔍 Indeed Job Scraping \u0026 Analysis Tool\n\n\u003cdiv align=\"center\"\u003e\n  \n![GitHub stars](https://img.shields.io/github/stars/rayxiang03/Indeed-Job-Scraping?style=social)\n![GitHub forks](https://img.shields.io/github/forks/rayxiang03/Indeed-Job-Scraping?style=social)\n![GitHub issues](https://img.shields.io/github/issues/rayxiang03/Indeed-Job-Scraping)\n![GitHub license](https://img.shields.io/github/license/rayxiang03/Indeed-Job-Scraping)\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n\n\u003c/div\u003e\n\n\n## 📋 Table of Contents\n\n- [Overview](#-overview)\n- [Features](#-features)\n- [System Requirements](#-system-requirements)\n- [Installation](#-installation)\n- [Project Structure](#-project-structure)\n- [Usage Guide](#-usage-guide)\n- [Visualization Examples](#-visualization-examples)\n- [Troubleshooting](#-troubleshooting)\n- [Contributing](#-contributing)\n- [License](#-license)\n\n## 🚀 Overview\n\nThis toolkit provides robust solutions for web data extraction, preprocessing, and advanced visualization. It's designed specifically for analyzing job market data, with built-in mechanisms to handle anti-scraping measures, perform natural language processing on job descriptions, and generate actionable insights through comprehensive visualizations.\n\n## ✨ Features\n\n\u003cdiv align=\"center\"\u003e\n  \u003ctable\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/1.png\" width=\"64\" height=\"64\"\u003e\u003cbr\u003e\n        \u003cb\u003eAdvanced Scraping\u003c/b\u003e\u003cbr\u003e\n        \u003csmall\u003eBypasses common anti-scraping protections\u003c/small\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/2.png\" width=\"64\" height=\"64\"\u003e\u003cbr\u003e\n        \u003cb\u003eData Cleaning\u003c/b\u003e\u003cbr\u003e\n        \u003csmall\u003eAutomated text normalization \u0026 correction\u003c/small\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/3.png\" width=\"64\" height=\"64\"\u003e\u003cbr\u003e\n        \u003cb\u003eNLP Integration\u003c/b\u003e\u003cbr\u003e\n        \u003csmall\u003eTransformer models for text analysis\u003c/small\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/Visualization.gif\" width=\"64\" height=\"64\"\u003e\u003cbr\u003e\n        \u003cb\u003eData Visualization\u003c/b\u003e\u003cbr\u003e\n        \u003csmall\u003eMultiple chart types \u0026 word clouds\u003c/small\u003e\n      \u003c/td\u003e\n      \u003ctd align=\"center\"\u003e\n        \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/5.png\" width=\"64\" height=\"64\"\u003e\u003cbr\u003e\n        \u003cb\u003eInsight Generation\u003c/b\u003e\u003cbr\u003e\n        \u003csmall\u003eExtract actionable job market trends\u003c/small\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/table\u003e\n\u003c/div\u003e\n\n## 💻 System Requirements\n\n- Python 3.8 or higher\n- 4GB+ RAM (8GB+ recommended for larger datasets)\n- Active internet connection for data scraping\n- IDE: Visual Studio Code or IntelliJ IDEA (recommended)\n\n## 📦 Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/rayxiang03/Indeed-Job-Scraping.git\ncd Indeed-Job-Scraping\n\n# Create and activate virtual environment (optional but recommended)\npython -m venv venv\nsource venv/bin/activate  # On Windows: venv\\Scripts\\activate\n\n# Install required packages\npip install beautifulsoup4 cloudscraper requests pandas matplotlib fake-useragent nltk \\\n            torch sentencepiece transformers pyspellchecker wordcloud numpy\n```\n\nAlternatively, you can install the required packages directly:\n\n```bash\npip install beautifulsoup4 cloudscraper requests pandas matplotlib fake-useragent nltk \\\n            torch sentencepiece transformers pyspellchecker wordcloud numpy\n```\n\n## 📁 Project Files\n\n```\n├── code_WebScraping.py      # Web scraping and data preprocessing script\n├── code2_Analysis.py        # Data visualization and analysis script\n└── indeed_job.csv           # Generated dataset (after running code_WebScraping.py)\n```\n\n## 🔧 Usage Guide\n\n### Web Scraping Module (`code_WebScraping.py`)\n\nThis script handles data extraction from targeted websites, text preprocessing, and dataset creation:\n\n1. Open the script in your preferred IDE (VS Code or IntelliJ IDEA)\n2. Verify your internet connection\n3. Execute the script:\n\n```bash\npython code_WebScraping.py\n```\n\nThe script will:\n- Connect to specified job websites\n- Extract job listings data\n- Clean and preprocess text content\n- Create a structured DataFrame\n- Export the data to `indeed_job.csv`\n\n### Visualization Module (`code2_Analysis.py`)\n\nThis script loads the previously scraped data and generates various visualizations:\n\n1. Ensure `indeed_job.csv` is in the same directory\n2. Run the script:\n\n```bash\npython code2_Analysis.py\n```\n\nThe script will generate visualizations for:\n- Job category distributions\n- Geographic job distribution\n- Salary range analysis\n- Keyword frequency analysis\n- Word clouds of most common terms\n- Other insightful data visualizations\n\n## 📊 Visualization Examples\n\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/Top%2010%20Job%20Categories.png\" alt=\"Job Categories Chart\" width=\"400\"\u003e\n  \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/image.png\" alt=\"Location Distribution\" width=\"400\"\u003e\n  \u003cbr\u003e\n  \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/Average%20Salary%20by%20Location.png\" alt=\"Salary Distribution\" width=\"400\"\u003e\n  \u003cimg src=\"https://github.com/rayxiang03/Indeed-Job-Scraping/blob/master/Screenshots/wordcloud.png\" alt=\"Skills Word Cloud\" width=\"400\"\u003e\n\u003c/div\u003e\n\n## ⚠️ Troubleshooting\n\n### Common Issues and Solutions\n\n| Issue | Solution |\n|-------|----------|\n| **403 Forbidden Errors** | • Use a VPN to change your IP address\u003cbr\u003e• Connect to a mobile hotspot\u003cbr\u003e• Switch to a different WiFi network\u003cbr\u003e• Increase delay between requests |\n| **Missing Dependencies** | Install all required packages using the pip command in the installation section |\n| **Memory Errors** | Reduce batch size in data processing or use a machine with more RAM |\n| **Visualization Errors** | Ensure matplotlib backend is properly configured for your environment |\n| **CSV Loading Errors** | Verify `indeed_job.csv` exists and has proper formatting |\n\n### Advanced IP Rotation Techniques\n\nFor persistent scraping issues, consider implementing:\n\n- Proxy rotation services\n- Tor network integration\n- Cloud-based scraping with IP rotation\n\n## 🤝 Contributing\n\nContributions are welcome! Here's how you can help:\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\nPlease ensure your code follows the project's coding style and includes appropriate tests.\n\n## 📄 License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n\u003cdiv align=\"center\"\u003e\n  \u003cp\u003e\n    \u003csub\u003eBuilt with ❤️ by \u003ca href=\"https://github.com/rayxiang03\"\u003erayxiang03\u003c/a\u003e\u003c/sub\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayxiang03%2Findeed-job-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frayxiang03%2Findeed-job-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayxiang03%2Findeed-job-scraping/lists"}