{"id":29741756,"url":"https://github.com/yazan-sharaya/upwork_analysis","last_synced_at":"2025-07-26T01:19:51.804Z","repository":{"id":241450366,"uuid":"800983140","full_name":"Yazan-Sharaya/upwork_analysis","owner":"Yazan-Sharaya","description":"Python package to collect and perform data analysis on Upwork job listings.","archived":false,"fork":false,"pushed_at":"2024-05-28T05:49:29.000Z","size":153,"stargazers_count":8,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-20T23:19:15.955Z","etag":null,"topics":["analysis","data-visualization","scraper","upwork","upwork-scraper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Yazan-Sharaya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-15T11:26:01.000Z","updated_at":"2025-06-12T05:01:10.000Z","dependencies_parsed_at":"2024-05-28T12:25:00.721Z","dependency_job_id":"f5956fc4-805b-4e88-b54f-dbe98cc53cfd","html_url":"https://github.com/Yazan-Sharaya/upwork_analysis","commit_stats":null,"previous_names":["yazan-sharaya/upwork_analysis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Yazan-Sharaya/upwork_analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yazan-Sharaya%2Fupwork_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yazan-Sharaya%2Fupwork_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yazan-Sharaya%2Fupwork_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yazan-Sharaya%2Fupwork_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Yazan-Sharaya","download_url":"https://codeload.github.com/Yazan-Sharaya/upwork_analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Yazan-Sharaya%2Fupwork_analysis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267098445,"owners_count":24035757,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analysis","data-visualization","scraper","upwork","upwork-scraper"],"created_at":"2025-07-26T01:19:48.539Z","updated_at":"2025-07-26T01:19:51.796Z","avatar_url":"https://github.com/Yazan-Sharaya.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Upwork Analysis\n===============\n[![PyPI version](https://badge.fury.io/py/upwork_analysis.svg)](https://badge.fury.io/py/upwork_analysis)\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/upwork_analysis)\n\nUpwork is a freelancing platform where clients post jobs and freelancers fight to get hired.\nI do freelance work there from time to time and I decided to follow the advice \"Work smarter not harder\" therefor I created this project.\nThe aim is to scrape jobs' data on Upwork and conduct data analysis to extract insights from the data.\n\n\n\u003c!-- TOC --\u003e\n* [Objective](#objective)\n* [Features](#features)\n* [Usage](#usage)\n    * [The easy way](#the-easy-way)\n    * [Scraping](#scraping)\n        * [CLI usage](#cli-usage)\n        * [Parameters](#parameters)\n        * [Python](#python)\n    * [Analysis](#analysis)\n        * [CLI usage](#cli-usage-1)\n        * [Python](#python-1)\n        * [Jupyter](#jupyter)\n* [Installation](#installation)\n    * [Automatic](#automatic)\n    * [Manual](#manual)\n* [Documentation](#documentation)\n* [Limitations](#limitations)\n* [License](#license)\n\u003c!-- TOC --\u003e\n\n\nObjective\n---------\nThis repository aims to perform the following two tasks.\n1. Scrape Job listings on Upwork for a specific search query.\n2. Analyze the scraped data to find the following:\n    1. The countries that pay the most.\n    2. Job frequency on different days of the week.\n    3. The most asked for skills.\n    4. The budget ranges/groups and their frequency.\n    5. The skills that correlate with higher budgets.\n    6. The relationship between these skills and the number of proposals.\n\n\nFeatures\n--------\n* **Fault tolerance**: The scraper has been built with fault tolerance in mind and has been thoroughly tested to ensure that.\n  Even if an error occurs that's not caused by the scraping process like a network error, a captcha or the user closing the\n  browser session, the scraper will gracefully stop scraping and it will save any scraped jobs.\n* **Retry**: Retrying functionality is baked in, so if a network error occurs or a captcha pops up, the scraper will retry to scrape\n  the specific page where the error occurred. This can be controlled by the 'retries' argument.\n* **Undetectability**: Upwork is guarded by Cloudflare's various protection mesures, the scraper bypasses these protections,\n  and from my many test runs, it didn't trigger any of the protections at all.\n* **Concurrency**: The scraping workload can distributed across multiple workers to speed up the scraping process tremendously.\n  This can be controlled by the 'workers' argument.\n* **No API**: The scraper doesn't need an Upwork account or an API key to work making it more broadly available, especially because\n  of hard it is to get an Upwork API key (they ask for more legal documents than when you sign up!)\n\n\nUsage\n-----\nThere are two main ways to get started with this project, one is through the command\nline and the other is by directly importing its functions and classes into your script.\nBoth achieve the same functionality, choose the one that best suits your needs.\n\n### The easy way\nIf you installed the package using pip, the easiest and most straightforward way to use is through the entry points.\n\nFor scraping\n```\nscrape-upwork scrape Python Developer -o python_jobs.json --headless\n```\nFor analysis\n```\nanalyze-upwork SAVED/JOBS/FILE.json -o PATH/TO/SAVE/DIR -s\n```\n\nContinue reading down below for more methods to use the package.\n\n### Scraping\n\n##### CLI usage\nTo scrape a new search query from scratch\n```\ncd PATH/TO/upwork_analysis\npython scrape_data.py scrape Python Developer -o python_jobs.json --headless\n```\nTo update existing data for a search query with any new job listings\n```\ncd PATH/TO/upwork_analysis\npython scrape_data.py update Python Developer -o PATH/TO/SAVE/FILE.json --headless\n```\n\n##### Parameters\n\n| Parameter              | Options       | Default         | Description                                                                                       |\n|------------------------|---------------|-----------------|---------------------------------------------------------------------------------------------------|\n| Action                 | scrape/update | scrape          | Scrape new jobs, or update existing scraped data with any new job postings.                       |\n| -q / --search-query    | str           | None (Required) | The query to search for.                                                                          |\n| -j / --jobs-per-page   | 10, 20, 50    | 10              | How many jobs should be displayed per page.                                                       |\n| -s / --start-page      | int           | 1               | The page number to start searching from.                                                          |\n| -p / --pages-to-scrape | int           | 10              | How many pages to scrape. If not passed, scrape all the pages.¹                                   |\n| -o / --output          | str           | -               | Where to save the scraped data.                                                                   |\n| -r / --retries         | int           | 3               | Number of retries when encountering a Captcha or timeout before failing.                          |\n| -w / --workers         | int           | 1               | How many webdriver instances to concurrently spin up for scraping.                                |\n| -f / --fast            | passed or not | False           | Whether to use the fast scraping method, can be 10 to 50x faster but leaves some information out. |\n| --headless             | passed or not | False           | Whether to enable headless mode (slower and more detectable).                                     |\n\n\u003csup\u003e¹ See the [limitations](#limitations) section\u003c/sup\u003e\n\n##### Python\n```python\nfrom upwork_analysis.scrape_data import JobsScraper\njobs_scraper = JobsScraper(\n    search_query=\"Python Developer\",\n    jobs_per_page=10,\n    start_page=1,\n    pages_to_scrape=10,\n    save_path='PATH/TO/SAVE/FILE.json',\n    retries=3,\n    headless=True,\n    workers=2,\n    fast=False)\njobs_scraper.scrape_jobs()\njobs_scraper.update_existing()\n```\n\nThis will scrape all resulting job listings for \"Python Developer\" from page 1 to page 10 and save the results to\n\"PATH/TO/SAVE/FILE.json\" using a headless browser. It will scrape a total of `jobs_per_page` * `pages_to_scrape` jobs or\n100 in this case.\n\n### Analysis\nA quick note, even though the analysis might run with as low as 1 data point _(or it might not and throw errors\nbecause there isn't enough data)_, it's better to scrape more data for the results to be meaningful.\n\n##### CLI usage\n```\ncd PATH/TO/upwork_analysis\npython analyize_data.py SAVED/JOBS/FILE.json -o PATH/TO/SAVE/DIR -s\n```\n\n##### Python\n```python\nfrom upwork_analysis.analyze_data import perform_analysis\nperform_analysis(\n    dataset_path='SAVED/JOBS/FILE.json',\n    save_plots_dir='PATH/TO/SAVE/DIR',\n    show_plots=True)\n```\n\n##### Jupyter\n```jupyter\njupyter notebook data_analysis.ipynb\n# Change the 1st line of cell 3 from\ndataset_path = \"\"\n# To\ndataset_path = \"SAVED/JOBS/FILE.json\"\n```\n\nThis will analyze the data saved at \"SAVED/JOBS/FILE.json\", save the resulting plots \"PATH/TO/SAVE/DIR\" and\n-s (--show, or show_plots=True) will show the resulting plots all at once.\n\nFor more documentation about available functions, their parameters and how to use them, see the [Documentation](#documentation) section.\n\n\nInstallation\n------------\nThis package requires python 3.7 or later.\n\n##### Automatic\n```\npip install upwork_analysis\n```\n**Note:** If you encounter an error during installing, update pip using `python -m pip install -U pip` and it should fix the issue.\n\n##### Manual\n1. Clone this repository\n    ```\n    git clone https://github.com/Yazan-Sharaya/upwork_analysis\n    ```\n2. Download the dependencies\n    * If you just want the scraping functionality\n    ```\n    pip install seleniumbase beautifulsoup4\n    ```\n    * And additionally for analyzing the data\n    ```\n    pip install pandas scipy seaborn scikit-learn\n    ```\n3. Build the package using\n    ```\n    python -m build -s\n    ```\n4. Install the package\n    ```\n    pip install upwork_analysis-1.0.0.tar.gz\n    ```\n\n\nDocumentation\n-------------\nBoth modules and all the functions they implement are documented using function and module docstrings.\nTo save you sometime, here's a list that covers the documentation for 99% of use cases.\n* For documentation about scraping, check out the docstrings for `JobsSraper`, `JobsScraper.scrape_jobs` and `JobsScraper.update_existing`.\n  **Note:** You can display the documentation of a function, class or module using the build-in `help()` function.\n* As for the data analysis part, checkout out `analyze_data` module docstring and `perform_analysis` function.\n* For help on command line usage, append `-h` option to `python scrape_data.py` or `python analyze_data.py`.\n\n\nLimitations\n-----------\n* Upwork won't load more than 5050 jobs on their website even if the website says there are more.\\\n  You can still get more than 5050 jobs, first scrape all the data, then keep updating the scraped data routinely.\n  For information on how to do this, see [usage](#cli-usage) or [documentation](#documentation) sections. \n* The post data for jobs is relative and its accuracy decreases as you go further into the past.\\\n  Minute precision for jobs posted up to an hour ago, hour for a day (2 hours ago), days up to a week ago (3 days ago) and so on.\\\n  This can also be worked around using the same method mentioned in point 1.\n\n\nLicense\n-------\nThis project is licensed under the [MIT license](https://github.com/Yazan-Sharaya/upwork_analysis/blob/main/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyazan-sharaya%2Fupwork_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyazan-sharaya%2Fupwork_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyazan-sharaya%2Fupwork_analysis/lists"}