https://github.com/mohamad27911/jobscraper
A Selenium-based job scraper with a FastAPI backend and a React TypeScript frontend, automating job listing extraction and display with filtering.
https://github.com/mohamad27911/jobscraper
python react selenium selenium-python web webscraping
Last synced: about 1 month ago
JSON representation
A Selenium-based job scraper with a FastAPI backend and a React TypeScript frontend, automating job listing extraction and display with filtering.
- Host: GitHub
- URL: https://github.com/mohamad27911/jobscraper
- Owner: mohamad27911
- Created: 2025-02-03T17:44:18.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-02-14T16:38:13.000Z (over 1 year ago)
- Last Synced: 2025-02-14T17:35:15.625Z (over 1 year ago)
- Topics: python, react, selenium, selenium-python, web, webscraping
- Homepage: https://job-scraping-mohamad27911s-projects.vercel.app/
- Size: 4.4 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Job Scraper using React + FastApi
This project is a job scraper API that collects remote job listings from multiple platforms and displays them in a React frontend. The backend is built using FastAPI, with Selenium handling web scraping. The frontend is developed using React, providing a seamless user experience for searching and viewing job listings.
## Full Project Link
You can view the full project here: [Job Scraper - Full Project](https://job-scraping-mohamad27911s-projects.vercel.app/)
## Table of Contents
* [Technologies Used](#technologies-used)
* [Getting Started](#getting-started)
* [Backend Setup](#backend-setup)
* [Frontend Setup](#frontend-setup)
* [API Endpoints](#api-endpoints)
* [Deployment](#deployment)
* [Contributing](#contributing)
## Technologies Used
* **Frontend**: React
* **Backend**: FastAPI
* **Web Scraping**: Selenium
* **Database**: None (JSON file storage for job listings)
* **Git**: For version control and pushing updates
## Getting Started
### Prerequisites
Before starting, ensure you have the following installed:
* Python 3.x
* Node.js and npm
* Chrome WebDriver (for Selenium)
* Git (for version control)
### Backend Setup
1. Clone the repository:
```bash
git clone https://github.com/mohamad27911/JobScraper.git
git checkout dev
```
2. Ensure you have Chrome WebDriver installed and properly set up.
4. Change directory to backend
```bash
cd backend
```
2. Create a virtual environment and install dependencies:
```bash
python -m venv env
env\Scripts\Activate
```
4. Change directory to src, where the main.py is located
```bash
cd src
```
5. Download requirements
``` bash
pip install -r requirements.txt
```
4. Run the FastAPI server:
```bash
uvicorn main:app --reload
```
The API should now be running at `http://localhost:8000`.
### Frontend Setup
1. Clone the frontend repository or navigate to the frontend directory:
```bash
cd frontend
```
2. Install the frontend dependencies:
```bash
npm install
```
3. Start the React development server:
```bash
npm start
```
The frontend should now be running at `http://localhost:5173`.
### Running the Application
* The frontend will display job listings that the backend API scrapes from various job boards.
* You can manually trigger scraping through the `/scrape-now` endpoint by clicking a button in the frontend.
* The job listings are saved as JSON files and updated periodically every 24 hours.
## API Endpoints
* **`/scrape-now`**: Manually triggers job scraping from various platforms.
* **Method**: `GET`
* **Response**: `{ "message": "Job scraping started manually" }`
* **`/`**: Returns a simple welcome message to the API.
* **Method**: `GET`
* **Response**: `{ "message": "Welcome to the Job Scraper API" }`
## Deployment
For deploying the backend:
* Choose a cloud provider or hosting solution that supports Python and selenium (I prefer Render).
* For deployment of the frontend, you can use services like Vercel or Netlify.
### Git Integration
The project includes a Git integration to automatically push updates of scraped job listings to your repository every time new data is scraped. Make sure your local environment is set up with Git and has access to the remote repository.
## Contributing
Feel free to fork the repository and create pull requests. Contributions are welcome!