---

## Types of Websites We Can Scrape

The **Web Scraper API** is designed for scraping content from **static websites** where data is readily available in the HTML structure. Here are examples of websites it works well with:

- **Blogs and News Sites:** Extract article content, headlines, and publication dates.
- **Documentation Websites:** Scrape text from user manuals, API docs, and help pages.
- **Product Pages:** Extract product names, descriptions, and prices from static e-commerce websites.
- **Research Papers or Journals:** Extract titles, abstracts, and references from academic papers that are pre-rendered in HTML.
- **Company Websites:** Scrape static information from company pages like about sections, team details, and contact information.

The tool is ideal for websites where the content is delivered without requiring JavaScript to render dynamically. For dynamic sites that require JavaScript execution, other scraping methods would be needed.

---

## Installation

1. Clone the repository:
```bash
https://github.com/Abhimanyu-Gaurav/Web-Scraper-API

2. Navigate to the project directory:
```bash
cd web-scraper-api

3. Set up a virtual environment (optional but recommended):
```bash
python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`

4. Install the required dependencies:
```bash
pip install -r requirements.txt

---

## How to Use

1. Run the FastAPI server:
```bash
uvicorn main:app --reload

2. Open your browser (Safari, Chrome, Brave) and enter the URL:
```bash
http://localhost:8000/docs#

3. This should display the Swagger UI for your FastAPI application.

4. Test the /scrape endpoint directly from the Swagger UI to ensure it is working and accessible.

5. Use the POST method (/scrape) and provide a JSON in the body like this:
- Click on the "Try it out" button.
- Provide a JSON in the request body:
```json
{
"url": "https://timesofindia.indiatimes.com/"
}

6. Click on the "Execute" button to send the request.

7. You should see the scraped data in the response section if the request is successful.

### Send a POST request using API clients:
1. Using Postman:
- Open Postman and click the "New" button to create a new request.
- Set the request type to POST from the dropdown menu next to the URL field.
- Enter the URL in the URL field:
```bash
http://localhost:8000/scrape/

- Go to the "Body" tab.
- Select the "raw" option and choose "JSON" from the dropdown.
- Paste the following JSON into the body:
```json
{
"url": "https://timesofindia.indiatimes.com/"
}

- Click the "Send" button to execute the request.
- You should see the scraped data in the response section if the request is successful.

2. Using cURL:
- Open your terminal and run:
```bash
curl -X POST "http://localhost:8000/scrape/" -H "Content-Type: application/json" -d '{"url": "https://timesofindia.indiatimes.com/"}'

- search: The term you want to search (e.g., business name or type).
- total: The number of listings to retrieve (if available).

---

## License

- This project is licensed under the MIT License - see the [License](License) file for details.

---