https://github.com/aliosmandev/python-101
A simple Twitter scraper built with FastAPI and Selenium. This project allows you to scrape tweets from a specified Twitter user, retrieving the profile title and posts in JSON format.
https://github.com/aliosmandev/python-101
fastapi python scraping selenium swagger twitter
Last synced: about 2 months ago
JSON representation
A simple Twitter scraper built with FastAPI and Selenium. This project allows you to scrape tweets from a specified Twitter user, retrieving the profile title and posts in JSON format.
- Host: GitHub
- URL: https://github.com/aliosmandev/python-101
- Owner: aliosmandev
- Created: 2024-06-03T19:53:27.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-06-03T21:02:44.000Z (about 2 years ago)
- Last Synced: 2025-02-13T21:19:07.429Z (over 1 year ago)
- Topics: fastapi, python, scraping, selenium, swagger, twitter
- Language: Python
- Homepage:
- Size: 3.91 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Python Twitter Scrape
## Overview
This project is a simple Twitter scraper built with FastAPI and Selenium. It allows you to scrape tweets from a specified Twitter user.
## Features
- Scrape tweets from a Twitter user.
- Retrieve the profile title and posts.
- Asynchronous processing with FastAPI.
- JSON response with scraped data.
## Requirements
- Python 3.8+
- Selenium
- FastAPI
- Uvicorn
- WebDriver (e.g., ChromeDriver)
## Installation
### 1. Clone the repository:
```sh
git clone https://github.com/osmandlsmn/python-101.git
cd twitter-scrape
```
### 2. Create and activate a virtual environment:
```sh
python -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`
```
### 3. Install the requirements:
```sh
pip install -r requirements.txt
```
### 4. Run the application:
```sh
make run
```
#### Alternatively, you can run the application directly with Uvicorn:
```sh
uvicorn src.main:app --host 0.0.0.0 --port 3000 --reload
````
### Project Structure
```
twitter-scrape/
├── src/
│ ├── main.py
│ └── utils/
│ └── selectors.py
├── requirements.txt
├── Makefile
└── README.md
```