https://github.com/dbrennand/twitter-stream-bot-data-gatherer
An application to watch the Twitter stream and send accounts to the Botometer API for analysis. The results are stored in a SQLite database.
https://github.com/dbrennand/twitter-stream-bot-data-gatherer
bot-detection botometer data-gathering twitter-api twitter-bot-detection twitter-streaming-api
Last synced: about 2 months ago
JSON representation
An application to watch the Twitter stream and send accounts to the Botometer API for analysis. The results are stored in a SQLite database.
- Host: GitHub
- URL: https://github.com/dbrennand/twitter-stream-bot-data-gatherer
- Owner: dbrennand
- License: mit
- Created: 2022-02-26T10:46:58.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2022-02-28T20:02:13.000Z (over 3 years ago)
- Last Synced: 2025-04-02T22:39:04.704Z (2 months ago)
- Topics: bot-detection, botometer, data-gathering, twitter-api, twitter-bot-detection, twitter-streaming-api
- Language: Python
- Homepage:
- Size: 35.2 KB
- Stars: 5
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# twitter-stream-bot-data-gatherer
An application to watch the Twitter stream and send accounts to the [Botometer API](https://botometer.osome.iu.edu/) for analysis. The results are stored in a SQLite database.
## Prerequisites
Before using this application, credentials for the following are required:
* [The Twitter API](https://developer.twitter.com/en/docs)
* [The Botometer API](https://rapidapi.com/OSoMe/api/botometer-pro)
### Twitter API Credentials
The application connects to the Twitter stream API to query Twitter user's to Botometer for analysis.
To get access to the Twitter API, a developer account is needed. You can apply for one [here](https://developer.twitter.com/en/apply-for-access) and find more information about the application process [here](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api).
Once you have a developer account, go to the [developer portal](https://developer.twitter.com/en/portal/dashboard) to create an application.
When creating the application, provide a unique *name* and follow the instructions. The application's API key and secret will be shown (aka. `consumer_key` and `consumer_secret`). Furthermore, you will need to generate an access token and access token secret to use the Twitter stream API. Go to your application, click **Keys and tokens** > **Access Token and Secret** > **Generate**. Store the credentials somewhere safe as they will be needed later.
### Botometer API Credentials
The application uses the [Botometer API](https://rapidapi.com/OSoMe/api/botometer-pro/details) to collect bot scores for Twitter users.
To get access to the Botometer API, a RapidAPI account is needed. You can create one [here](https://rapidapi.com/auth/sign-up).
Once you have a RapidAPI account, go to the Botometer API [pricing](https://rapidapi.com/OSoMe/api/botometer-pro/pricing) page and subscribe to the *Basic* or *Pro* plan.
> [!NOTE]
>
> The *Pro* plan requires a credit or debit card to be added to the RapidAPI account.Once you have chosen a plan, go to the RapidAPI [developer dashboard](https://rapidapi.com/developer/apps). Then, go to security page for the `default-application`. The application key (API key) will be shown. Store it somewhere safe as it will be needed later.
Further details on accessing the Botometer API can be found [here](https://github.com/IUNetSci/botometer-python#rapidapi-and-twitter-access-details).
## Dependencies
Install the application's dependencies using one of the options below:
```bash
# Poetry
poetry install --no-dev
# Pip
pip install -r requirements.txt
# Or use Docker!
# Please refer to the heading below
```## Usage
```
$ python twitter-stream-bot-data-gatherer/main.py -husage: main.py [-h] [-t TRACK] [-f DATABASE_NAME] [-d]
rapidapi_key twitter_app_authAn application to watch the Twitter stream and send accounts to the Botometer
API for analysis. The results are stored in a SQLite database.positional arguments:
rapidapi_key Botometer Rapid API key.
twitter_app_auth Twitter application credentials.optional arguments:
-h, --help show this help message and exit
-t TRACK, --track TRACK
A hashtag to track. Can be specified more than once.
-f DATABASE_NAME, --database_name DATABASE_NAME
Name of the database file. Defaults to: twitter-
stream-bot-data-gatherer.
-d, --debug Enable debug messages.
```### Tracking a Single Hashtag
```bash
python twitter-stream-bot-data-gatherer/main.py \
'{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
--track '#StandWithUkriane'
```### Tracking Multiple Hashtags
```bash
python twitter-stream-bot-data-gatherer/main.py \
'{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
-t '#StandWithUkriane' \
-t '#RefugeesWelcome' \
-t '#Ukrania'
```### Modifying the SQLite Database Name
```bash
python twitter-stream-bot-data-gatherer/main.py \
'{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
-t '#StandWithUkriane' \
# The database will be named 'twitter-db.db'
-f 'twitter-db'
```### Docker 🐋
Build the container image:
```bash
docker build -t twitter-stream-bot-data-gatherer:v0.1.0 .
```Run the application and persist the SQLite database:
```bash
docker run -it --name twitter-stream-bot-data-gatherer \
-v absolute/path/to/store/db:/usr/src/app/db \
twitter-stream-bot-data-gatherer:v0.1.0 \
'{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
--track '#StandWithUkriane'
```## SQLite Database Structure
The SQLite database will contain one table called `data`. Within the table will be three columns:
1. `screen_name`: The screen name of the Twitter account.
2. `status_json`: The full JSON response of the [Tweet Object (aka. Status)](https://developer.twitter.com/en/docs/twitter-api/v1/data-dictionary/object-model/tweet) from the Twitter stream.
3. `botometer_json`: The full JSON response from the Botometer API.
## My Related Work
* [dbrennand/Final-Year-Project](https://github.com/dbrennand/Final-Year-Project) - An application that generates a report to identify potential bots that a Twitter user is following.
## Acknowledgments
This application would not be possible without the work of the [Observatory on Social Media at Indiana University](https://osome.iu.edu/) who created Botometer. Please see the [Botometer FAQ](https://botometer.osome.iu.edu/faq) page for further information on Botometer.
## Authors -- Contributors
* [**dbrennand**](https://github.com/dbrennand) - *Author*
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) for details.