Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/igolaizola/twai
twai is a tool to scrape tweets and add a score to them using AI.
https://github.com/igolaizola/twai
ai elo-rating rating scrape twitter
Last synced: about 1 month ago
JSON representation
twai is a tool to scrape tweets and add a score to them using AI.
- Host: GitHub
- URL: https://github.com/igolaizola/twai
- Owner: igolaizola
- License: mit
- Created: 2024-05-29T18:56:47.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-06-03T19:15:36.000Z (7 months ago)
- Last Synced: 2024-11-26T23:34:55.883Z (about 1 month ago)
- Topics: ai, elo-rating, rating, scrape, twitter
- Language: Go
- Homepage:
- Size: 40 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE
Awesome Lists containing this project
README
# twai đĻđ¤
**twai** is a tool to scrape tweets and add a score to them using AI.
> đĸ Connect with us! Join our Telegram group for support and collaboration: [t.me/igohub](https://t.me/igohub)
## đ Features
- Scrape tweets from Twitter using a headless chrome browser
- Add a score from 1 to 10 to each tweet using AI
- Add a score using Elo rating system, comparing tweets to each other
## đĻ Installation
You can use the Golang binary to install **twai**:
```bash
go install github.com/igolaizola/twai/cmd/twai@latest
```
Or you can download the binary from the [releases](https://github.com/igolaizola/twai/releases)
## đ Requirements
### Twitter cookie
You need to capture the cookie from [Twitter](https://x.com/) website.
1. Go to https://x.com/home
2. Login if you are not already logged in
3. Open the developer tools (F12)
4. Go to the "Network" tab
5. Refresh the page
6. Click on the first request to https://x.com/home
7. Go to the "Request Headers"
8. Copy the "cookie" header and save it in a `cookie.txt` file
### AI model
You need to have an AI model to score the tweets.
You can use a local model via [Ollama](https://ollama.com/).
- Download and install Ollama tool
- Run `ollama run llama3` to download the model
- Set the `host` to `http://localhost:11434/v1` and the `model` to `llama3`
Or you can use an online model like [OpenAI](https://openai.com/).
- Get an API key from OpenAI
- Set the `model` to `gpt-3.5-turbo` and the `token` to your API key
## đšī¸ Usage
You can launch the twai command without any arguments or simply double-click on the binary to start the web UI.
The web UI will be available at the address printed in the console.
Alternatively, you can use the command-line interface to interact with the tool.
### Scrape tweets
Scrape tweets from a user or from the home page:
```bash
twai scrape --config scrape.yaml
```
```yaml
#scrape.yaml
debug: false #(bool): Debug mode
page: home #(string): Page to fetch (home or username)
n: 50 #(int): Number of tweets to fetch
followers: false #(bool): Fetch followers stats, it will take longer
output: scape.csv #(string): Output file
show-browser: false #(bool): Show browser
cookie-file: cookie.txt #(string): Cookie file
```
### Add simple score
Add a score to the tweets:
```bash
twai score --config score.yaml
```
```yaml
#score.yaml
debug: false #(bool): Debug mode
concurrency: 1 #(int): Number of concurrent requests to the AI
input: scrape.csv #(string): Input file (generated by scrape command)
output: score.csv #(string): Output file (csv)
prompt: "Rate the following tweet from 1 to 10 based on relevance, clarity, engagement, and impact. Only answer with a number." #(string): Prompt
model: llama3 #(string): AI model (e.g., llama3, gpt-3.5-turbo)
host: "http://localhost:11434/v1" #(string): AI endpoint host (not needed for openai)
token: "" #(string): Authorization token (required for openai)
```
### Add Elo score
Add a score to the tweets using the Elo rating system:
```bash
twai elo --config elo.yaml
```
```yaml
#elo.yaml
debug: false #(bool): Debug mode
concurrency: 1 #(int): Number of concurrent requests to the AI
input: scrape.csv #(string): Input file (generated by scrape command)
output: elo.csv #(string): Output file (csv)
iterations: 10 #(int): Number of iterations
prompt: "Which tweet is best? 1 or 2? Answer only with the number 1 or 2." #(string): Prompt
model: llama3 #(string): AI model (e.g., llama3, gpt-3.5-turbo)
host: "http://localhost:11434/v1" #(string): AI endpoint host (not needed for openai)
token: "" #(string): Authorization token (required for openai)
```
### Help
Launch `twai` with the `--help` flag to see all available commands and options:
```bash
twai --help
```
You can use the `--help` flag with any command to view available options:
```bash
twai scrape --help
```
## â ī¸ Disclaimer
The automation of Twitter accounts is a violation of their Terms of Service and will result in your account(s) being terminated.
Read about Twitter Terms of Service and Community Guidelines.
twai was written as a proof of concept and the code has been released for educational purposes only. The authors are released of any liabilities which your usage may entail.
## đ Support
If you have found my code helpful, please give the repository a star â
Additionally, if you would like to support my late-night coding efforts and the coffee that keeps me going, I would greatly appreciate a donation.
You can invite me for a coffee at ko-fi (0% fees):
[](https://ko-fi.com/igolaizola)
Or at buymeacoffee:
[](https://buymeacoffee.com/igolaizola)
Donate to my PayPal:
[paypal.me/igolaizola](https://www.paypal.me/igolaizola)
Or sponsor me on GitHub:
[github.com/sponsors/igolaizola](https://github.com/sponsors/igolaizola)
Thanks for your support!