https://github.com/jaketurner616/latest-useragent-api
https://github.com/jaketurner616/latest-useragent-api
Last synced: 3 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/jaketurner616/latest-useragent-api
- Owner: JakeTurner616
- License: gpl-3.0
- Created: 2023-12-30T22:31:51.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2023-12-30T22:41:55.000Z (over 1 year ago)
- Last Synced: 2025-01-07T17:46:42.433Z (5 months ago)
- Language: Python
- Size: 16.6 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# User Agent Scraper
Latest-useragent-api is a flask api that uses Selenium and BeautifulSoup to dynamically fetch the most up-to-date user agents from "https://www.useragents.me/" sorted by popularity. It offers two convenient routes for accessing the scraped data:
1. **JSON Route** (`/json`): Delivers the user agent data in JSON format as a response.
2. **TSV Route** (`/tsv`): Presents the user agent data in TSV (Tab-Separated Values) format as a direct download.
## Features
- Utilizes Selenium for navigating a headless Chrome browser, ensuring accurate and current user agent data.
- Sorts user agents by popularity, providing insights into commonly used configurations.
- Ignores entries containing the text "Check out smartproxy’s residential rotating proxy servers."
- Offers data retrieval in both JSON and downloadable TSV formats.
## Purpose
This user agent scraping api serves has the potential to serve a crucial purpose in the realm of web scraping and automated interactions with web servers. By retrieving the latest and most popular user agents from "https://www.useragents.me/", it provides a valuable resource for programs engaged in scraping activities that necessitate up-to-date user agents. Automated programs can utilize the latest user agents to avoid triggering basic bot detection mechanisms implemented by webservers. This proactive approach helps prevent the recording of anomalous patterns in browser usage, especially in scenarios where automated scraping pipelines may inadvertently use outdated or easily detectable user agents, thereby minimizing the risk of triggering primitive bot detection and ensuring a smoother interaction between automated systems and web servers.