{"id":29715674,"url":"https://github.com/afa7789/pyscrapper","last_synced_at":"2026-06-08T16:33:52.698Z","repository":{"id":298345562,"uuid":"999650127","full_name":"afa7789/pyscrapper","owner":"afa7789","description":"Exemplo generalizado de um scrapper que analisa marketplaces, p/ um amigo.","archived":false,"fork":false,"pushed_at":"2025-09-16T14:59:21.000Z","size":2788,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-16T16:44:04.871Z","etag":null,"topics":["bot-telegram","python","scrapper"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/afa7789.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-10T15:16:52.000Z","updated_at":"2025-09-16T14:59:25.000Z","dependencies_parsed_at":"2025-06-24T23:23:11.257Z","dependency_job_id":"ffd8cf67-062e-4429-bfcc-c119fc30d514","html_url":"https://github.com/afa7789/pyscrapper","commit_stats":null,"previous_names":["afa7789/pyscrapper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/afa7789/pyscrapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afa7789%2Fpyscrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afa7789%2Fpyscrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afa7789%2Fpyscrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afa7789%2Fpyscrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/afa7789","download_url":"https://codeload.github.com/afa7789/pyscrapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afa7789%2Fpyscrapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34071655,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot-telegram","python","scrapper"],"created_at":"2025-07-24T05:38:51.234Z","updated_at":"2026-06-08T16:33:52.681Z","avatar_url":"https://github.com/afa7789.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# 🛒 Marketplace Scraper Program\n\n\u003e **Disclaimer:** This program was developed **for educational purposes only**. It is not intended for commercial use or profit. The main goal is to demonstrate a technology to friends who are currently learning.\nThis project is a marketplace scraper built for educational purposes.\nIt demonstrates how to integrate a desktop GUI (using Tkinter) with a web version featuring a server and an admin panel. \nThe goal is to provide insights into both desktop and web application development while experimenting with various technology elements.\n\nThe scraper in this project (implemented in files like scraper.py and scraper_cloudflare.py) is designed to:\n\n- Fetch Marketplace Data: It sends HTTP requests to target marketplace websites to retrieve their data.\n- Parse Content: Using libraries such as BeautifulSoup it parses the HTML response to extract relevant information like listings, prices, and other details.\n- Handle Different Blocking Modes: The use of a separate scraper (scraper_cloudflare.py) indicates that one version is specifically tuned to bypass or handle Cloudflare's protections if the standard approach fails.\n- Integration with Other Modules: The scraper is part of a larger system that includes a GUI, a server with an admin panel, and even a Telegram bot for sending notifications. This lets users view data in a desktop application, on a web dashboard, or receive alerts in real-time.\n\nOverall, the scraper automates the process of collecting marketplace data, which can then be displayed or processed by other parts of the application.\n\n---\n\n## 📁 Project Structure\n\n```bash\n.\n├── gui.py                 # Graphical user interface (Tkinter)\n├── main.py                # Entry point (integrates all modules)\n├── monitor.py             # Background monitoring logic\n├── requirements.txt       # Requirements to install python packages easier\n├── scraper.py             # MarketRoxo scraping .\n├── scraper_cloudflare.py  # Scraping but cloudflare does not block me.\n├── server.py              # Server, to host in a VPS instead of GUI locally\n└── telegram_bot.py        # Sends messages via Telegram\n```\n\n---\n\n## ⚙️ How to Generate an Executable (Binary)\n\n### using .env\n\ndo not forget to fill the .env it will crash if running `server.py`\n\nremember to fill the .env if you don't want to have to write the same information everytime you open the app.\n\n```bash\ncp .env.example .env\n```\n\n### 1. Set up a Virtual Environment (Optional but recommended)\n\nUse newer python (3.11)\n\n```bash\npython3 -m venv ./venv_project\nsource ./venv_project/bin/activate\n```\n\u003c!-- python3 -m venv ./venv_otavio --\u003e\n\n### 2. Install Dependencies\n\n```bash\npip install requests beautifulsoup4 tkinter pyinstaller\npip install gunicorn\npip freeze \u003e requirements.txt\npip install -r requirements.txt\n```\n\n## SERVER VERSION\n\nfill admin values\n```bash\npip install flask python-dotenv requests beautifulsoup4 # missing\npython3 server.py\n# in prod:\ngunicorn -w 4 -b 0.0.0.0:5000 server:app\n```\n\n![alt text](image_admin_panel_web.png)\n\n\n## Format code!\n```bash\n    pip install autopep8\n    autopep8 --in-place --recursive .\n```\n\n# Docker for local testing\n\n```\n    docker build -t my-python-app .\n    docker run -p 5000:5000 -v $(pwd):/app my-python-app\n    docker run -p 5000:5000 --env-file .env my-python-app\n    docker run -p 5000:5000 -v $(pwd):/app -w /app my-python-app python server.py --reload\n```\n## Pre-requisites\n\nEnsure that you have the following:\n- A VPS (Virtual Private Server)\n- A registered domain\n- A residential proxy service (e.g., Royal Proxy or an alternative)\n- DNS configuration to point the domain to your VPS\n- Python installed\n- NGINX installed\n- Certbot configured for SSL certificate management\n- NGINX properly configured\n- Python dependencies installed\n- A correctly filled .env file\n\n## :)\n\n```bash\nfind . -maxdepth 1 -type f -name \"*.py\" -exec wc -l {} + | sort -n | awk '{print $2 \": \" $1 \" lines\"}'\n./emoji_sorter.py: 58 lines\n./telegram_bot.py: 92 lines\n./small_test_scraper.py: 111 lines\n./request_stats.py: 202 lines\n./logging_config.py: 287 lines\n./scraper_cloudflare.py: 448 lines\n./monitor.py: 478 lines\n./server.py: 716 lines\ntotal: 2392 lines\n```\n\n# Licensing \u0026 more\n\nNo code of conduct\n\nNo licensing","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafa7789%2Fpyscrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fafa7789%2Fpyscrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafa7789%2Fpyscrapper/lists"}