{"id":19331830,"url":"https://github.com/hitiksha-patel/scrapebot","last_synced_at":"2026-04-30T16:31:56.691Z","repository":{"id":247007835,"uuid":"823410973","full_name":"hitiksha-patel/scrapebot","owner":"hitiksha-patel","description":"This project is a Python-based web application using the Falcon framework to retrieve and display details about Android apps from the Aptoide marketplace through web scraping. It fetches information such as app name, version, downloads, release date, and description.","archived":false,"fork":false,"pushed_at":"2024-07-06T01:40:40.000Z","size":585,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-21T11:52:42.541Z","etag":null,"topics":["css","falcon","falcon-framework","html","javascript","logging","python","scraping","test-driven-development"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hitiksha-patel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-03T01:49:58.000Z","updated_at":"2024-09-18T17:59:38.000Z","dependencies_parsed_at":"2024-07-06T05:07:40.456Z","dependency_job_id":null,"html_url":"https://github.com/hitiksha-patel/scrapebot","commit_stats":null,"previous_names":["hitiksha-patel/scrapebot"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hitiksha-patel/scrapebot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hitiksha-patel%2Fscrapebot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hitiksha-patel%2Fscrapebot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hitiksha-patel%2Fscrapebot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hitiksha-patel%2Fscrapebot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hitiksha-patel","download_url":"https://codeload.github.com/hitiksha-patel/scrapebot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hitiksha-patel%2Fscrapebot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32470879,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["css","falcon","falcon-framework","html","javascript","logging","python","scraping","test-driven-development"],"created_at":"2024-11-10T02:42:35.605Z","updated_at":"2026-04-30T16:31:56.676Z","avatar_url":"https://github.com/hitiksha-patel.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Aptoide App Information Web Scraper - ScrapeBot\n\n## Overview\n\nThis project is a web application built with Python and Falcon framework, designed to retrieve and display information about Android applications available on the Aptoide marketplace. The application uses web scraping techniques to fetch details like app name, version, downloads, release date, and description from the Aptoide platform.\n\n### Features\n\n- **Web Scraping**: Retrieves app details from Aptoide by parsing HTML content.\n- **API Endpoint**: Accepts a POST request with an Aptoide app URL to fetch and return app information as JSON.\n- **Error Handling**: Handles various error scenarios such as missing URL, invalid URL format, and app not found on Aptoide.\n- **Logging**: Logs events and errors to `Scrapebot.log` for debugging and monitoring.\n- **Unit Testing**: Includes unit tests to verify functionality using the `unittest` framework and mock data.\n\n## Project Structure\n\nThe project consists of the following main files and directories:\n\n- **`run.py`**: Initializes a WSGI server to run the Falcon application.\n- **`app.py`**: Contains Falcon resources (`AppInfoResource` and `IndexResource`) for handling HTTP requests.\n- **`test_app.py`**: Unit tests for `AppInfoResource` using `unittest` and `falcon.testing.TestClient`.\n- **`mypy.ini`**: Configuration file for `mypy` type checker.\n- **`typings/falcon_stub.pyi`**: Stub file defining types for Falcon framework objects.\n\n## Installation and Usage\n\n### Prerequisites\n\n- Python 3.x\n- Dependencies (`falcon`, `requests`, `beautifulsoup4`) installed via pip.\n\n### Installation\n\n1. Unzip the Project:\n\n   - Unzip the downloaded file (`data_theorem_test.zip`) to a directory of your choice on your local machine.\n\n2. Create and activate a virtual environment:\n\n   On macOS and Linux:\n   ```bash\n   python3 -m venv venv\n   source venv/bin/activate\n   ```\n\n   On Windows:\n   ```bash\n   python -m venv venv\n   .\\venv\\Scripts\\activate\n   ```\n\n3. Install dependencies:\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n   \n### Running the Application\n\nTo start the Falcon web server locate to ScrapeBot:\n\n```bash\npython run.py\n```\n\nThe application will run locally at `http://localhost:8000`.\n\n### Usage\n\n1. **Home Page**: Access the home page at `http://localhost:8000/`. You can input an Aptoide app URL in a form and submit it.\n   \n2. **Get App Info**: Post a request to `/get-app-info` endpoint with a URL parameter (`url=https://en.aptoide.com/app`) to retrieve JSON formatted information about the app.\n\n\n## Testing\n\n1. **Type Checking with MyPy**: Run mypy using `mypy`:\n\n```bash\nmypy app.py run.py\n```\n\n2. **Unit test**: Run unit tests using `unittest`:\n\n```bash\npython test_app.py\n```\n\nThe tests cover:\n- Successful retrieval of app information.\n- Handling missing URL parameter.\n- Handling invalid and non-existent app URLs.\n\n## Notes\n\n- Ensure Python type annotations are respected throughout the codebase to maintain clarity and support type checking with `mypy`.\n- Modify logging configurations (`logging.basicConfig` in `app.py`) as needed for your local environments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhitiksha-patel%2Fscrapebot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhitiksha-patel%2Fscrapebot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhitiksha-patel%2Fscrapebot/lists"}