{"id":29261194,"url":"https://github.com/pointer2alvee/fk-web-crawler","last_synced_at":"2025-07-04T08:07:03.238Z","repository":{"id":300554539,"uuid":"1006459828","full_name":"pointer2Alvee/fk-web-crawler","owner":"pointer2Alvee","description":"Take home assignment for filerskeepers","archived":false,"fork":false,"pushed_at":"2025-06-27T19:11:43.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-27T19:22:15.943Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pointer2Alvee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-22T10:19:29.000Z","updated_at":"2025-06-27T19:11:46.000Z","dependencies_parsed_at":"2025-06-27T19:22:21.372Z","dependency_job_id":"1a9a77dd-491a-4630-b43f-6acc0e57b29d","html_url":"https://github.com/pointer2Alvee/fk-web-crawler","commit_stats":null,"previous_names":["pointer2alvee/fk-web-crawler-assignment","pointer2alvee/fk-web-crawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/pointer2Alvee/fk-web-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointer2Alvee%2Ffk-web-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointer2Alvee%2Ffk-web-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointer2Alvee%2Ffk-web-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointer2Alvee%2Ffk-web-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pointer2Alvee","download_url":"https://codeload.github.com/pointer2Alvee/fk-web-crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointer2Alvee%2Ffk-web-crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263472277,"owners_count":23471812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-04T08:07:00.862Z","updated_at":"2025-07-04T08:07:03.226Z","avatar_url":"https://github.com/pointer2Alvee.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/fastapi_1.png\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n## 📚 fk-web-crawler: \n\n#### 📌 Summary \nImplemented a Web Crawler system using scrapy and stored the scraped data and change logs in MongoDB with features like Change Detection, Scheduler, Report generation and REST API Features using FastAPI\n\n#### 🧠 Summary \nA fully-featured Python-based web crawler that:\n- Crawls all pages and scrapes book data from `books.toscrape.com`\n- Stores scraped book data including metadata and raw html snapshot to MongoDB Atlas\n- Detects changes of existing books using fingerprinting strategy\n- Logs changes and inserts to DB\n- Able to resume from the last successful crawl \n- Schedules daily updates using APScheduler\n- Serves data through a FastAPI-secured RESTful API with rate limiting and authentication\n\n#### 🚀 Features\n- ✅ Scrapy-powered web crawler\n- ✅ MongoDB Atlas integration with deduplication\n- ✅ Hash/Fingerprint-based change detection\n- ✅ Change logging and raw HTML snapshot\n- ✅ APScheduler-powered daily job\n- ✅ Daily change reports in JSON\n- ✅ RESTful FastAPI server using FastAPI:\n  - `/books` with filtering, pagination, sorting\n  - `/books/{book_id}` for book details\n  - `/changes` to get recent logs\n- ✅ API Key Authentication\n- ✅ Rate limiting (100 req/hr per IP)\n- ✅ OpenAPI (Swagger) docs\n\n---\n\n#### 📁 Project Structure\n\n```\nFK-CRAWLER/\n│\n├── api/                            # FastAPI server\n│   ├── auth/                      \n│   |   ├── security.py             # API-KEY Authentication\n│   ├── models/                     # Pydantic models \n│   |   ├── book.py                 \n│   |   ├── change.py               \n│   |   ├── schemas.py              # Serialization for book and logs\n│   │   │\n│   ├── routes/                     # API endpoints\n│   |   ├── __init__.py              \n│   |   ├── books.py                 \n│   |   ├── changes.py\n│   ├── utils/                     # API endpoints\n│   |   ├── rate_limiter.py\n│   ├── __init__.py                  \n│   ├── main_test.py                # For testing\n│   └── main.py                     # main api file\n│\n├── crawler/                        # Web scraping logic\n│   ├── fkcrawling/\n│   |   ├── spiders/                # Scrapy spiders\n│   |   |   ├── __init__.py         \n│   |   |   ├── book_schema.py      # Pydantic model\n│   |   |   ├── crawling_spider.py  # Web Crawler\n│   |   |   └── mongodb_client.py   # MongoDB connection\n│   |   ├── __init__.py             \n│   |   ├── items.py                \n│   |   ├── middlewares.py          \n│   |   ├── pipelines.py            \n│   └── ├── settings.py             # Scrapy config\n│ \n├── scheduler/                      # Daily job scheduler\n│   ├── daily_scheduler.py\n│   └── crawler_runner.py\n│\n├── utilities/                      # Helper utilities\n│   ├── logs/\n│   |   ├── activity.log            # activity logging\n│   ├── reports/\n│   |   ├── report.json             # Generated Report\n│   ├── assets/\n│   |   ├── images/                 # images\n│   ├── generate_report.py          # Daily changes report\n│   └── log_config.py               # Log setup\n│\n├── tests/                          # Unit \u0026 integration \n│   ├── __init__.py   \n│   ├── test_db.py   \n│   └── test_crawler.py \n├── .env                            # Secure API_KEY and mongoDB URI\n├── .gitignore\n├── requirements.txt                # required packages\n└── README.md                       # This file\n```\n---\n\n## 🔧 Setup Instructions\n\n### 📦 Requirements\n\n- Python 3.10+\n- VSCode\n- MongoDB Atlas account\n\n\n### 📁 1. Clone the Repository\n\n```bash\ngit clone https://github.com/pointer2Alvee/fk-web-crawler.git\ncd fk-crawler\n```\n\n### 📁 2. Install Dependencies\n\n```bash\npip install -r requirements.txt\n```\n\n### ⚙️ 3. Create `.env` File\n\nCreate a `.env` file at the root:\n\n```\nMONGODB_URI= \u003cmongodb+srv://\u003cuser\u003e:\u003cpass\u003e@cluster.mongodb.net/\u003e\nAPI_KEY= \u003cSet your own key\u003e\n```\n\n\u003e ✅ `.env` is automatically loaded using `dotenv`.\n\n---\n\n## 🕷️ Run Crawler\n\n```bash\ncd crawler/fkcrawling\nscrapy crawl fkcrawler\n```\n\n- Inserts newly scraped books to MongoDB collection \"books\"\n- Deduplication and Logs changes (if any) to MongoDB collection \"change_log\"\n- Logs output to `/logs/activity.log`\n\n---\n\n## 🗓️ Run Scheduler - Runs Cralwer + Change Report Generator\n\nIn daily_scheduler.py :- \nhour=13,\nminute=15\n- Put hour and minute at the time you want to schedule the scheduler\n- Here the scheduler will run daily at 13:15 or 1:15 PM\n\n\n```bash\ncd scheduler\npython daily_scheduler.py\n```\n\n- Crawls every day using APScheduler\n- Detects new books or changes\n- Logs them in MongoDB and filesystem\n- Generates Daily change Report in JSON\n\n---\n\n## 🧪 Run FastAPI Server\n\n```bash\ncd api\nuvicorn main:app --reload\n```\n\n- API is hosted at `http://127.0.0.1:8000/`\n- Swagger docs: `http://127.0.0.1:8000/docs`\n\n---\n\n## 🔐 API Key Usage\n\nAll endpoints are protected via API key.\n\n### Headers:\n\n```\nFKCRAWLER-API-KEY: \u003cSet your own key\u003e\n```\n\n---\n\n## 📂 API Endpoints\n\n| Endpoint          | Method | Description                              |\n|-------------------|--------|------------------------------------------|\n| `/books`          | GET    | Get all books (filter, sort, paginate)   |\n| `/books/{id}`     | GET    | Get book by MongoDB ObjectId             |\n| `/changes`        | GET    | Get recent changes                       |\n| `/docs`           | GET    | Swagger UI (OpenAPI spec)                |\n\n---\n\n## 📤 Daily Report Output\n\nOn successful run, you'll get:\n\n```bash\n/reports/\n├── change_report_YYYY-MM-DD.json\n```\n\nIncludes:\n- New insertions\n- Fields changed\n- Source URLs and timestamps\n\n---\n\n## 🧪 Testing\n\nUnit and integration tests will be added in `/tests/`.\n\nImplemented with `pytest` for:\n- DB operations\n- Crawling output\n\nMake sure you're in the root of the project and run:\n\n```bash\npytest tests/\n```\n\nDemo Output summary :- \n```\n===================== test session starts =================\ncollected 2 items\n\ntests/test_crawler.py ....                          [66%]\ntests/test_db.py .                                  [83%]\n====================== 2 passed in 2.31s ==================\n\n```\n\n---\n\n\n## Demonestration - \n\n### - mongoDB\n---\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/mdb_1.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/mdb_2.PNG\" alt=\"Image 1\" style=\"width: 50%; margin: 10px;\"\u003e\n   \u003cimg src=\"utilities/assets/images/mdb_3.PNG\" alt=\"Image 1\" style=\"width: 50%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n\n### - log\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/log.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n### - report\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/report.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n### - fastapi\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/fastapi_1.png\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n#### - fastapi - GET/Books\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/fastapi_2.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/fastapi_3.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n#### - fastapi - GET/{Book_id} \u0026 GET/{change_log}\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/fastapi_4.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\u003cdiv style=\"display: flex; justify-content: space-around; align-items: center;\"\u003e\n  \u003cimg src=\"utilities/assets/images/fastapi_5.PNG\" alt=\"Image 1\" style=\"width: 100%; margin: 10px;\"\u003e\n\u003c/div\u003e\n\n## 💡 Sample MongoDB Document\n\n**books Document Structure**\n```json\n{\n  \"_id\": ObjectId(\"123...\"),\n  \"name\": \"A Light in the Attic\",\n  \"description\" : \"It's hard to..\",\n  \"category\" : \"Poetry\",\n  \"price_with_tax\": 12.99,\n  \"price_with_out_tax\": 12.99,\n  \"availability\": \"22\",\n  \"review\" : 0,\n  \"cover_image_url\" : \"https://books.toscrape.com/../fe72aea293c.jpg\",\n  \"rating\": 3,\n  \"crawl_timestamp\": \"2025-06-27T10:00:00Z\",\n  \"source_url\": \"https://books.toscrape.com/catalogue/.../index.html\",\n  \"raw_html\": \"\u003chtml\u003e...\u003c/html\u003e\",\n  \"fingerprint\": \"abc123...\",\n\n}\n```\n**change_log Document Structure**\n\n```json\n{\n  \"_id\": ObjectId(\"123...\"),\n  \"source_url\": \"https://books.toscrape.com/catalogue/.../index.html\",\n  \"name\": \"A Light in the Attic\",\n  \"timestamp\": \"2025-06-27T10:00:00Z\",\n  \"changes\" : Object,\n}\n```\n---\n\n## 🧾 Deliverables Checklist (from PDF ✅)\n\n| Requirement                               | Status     |\n|-------------------------------------------|------------|\n| ✅ Crawler using Scrapy                  | Done       |\n| ✅ Scheduler with change detection       | Done       |\n| ✅ Change log storage/collection         | Done       |\n| ✅ FastAPI server                        | Done       |\n| ✅ API key + rate limiting               | Done       |\n| ✅ Swagger UI                            | Done       |\n| ✅ `.env` support                        | Done       |\n| ✅ Daily reports (JSON + CSV)            | Done       |\n| ✅ Screenshot/logs of scheduler/crawler  | ✔️ See `/logs` |\n\n---\n\n## 📬 Postman / Swagger UI\n\nUse [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs) to interactively test all endpoints.\n\n---\n\n## 🧠 Future Improvements\n- Add email alerts for major changes\n- Add export formats: CSV, PDF, Excel\n\n---\n\n## 🧑‍💻 Author\n\n**Alvee**  \n📧 pointer2alvee@gmail.com\n🔗 [GitHub](https://github.com/pointer2Alvee)\n\n---\n\n### 🙏 Acknowledgements\n- Open-source contributors and net\n- Youtube videos :- \n  * [1](https://www.youtube.com/watch?v=mBoX_JCKZTE) , [2](https://www.youtube.com/watch?v=GogxAQ2JP4A), [3](https://www.youtube.com/watch?v=rvFsGRvj9jo)\n---\n\n## 📄 License\nMIT License – feel free to use, improve, and contribute!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpointer2alvee%2Ffk-web-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpointer2alvee%2Ffk-web-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpointer2alvee%2Ffk-web-crawler/lists"}