{"id":18449546,"url":"https://github.com/kianoushamirpour/crawl_google_scholar_with_selenium_fastapi_mongodb","last_synced_at":"2026-04-16T04:01:13.560Z","repository":{"id":223808744,"uuid":"721010539","full_name":"KianoushAmirpour/Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB","owner":"KianoushAmirpour","description":"Crawl google scholar profiles with selenium, store the extracted data in the MongoDB and serve the queries with FastAPI.","archived":false,"fork":false,"pushed_at":"2024-01-06T12:57:09.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-17T20:22:17.909Z","etag":null,"topics":["crawler","fastapi","google-scholar","mongodb","python","selenium"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KianoushAmirpour.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-11-20T07:07:32.000Z","updated_at":"2024-02-22T05:30:16.000Z","dependencies_parsed_at":"2024-02-22T07:47:11.202Z","dependency_job_id":null,"html_url":"https://github.com/KianoushAmirpour/Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB","commit_stats":null,"previous_names":["kianoushamirpour/crawl_google_scholar_with_selenium_fastapi_mongodb"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KianoushAmirpour/Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FCrawl_Google_Scholar_with_Selenium_FastAPI_MongoDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FCrawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FCrawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FCrawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KianoushAmirpour","download_url":"https://codeload.github.com/KianoushAmirpour/Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KianoushAmirpour%2FCrawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31870516,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"online","status_checked_at":"2026-04-16T02:00:06.042Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","fastapi","google-scholar","mongodb","python","selenium"],"created_at":"2024-11-06T07:20:33.611Z","updated_at":"2026-04-16T04:01:13.514Z","avatar_url":"https://github.com/KianoushAmirpour.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB\n## The project is broken into below parts:\n1. Scrape the google scholar profiles of given scholars with selenium.\n2. Storing the extracted data to MongoDB which is hosted on the cloud.\n3. [Develop API endpoints using FastAPI to serve query results](#queries).\n\n## What is google scholar Profile?\nGoogle Scholar Profiles provide a simple way for authors to showcase their academic publications. You can check who is citing your articles, graph citations over time, and compute several citation metrics.\nfor example, you can see the Albert Einstein's profile in the below images:\n\n![Untitled1](https://github.com/KianoushAmirpour/Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/assets/112323618/64805099-24be-4e73-ae45-92a5c6e8e212)\n\n## Database and collections\nFor this project, we considerd having one database with two collections, namely `profiles` and `papers`.\n\nThe `profiles` collection includes documents in JSON format, each containing information such as name, university, total_citations, h_index, and i_10_index for each scholar.\n\nThe `papers` collection consists of documents in JSON format, each containing the scholar's name and a list of all published papers.\nFor each paper, details such as `title`, `author`s, `journal`, `number of citations`, and `the publication year` are stored.\n\n## Endpoints\nThe picture displays a list of all available endpoints. \n\n![Untitled](https://github.com/KianoushAmirpour/Crawl_Google_Scholar_with_Selenium_FastAPI_MongoDB/assets/112323618/5d4cb148-d3cf-408c-b517-ccba6a046809)\n\n## Queries\nSome of the queries used to retrieve data from database are shown below:\n\n### Recent_papers\nthis will return the last three papers for the scholar.\n```\n{\"$match\": {\"name\": scholar_name}},\n        {\"$project\": {\"_id\": 0, \"name\": 1,\n                      \"papers\": {\"$slice\": [\"$papers\", 3]}}}\n```\n\n### Best_papers\nThis one finds scholars with an h_index greater than 20 and then finds their papers which have more than 200 citations.\n```\n{\"$lookup\": {\n            \"from\": \"papers\",\n            \"localField\": \"name\",\n            \"foreignField\": \"name\",\n            \"as\": \"all_papers\"\n        }\n        },\n        {\"$unwind\": \"$all_papers\"},\n        {\"$unwind\": \"$all_papers.papers\"},\n        {\"$match\": {\"$and\": [{\"h_index\": {\"$gte\": 20}}, {\n            \"all_papers.papers.citation\": {\"$gte\": 200}}]}},\n        {\"$group\": {\n            \"_id\": {\"name\": \"$name\", \"h_index\": \"$h_index\"},\n            \"papers\": {\"$push\": \"$all_papers.papers\"}\n        }\n        },\n        {\"$project\": {\"_id\": 0, \"name\": \"$_id.name\",\n                      \"h_index\": \"$_id.h_index\", \"papers\": 1}}\n```\n## How to Use:\nWe used a local cloud provider for MongoDB which provided us a URI and We used localhost for FastAPI.\n### Before Running:\n1. Create a folder named `urls`, and within it, create a `.txt` file named `urls.txt` and add the links of profiles you want to crawl.\n2. Create a `.env` file and set (`DB_URI` and `DB_NAME`).\n\n### How to Run:\n1. Set up a virtual environment: `python -m venv venv`.\n2. Activate the environment: `venv\\Scripts\\activate.bat`.\n3. Install dependencies: `pip install -r requirements.txt`.\n4. Start FastAPI server: `uvicorn app.main:app --reload`.\n5. Run the crawler: `python gs-crawler\\crawler.py`.\n   - Reads URLs from `urls.txt` and crawls them one by one.\n   - Utilizes logging to capture exceptions and useful crawling details.\n   - Sends a POST request to the database for each link to store new records.\n6. Use MongoDB Compass and Postman to interact with your database.\n\n## Todo:\n- updating the database regularly\n- async crawling\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkianoushamirpour%2Fcrawl_google_scholar_with_selenium_fastapi_mongodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkianoushamirpour%2Fcrawl_google_scholar_with_selenium_fastapi_mongodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkianoushamirpour%2Fcrawl_google_scholar_with_selenium_fastapi_mongodb/lists"}