{"id":19491306,"url":"https://github.com/faisal-fida/news-scraper-api","last_synced_at":"2026-04-19T19:33:22.385Z","repository":{"id":218587211,"uuid":"746835910","full_name":"faisal-fida/News-Scraper-API","owner":"faisal-fida","description":"This project is a FastAPI-based web application designed to scrape news data from the web and store it in a Supabase database. It uses asynchronous programming to efficiently fetch and process large amounts of data.","archived":false,"fork":false,"pushed_at":"2024-10-06T07:40:26.000Z","size":116,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-08T08:46:06.111Z","etag":null,"topics":["asynchronous","database","fastapi","newsapi","scraper","supabase"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/faisal-fida.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-22T19:10:39.000Z","updated_at":"2024-10-06T07:41:30.000Z","dependencies_parsed_at":"2024-01-22T23:13:49.348Z","dependency_job_id":"a590a942-206c-4d72-81f8-eccc33a518ef","html_url":"https://github.com/faisal-fida/News-Scraper-API","commit_stats":null,"previous_names":["faisal-fida/news_scraper_api","faisal-fida/news-scraper-api"],"tags_count":0,"template":false,"template_full_name":"github/codespaces-blank","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faisal-fida%2FNews-Scraper-API","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faisal-fida%2FNews-Scraper-API/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faisal-fida%2FNews-Scraper-API/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/faisal-fida%2FNews-Scraper-API/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/faisal-fida","download_url":"https://codeload.github.com/faisal-fida/News-Scraper-API/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240738091,"owners_count":19849546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["asynchronous","database","fastapi","newsapi","scraper","supabase"],"created_at":"2024-11-10T21:16:27.961Z","updated_at":"2026-04-19T19:33:22.319Z","avatar_url":"https://github.com/faisal-fida.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# News Scraper API\n\nThis project is a FastAPI-based web application designed to scrape news data from the web and store it in a Supabase database. It uses asynchronous programming to efficiently fetch and process large amounts of data.\n\n## Table of Contents\n\n- [Installation](#installation)\n- [Usage](#usage)\n- [Complexities](#complexities)\n- [Solutions](#solutions)\n- [Challenges](#challenges)\n- [License](#license)\n\n## Installation\n\n1. Clone the repository:\n\n```sh\ngit clone https://github.com/faisal-fida/news_scraper_api.git\ncd news_scraper_api\n```\n\n2. Install the dependencies:\n\n```sh\npip install -r requirements.txt\n```\n\n3. Create a `.env` file with the following content:\n\n```\nSUPABASE_URL=\u003cyour_supabase_url\u003e\nSUPABASE_KEY=\u003cyour_supabase_key\u003e\n```\n\n4. Run the application:\n\n```sh\nuvicorn app:app --reload\n```\n\n## Usage\n\nThe application provides two endpoints:\n\n1. `GET /` - Returns the status of the API.\n2. `GET /news` - Fetches the latest news data.\n\n## Features\n\n- **Asynchronous Programming**: The project uses Python's `asyncio` and `httpx` libraries to perform asynchronous HTTP requests, significantly improving the efficiency and performance of data scraping.\n- **Database Integration**: Utilizes Supabase as a backend database to store and manage the scraped news data.\n- **Web Scraping**: Uses BeautifulSoup to parse HTML and extract relevant news information from web pages.\n- **Efficient Data Fetching**: By implementing a semaphore and asynchronous requests, the project is capable of handling multiple concurrent connections, thus speeding up the data fetching process.\n- **Database Operations**: The project includes functions to insert, retrieve, and update data in Supabase, ensuring that the data is stored and managed effectively.\n- **Error Handling**: The project includes basic error handling to manage scenarios where data cannot be fetched or inserted into the database.\n- **Rate Limiting**: Managing the rate at which requests are made to avoid being blocked by the target website.\n- **Data Consistency**: Ensuring that data is consistently and correctly scraped, parsed, and stored in the database.\n- **Scalability**: Making sure the application can handle large volumes of data and scale effectively with increased load.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffaisal-fida%2Fnews-scraper-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffaisal-fida%2Fnews-scraper-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffaisal-fida%2Fnews-scraper-api/lists"}