{"id":18356670,"url":"https://github.com/aaravmalani/webscraper","last_synced_at":"2025-04-10T01:49:25.972Z","repository":{"id":171286984,"uuid":"647692763","full_name":"AaravMalani/webscraper","owner":"AaravMalani","description":"Scrape the web using SQLite and requests_html! ","archived":false,"fork":false,"pushed_at":"2023-05-31T10:58:58.000Z","size":8,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-15T17:46:38.095Z","etag":null,"topics":["collaborate","hacktober","python","requests","requests-html","scraping","sql","sqlite","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AaravMalani.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-31T10:19:32.000Z","updated_at":"2023-10-15T18:14:07.000Z","dependencies_parsed_at":"2023-07-22T13:16:23.615Z","dependency_job_id":null,"html_url":"https://github.com/AaravMalani/webscraper","commit_stats":null,"previous_names":["aaravmalani/webscraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaravMalani%2Fwebscraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaravMalani%2Fwebscraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaravMalani%2Fwebscraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AaravMalani%2Fwebscraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AaravMalani","download_url":"https://codeload.github.com/AaravMalani/webscraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248142782,"owners_count":21054667,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collaborate","hacktober","python","requests","requests-html","scraping","sql","sqlite","webscraping"],"created_at":"2024-11-05T22:11:07.890Z","updated_at":"2025-04-10T01:49:25.942Z","avatar_url":"https://github.com/AaravMalani.png","language":"Python","readme":"## ⚠️ This code is unoptimized and needs contributors. [Read below](#drawbacks)\n\n# webscraper\nScrape the web using SQLite and requests_html! \n\n## Usage\n**Warning:** Using this script may get you blocked from certain websites, use at your own risk.\n```sh\npip3 install -r requirements.txt\npython main.py\n```\n\n## Data used\nThe SQLite database is stored at `dev.db`.\nIt has two tables, `TO_CHECK` AND `DATA`.\n\nThe `TO_CHECK` table stores the pages that the scraper has to check.\nIt has three columns, `id` (the ID), `uri` (the page to check) and `sitemap` (1 if it is a [sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) to check or 0 if it is a webpage)\n\nThe `DATA` table stores the pages the scraper has already checked.\nIts columns are\n- `id` (The ID)\n- `uri` (The URI of the page)\n- `error` (1 if there was an error in getting the page)\n- `type` (IOError if there was an error, else the MIME type of the page or NULL)\n- `title` (The title of the page if any)\n- `description` (The description of the page if any)\n- `keywords` (A JSON list of keywords)\n- `linksto` (A list of absolute links that the page links to)\n- `createdAt` (A timestamp at which the page was indexed)\n\n## Drawbacks\n- The script is fairly unoptimized\n- It doesn't run JavaScript so most major social media sites contain usable data\n- It doesn't have a proper indexing system and doesn't have a frontend to search the data used.\n\n## Contributions\nAs you can see from the drawbacks above, the script is in real use of help. If you can find the tiniest thing that would improve the script, please feel free to contribute.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaravmalani%2Fwebscraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faaravmalani%2Fwebscraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faaravmalani%2Fwebscraper/lists"}