{"id":24663997,"url":"https://github.com/sujalchoudhari/websiteindexingchecker","last_synced_at":"2026-05-16T12:33:55.378Z","repository":{"id":237400907,"uuid":"679310389","full_name":"SujalChoudhari/WebsiteIndexingChecker","owner":"SujalChoudhari","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-01T07:54:57.000Z","size":32592,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-21T08:48:43.897Z","etag":null,"topics":["bs4","flask","google-indexing","python","scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SujalChoudhari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-16T14:55:21.000Z","updated_at":"2024-05-01T07:57:18.000Z","dependencies_parsed_at":"2024-05-01T09:05:12.381Z","dependency_job_id":null,"html_url":"https://github.com/SujalChoudhari/WebsiteIndexingChecker","commit_stats":null,"previous_names":["sujalchoudhari/websiteindexingchecker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/SujalChoudhari/WebsiteIndexingChecker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SujalChoudhari%2FWebsiteIndexingChecker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SujalChoudhari%2FWebsiteIndexingChecker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SujalChoudhari%2FWebsiteIndexingChecker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SujalChoudhari%2FWebsiteIndexingChecker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SujalChoudhari","download_url":"https://codeload.github.com/SujalChoudhari/WebsiteIndexingChecker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SujalChoudhari%2FWebsiteIndexingChecker/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33102896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-16T04:41:52.686Z","status":"ssl_error","status_checked_at":"2026-05-16T04:41:52.009Z","response_time":115,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bs4","flask","google-indexing","python","scraping"],"created_at":"2025-01-26T05:17:34.374Z","updated_at":"2026-05-16T12:33:55.363Z","avatar_url":"https://github.com/SujalChoudhari.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Website Index Checker App\n\nWelcome to the **Website Index Checker** app! This repository contains code that enables you to check if the URLs listed in a given sitemap are indexed by Google. This README provides instructions for team members on how to work with this app.\n\n## Getting Started\n\nTo get started with the **Website Index Checker** app, follow these steps:\n\n### 1. Modify Sheets URL\nIn the `constants.py` file, you will find a section where you need to modify the URLs related to Google Sheets and indexing search. Open the `constants.py` file and update the following variables:\n\n```python\nURL_TO_SHEETS = \"https://docs.google.com/spreadsheets/d/1up6bYQj2X5XoYfS9aN8zAT5oz8dFQIGZlXCr_Madgug/edit#gid=1700410190\"\nINDEXING_SEARCH_STRING = \"https://www.google.com/search?q=site:{}\u0026num=1\"\n```\nAlso add the email of the google sheets api service account with the sheets. This is required to give the service account access to the sheet. \n*EMAIL can be found in service_account.json*\n \nReplace the existing URLs with your own Google Sheets URL and indexing search string. Make sure to follow the correct format for URLs.\n\n### 2. Update Service Account for Google Sheets API\n\nThe **Website Index Checker** app uses the gspread library to interact with Google Sheets API. You will need to provide a `service_account.json` file for authentication. Replace the existing `service_account.json` file in the root directory with your own service account JSON file. This file should be obtained from the Google Cloud Console.\n\nsample file \n```js\n{\n    \"type\": \"service_account\",\n    \"project_id\": \"em...608\",\n    \"private_key_id\": \"991d8f.....6e3d\",\n    \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nM.....TA=\\n-----END PRIVATE KEY-----\\n\",\n    \"client_email\": \"sit...am.gserviceaccount.com\",\n    \"client_id\": \"103...5\",\n    \"auth_uri\": \"htt...2/auth\",\n    \"token_uri\": \"http...token\",\n    \"auth_provider_x509_cert_url\": \"https:/...rts\",\n    \"client_x509_cert_url\": \"http...t.com\",\n    \"universe_domain\": \"go..om\"\n}\n```\n\n### 3. Dependencies and Hosting\n\nThe app is built using Flask, Requests, BeautifulSoup4, and gspread. It is hosted on Heroku using Gunicorn. Make sure you have the required dependencies installed. You can install the dependencies using the following command:\n\n```bash\npip install -r requirements.txt\n```\n\n### 4. Running the App\n\nOnce you have set up the necessary configurations, you can run the app locally. Use the following command to start the Flask app:\n\n```bash\npython wsgi.py\n\n# OR\n\nflask run\n```\n\nThe app will start running locally, and you can access it by navigating to `http://127.0.0.1:5000/` in your web browser.\n\n## How the App Works\n\nThe **Website Index Checker** app checks if the URLs listed in the specified sitemap are indexed by Google. It does this by sending search queries to Google and checking if the URLs appear in the search results. The app uses a specific User-Agent and language settings for these requests.\n\n## Questions or Issues\n\nIf you have any questions or encounter any issues while working with the app, please contact [@SujalChoudhari](https://github.com/SujalChoudhari).\nHappy indexing!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsujalchoudhari%2Fwebsiteindexingchecker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsujalchoudhari%2Fwebsiteindexingchecker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsujalchoudhari%2Fwebsiteindexingchecker/lists"}