{"id":51153084,"url":"https://github.com/nb3n/sitemap-indexer","last_synced_at":"2026-06-26T08:01:32.096Z","repository":{"id":361227872,"uuid":"1253626109","full_name":"nb3n/sitemap-indexer","owner":"nb3n","description":"CLI tool for generating and managing sitemap index files for large websites","archived":false,"fork":false,"pushed_at":"2026-06-22T11:19:52.000Z","size":66,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-22T13:14:48.141Z","etag":null,"topics":["crawler","indexing","python","search-engine-optimization","seo","sitemap","sitemap-generator"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nb3n.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-29T16:49:39.000Z","updated_at":"2026-06-22T11:19:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nb3n/sitemap-indexer","commit_stats":null,"previous_names":["nb3n/sitemap-indexer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nb3n/sitemap-indexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nb3n%2Fsitemap-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nb3n%2Fsitemap-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nb3n%2Fsitemap-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nb3n%2Fsitemap-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nb3n","download_url":"https://codeload.github.com/nb3n/sitemap-indexer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nb3n%2Fsitemap-indexer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34808043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-26T02:00:06.560Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","indexing","python","search-engine-optimization","seo","sitemap","sitemap-generator"],"created_at":"2026-06-26T08:01:31.234Z","updated_at":"2026-06-26T08:01:32.090Z","avatar_url":"https://github.com/nb3n.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google Sitemap Indexer\n\nSubmits all URLs from a live XML sitemap to the [Google Indexing API](https://developers.google.com/search/apis/indexing-api/v3/quickstart).\n\nSupports standard sitemaps, sitemap index files (recursive), and gzip-compressed sitemaps. Handles retries, rate limiting, resume across interrupted runs, and detailed logging automatically.\n\n---\n\n## Prerequisites\n\n- Python 3.10 or higher\n- A Google Cloud project with the **Indexing API** enabled\n- A **Service Account** with a downloaded JSON key\n- The service account email added as an **Owner** in [Google Search Console](https://search.google.com/search-console)\n\n---\n\n## Setup\n\n**1. Clone the repository**\n\n```bash\ngit clone https://github.com/nb3n/sitemap-indexer.git\ncd sitemap-indexer\n```\n\n**2. Install dependencies**\n\n```bash\npip install -r requirements.txt\n```\n\n**3. Enable the Indexing API**\n\nGo to [Google Cloud Console](https://console.cloud.google.com/) and enable the **Indexing API** for your project.\n\n**4. Create a Service Account**\n\n- In Google Cloud Console, go to IAM \u003e Service Accounts\n- Create a new service account\n- Generate a JSON key and save it (e.g. `service_account.json`)\n\n**5. Add the service account as a Search Console Owner**\n\n- Open [Google Search Console](https://search.google.com/search-console)\n- Go to Settings \u003e Users and permissions\n- Add the service account email as an **Owner**\n\n---\n\n## Usage\n\n```bash\npython sitemap_indexer.py --sitemap https://example.com/sitemap.xml --key service_account.json\n```\n\n### All options\n\n| Flag | Required | Default | Description |\n|---|---|---|---|\n| `--sitemap` | Yes | | Live URL of your sitemap or sitemap index |\n| `--key` | Yes | | Path to your service account JSON key file |\n| `--delay` | No | `1.0` | Seconds between API requests (must be \u003e 0) |\n| `--retries` | No | `3` | Max retries on rate-limit or server errors (must be \u003e= 1) |\n| `--backoff` | No | `5.0` | Base backoff seconds, multiplied by attempt number (must be \u003e 0) |\n| `--log-file` | No | `logs/indexing.log` | Path to write the full DEBUG log (parent dirs created automatically) |\n| `--dry-run` | No | off | Parse and list URLs without submitting anything |\n| `--filter` | No | | Regex; only matching URLs are submitted |\n| `--resume-file` | No | | File to record and skip already-submitted URLs across runs |\n\n### Examples\n\n```bash\n# Basic usage\npython sitemap_indexer.py \\\n  --sitemap https://example.com/sitemap.xml \\\n  --key service_account.json\n\n# Preview URLs without submitting\npython sitemap_indexer.py \\\n  --sitemap https://example.com/sitemap.xml \\\n  --key service_account.json \\\n  --dry-run\n\n# Submit only blog posts\npython sitemap_indexer.py \\\n  --sitemap https://example.com/sitemap.xml \\\n  --key service_account.json \\\n  --filter '/blog/'\n\n# Resume an interrupted run\npython sitemap_indexer.py \\\n  --sitemap https://example.com/sitemap.xml \\\n  --key service_account.json \\\n  --resume-file submitted_urls.txt\n\n# Write log to a subdirectory (created automatically)\npython sitemap_indexer.py \\\n  --sitemap https://example.com/sitemap.xml \\\n  --key service_account.json \\\n  --log-file logs/run_2026.log\n\n# Slower submission with more retries\npython sitemap_indexer.py \\\n  --sitemap https://example.com/sitemap.xml \\\n  --key service_account.json \\\n  --delay 2.0 \\\n  --retries 5 \\\n  --backoff 10.0\n```\n\n---\n\n## Output\n\nThe script logs to both the terminal (INFO and above) and the log file (full DEBUG).\n\n```\n2026-01-15 10:32:01 [INFO] Sitemap : https://example.com/sitemap.xml\n2026-01-15 10:32:01 [INFO] Key file: service_account.json\n2026-01-15 10:32:01 [INFO] Sitemap Indexer started.\n2026-01-15 10:32:02 [INFO] Fetching sitemap: https://example.com/sitemap.xml\n2026-01-15 10:32:02 [INFO] Found 42 URLs in: https://example.com/sitemap.xml\n2026-01-15 10:32:02 [INFO] Total unique URLs to submit: 42\n2026-01-15 10:32:02 [INFO] [1/42] Submitting: https://example.com/\n2026-01-15 10:32:03 [INFO] Accepted. Notify time: 2026-01-15T10:32:03.123Z\n...\n2026-01-15 10:33:24 [INFO] --- Summary ---\n2026-01-15 10:33:24 [INFO] Total   : 42\n2026-01-15 10:33:24 [INFO] Success : 41\n2026-01-15 10:33:24 [INFO] Failed  : 1\n2026-01-15 10:33:24 [INFO] Full log written to logs/indexing.log\n```\n\n---\n\n## Exit codes\n\n| Code | Meaning |\n|---|---|\n| 0 | All URLs submitted successfully, or dry run completed |\n| 1 | Sitemap could not be fetched, or one or more submissions failed |\n\nThis makes the script safe to use in CI/CD pipelines: a non-zero exit code signals that action is needed.\n\n---\n\n## Quota\n\nThe Google Indexing API allows **200 requests per day** by default per Search Console property. The script warns automatically if a run is about to meet or exceed this limit.\n\nYou can request a quota increase in Google Cloud Console if needed. The default `--delay 1.0` keeps submissions safe and predictable.\n\n---\n\n## Error handling\n\n| HTTP status | Behaviour |\n|---|---|\n| 400 | Fails immediately. The URL is malformed or not owned by the Search Console property. |\n| 403 | Fails immediately. The service account is not an Owner in Search Console. |\n| 404 | Fails immediately. The URL is not publicly accessible. |\n| 429 | Retries with backoff up to `--retries` times, then fails. |\n| 5xx | Retries with backoff up to `--retries` times, then fails. |\n\nFailed URLs are listed individually in the summary and log file. A single failure does not stop remaining submissions.\n\n---\n\n## Resume support\n\nUse `--resume-file` to make long runs safe to interrupt. Every accepted URL is appended to the file immediately. On the next run with the same file, those URLs are skipped automatically.\n\n```bash\n# First run (interrupted at URL 80/200)\npython sitemap_indexer.py --sitemap ... --key ... --resume-file submitted_urls.txt\n\n# Second run picks up from where the first stopped\npython sitemap_indexer.py --sitemap ... --key ... --resume-file submitted_urls.txt\n```\n\n\n---\n\n## IndexNow Indexer\n\nFetches all URLs from a live XML sitemap and submits them to the **IndexNow API**, which notifies Bing, Yandex, and all other participating search engines simultaneously in a single batch request.\n\n**Key differences from Google (`sitemap_indexer.py`)**\n\n| | Google | Bing / IndexNow |\n|---|---|---|\n| Auth | Service account JSON key + OAuth | Plain API key string |\n| Submission | One URL per API call (200/day limit) | Up to 10,000 URLs per request |\n| Engines notified | Google only | Bing, Yandex, and all IndexNow participants |\n| Removal API | Yes (`URL_DELETED`) | No. Return 404 or 410 from the page. |\n\n### Prerequisites\n\n- A Bing Webmaster Tools account at [webmaster.bing.com](https://www.bing.com/webmasters)\n- Your site verified in Bing Webmaster Tools\n- An IndexNow API key (generate one at [bing.com/indexnow/getstarted](https://www.bing.com/indexnow/getstarted))\n- A key file hosted at `https://yourdomain.com/\u003cyour-key\u003e.txt` containing only your key as plain text\n\nThe key file is how IndexNow verifies you own the domain. Without it, submissions will be rejected with HTTP 403.\n\n### Setup\n\n**1. Generate your API key**\n\nGo to [bing.com/indexnow/getstarted](https://www.bing.com/indexnow/getstarted) and click Generate. Copy the key shown.\n\n**2. Host your key file**\n\nCreate a plain text file named `\u003cyour-key\u003e.txt`. The file must contain only your API key and nothing else.\n\nUpload it to the root of your website so it is publicly accessible at:\n\n```\nhttps://yourdomain.com/\u003cyour-key\u003e.txt\n```\n\n**3. Install dependencies**\n\nNo new dependencies required. `bing_indexer.py` uses only `requests`, which is already in `requirements.txt`.\n\n### Usage\n\n```bash\npython bing_indexer.py --sitemap https://example.com/sitemap.xml --key YOUR_API_KEY --host example.com\n```\n\n### All options\n\n| Flag | Required | Default | Description |\n|---|---|---|---|\n| `--sitemap` | Yes | | Live URL of your sitemap or sitemap index |\n| `--key` | Yes | | Your IndexNow API key (plain string, not a file path) |\n| `--host` | Yes | | Your domain without protocol (e.g. `example.com`) |\n| `--key-location` | No | | Full URL to your key file, if not hosted at the domain root |\n| `--retries` | No | `3` | Max retries on transient errors (must be \u003e= 1) |\n| `--backoff` | No | `5.0` | Base backoff seconds, multiplied by attempt number (must be \u003e 0) |\n| `--batch-delay` | No | `1.0` | Seconds between batch requests (only relevant for \u003e 10,000 URLs) |\n| `--log-file` | No | `logs/bing.log` | Path to write the full DEBUG log (parent dirs created automatically) |\n| `--dry-run` | No | off | Parse and list URLs without submitting anything |\n| `--filter` | No | | Regex; only matching URLs are submitted |\n\n### Examples\n\n```bash\n# Basic usage\npython bing_indexer.py \\\n  --sitemap https://yourdomain.com/sitemap.xml \\\n  --key YOUR_API_KEY \\\n  --host yourdomain.com\n\n# Preview URLs without submitting\npython bing_indexer.py \\\n  --sitemap https://yourdomain.com/sitemap.xml \\\n  --key YOUR_API_KEY \\\n  --host yourdomain.com \\\n  --dry-run\n\n# Submit only blog posts\npython bing_indexer.py \\\n  --sitemap https://yourdomain.com/sitemap.xml \\\n  --key YOUR_API_KEY \\\n  --host yourdomain.com \\\n  --filter '/blog/'\n\n# Key file hosted at a non-root location\npython bing_indexer.py \\\n  --sitemap https://yourdomain.com/sitemap.xml \\\n  --key YOUR_API_KEY \\\n  --host yourdomain.com \\\n  --key-location https://yourdomain.com/keys/mykey.txt\n```\n\n### Output\n\nAll URLs are sent in a single batch (or multiple batches for \u003e 10,000 URLs). The log reflects this.\n\n```\n2026-01-15 10:32:01 [INFO] Sitemap : https://yourdomain.com/sitemap.xml\n2026-01-15 10:32:01 [INFO] Host    : yourdomain.com\n2026-01-15 10:32:01 [INFO] API key : YOUR_API_KEY\n2026-01-15 10:32:01 [INFO] Bing Indexer started.\n2026-01-15 10:32:02 [INFO] Found 44 URLs in: https://yourdomain.com/sitemap.xml\n2026-01-15 10:32:02 [INFO] Submitting 44 URLs in 1 batch.\n2026-01-15 10:32:02 [INFO] Batch [1/1]: submitting 44 URLs.\n2026-01-15 10:32:03 [INFO] Batch [1/1]: accepted.\n2026-01-15 10:32:03 [INFO] --- Summary ---\n2026-01-15 10:32:03 [INFO] Total URLs  : 44\n2026-01-15 10:32:03 [INFO] Submitted   : 44\n2026-01-15 10:32:03 [INFO] Failed      : 0\n2026-01-15 10:32:03 [INFO] Batches OK  : 1\n2026-01-15 10:32:03 [INFO] Batches fail: 0\n2026-01-15 10:32:03 [INFO] Full log written to logs/bing.log\n```\n\n### Exit codes\n\n| Code | Meaning |\n|---|---|\n| 0 | All batches submitted successfully, or dry run completed |\n| 1 | Sitemap could not be fetched, or one or more batches failed |\n\n### Error handling\n\n| HTTP status | Behaviour |\n|---|---|\n| 200 / 202 | Accepted. 202 means key validation is still pending. |\n| 400 | Fails immediately. URLs are malformed or incorrectly formatted. |\n| 403 | Fails immediately. API key not found or key file not accessible. |\n| 422 | Fails immediately. URLs do not belong to the declared host, or key schema mismatch. |\n| 429 | Retries with backoff up to `--retries` times, then fails. |\n| 5xx | Retries with backoff up to `--retries` times, then fails. |\n\n### Removing URLs from Bing\n\nIndexNow has no deletion API. To remove a URL from Bing:\n\n- Return HTTP **410 Gone** (preferred) or **404 Not Found** from the page\n- Add `Disallow: /` to `robots.txt` for domains that should never be indexed (sandbox, CDN)\n- Optionally use the URL removal tool in [Bing Webmaster Tools](https://www.bing.com/webmasters) for faster manual removal\n\n---\n\n## deindex.py\n\nRemoves URLs from Google's index by submitting `URL_DELETED` notifications via the Google Indexing API v3.\n\nUse this to clean up sandbox URLs accidentally indexed, deleted pages, duplicate www subdomain URLs, CDN domains, or malformed query string URLs that should not appear in search results.\n\n### Usage\n\n```bash\n# Remove specific URLs directly\npython deindex.py --key service_account.json https://sandbox.example.com/page\n\n# Remove all URLs listed in a file\npython deindex.py --key service_account.json --url-file bad_urls.txt\n\n# Combine both\npython deindex.py --key service_account.json --url-file bad_urls.txt https://extra.example.com/page\n\n# Preview what would be removed without touching the API\npython deindex.py --key service_account.json --url-file bad_urls.txt --dry-run\n```\n\n### All options\n\n| Flag | Required | Default | Description |\n|---|---|---|---|\n| `URL` (positional) | No | | One or more URLs to remove, passed directly as arguments |\n| `--key` | Yes | | Path to your service account JSON key file |\n| `--url-file` | No | | Path to a plain text file with one URL per line (# lines are comments) |\n| `--delay` | No | `1.0` | Seconds between API requests (must be \u003e 0) |\n| `--retries` | No | `3` | Max retries on transient errors (must be \u003e= 1) |\n| `--backoff` | No | `5.0` | Base backoff seconds, multiplied by attempt number (must be \u003e 0) |\n| `--log-file` | No | `logs/deindex.log` | Path to write the full DEBUG log (parent dirs created automatically) |\n| `--dry-run` | No | off | List URLs that would be removed without calling the API |\n\n### URL file format\n\nPlain text, one URL per line. Lines starting with `#` are comments and are ignored.\n\n```\n# Sandbox URLs\nhttps://sandbox.example.com/en\nhttps://sandbox.example.com/about\n\n# CDN domain - serves assets only\nhttps://cdn.example.com\n\n# www duplicates\nhttps://www.example.com/contact\nhttps://www.example.com/services\n```\n\n### Output\n\n```\n2026-01-15 10:32:01 [INFO] Key file : service_account.json\n2026-01-15 10:32:01 [INFO] URLs to remove: 15\n2026-01-15 10:32:01 [INFO] Authenticated with Google Indexing API.\n2026-01-15 10:32:01 [INFO] [1/15] Removing: https://sandbox.example.com/en\n2026-01-15 10:32:02 [INFO] Accepted for removal.\n...\n2026-01-15 10:32:18 [INFO] --- Summary ---\n2026-01-15 10:32:18 [INFO] Total   : 15\n2026-01-15 10:32:18 [INFO] Removed : 15\n2026-01-15 10:32:18 [INFO] Failed  : 0\n2026-01-15 10:32:18 [INFO] Full log written to logs/deindex.log\n```\n\n### How long removal takes\n\nRemoval typically takes 3 to 7 days. The URL disappears from search results once Google recrawls and confirms the page is gone. If the page still returns HTTP 200, Google may ignore the removal and reindex it.\n\nFor permanent removal, combine the API call with two server-side changes:\n\n- Return HTTP 410 Gone (preferred) or 404 Not Found from the page\n- Add `Disallow: /` to `robots.txt` on the relevant domain (for subdomains like sandbox or CDN)\n\nA 410 is processed faster than a 404 because it signals the deletion is intentional and permanent.\n\n### Error handling\n\n| HTTP status | Behaviour |\n|---|---|\n| 400 | Fails immediately. The URL is malformed or not owned by the Search Console property. |\n| 403 | Fails immediately. The service account is not an Owner in Search Console. |\n| 404 | Fails immediately. The URL was not indexed or is already removed. |\n| 429 | Retries with backoff up to `--retries` times, then fails. |\n| 5xx | Retries with backoff up to `--retries` times, then fails. |\n\n### Quota\n\n`deindex.py` and `sitemap_indexer.py` share the same 200 requests/day quota. If you are running both on the same day, plan accordingly.\n\n## Security\n\nNever commit your service account JSON key to version control. The `.gitignore` in this repo excludes all `*.json` files for this reason. Store the key file outside the repo, or use an environment variable or secrets manager in production.\n\n## License\n\n[MIT](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnb3n%2Fsitemap-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnb3n%2Fsitemap-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnb3n%2Fsitemap-indexer/lists"}