{"id":47120816,"url":"https://github.com/fikrimohammad/secret-scraper","last_synced_at":"2026-03-12T19:28:12.276Z","repository":{"id":343290934,"uuid":"1177037033","full_name":"fikrimohammad/secret-scraper","owner":"fikrimohammad","description":"A Go-based HTTP service that scrapes public GitHub repositories for exposed secrets and API keys using configurable regex patterns.","archived":false,"fork":false,"pushed_at":"2026-03-09T17:40:58.000Z","size":15,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-09T23:06:54.626Z","etag":null,"topics":["api-key-scanner","github-scraper","go","golang","secret-detection","secret-scanner"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fikrimohammad.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-09T16:22:11.000Z","updated_at":"2026-03-09T17:41:02.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/fikrimohammad/secret-scraper","commit_stats":null,"previous_names":["fikrimohammad/secret-scraper"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/fikrimohammad/secret-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fikrimohammad%2Fsecret-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fikrimohammad%2Fsecret-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fikrimohammad%2Fsecret-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fikrimohammad%2Fsecret-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fikrimohammad","download_url":"https://codeload.github.com/fikrimohammad/secret-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fikrimohammad%2Fsecret-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30439724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-12T14:34:45.044Z","status":"ssl_error","status_checked_at":"2026-03-12T14:09:33.793Z","response_time":114,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-key-scanner","github-scraper","go","golang","secret-detection","secret-scanner"],"created_at":"2026-03-12T19:28:11.614Z","updated_at":"2026-03-12T19:28:12.258Z","avatar_url":"https://github.com/fikrimohammad.png","language":"Go","readme":"# secret-scraper\n\nA Go-based HTTP service that scrapes public GitHub repositories for exposed secrets and API keys using configurable regex patterns.\n\n## How It Works\n\n1. You send a POST request specifying a `secret_provider` and `secret_type`.\n2. The service looks up the matching search keyword and regex pattern from `config.yaml`.\n3. It queries the GitHub Code Search API across N pages (iterations), fetches the raw content of each matched file, and extracts secrets using the configured regex.\n4. Deduplicated secrets are returned in the response.\n\n## Project Structure\n\n```\nsecret-scraper/\n├── cmd/main.go                          # Entry point, wires dependencies, starts Fiber HTTP server\n├── config/config.go                     # Config loading from YAML\n├── files/config/\n│   ├── config.yaml                      # Active config (gitignored)\n│   └── config.yaml.sample               # Sample config\n├── handler/scraper/rest/\n│   └── scrape_secret.go                 # POST /v1/scraper/scrape_secret handler\n├── usecase/scraper/\n│   └── scrape_secret.go                 # Core scraping logic: search → fetch → regex match\n├── repository/\n│   ├── github/client/                   # GitHub API: code search + raw file fetch\n│   └── config/static/                   # Config-based scraper rule lookup\n├── model/\n│   ├── secret.go                        # Secret, SecretProvider, SecretType\n│   └── github.go                        # GithubCode model\n└── util/rest/response.go                # JSON error helper\n```\n\n## Prerequisites\n\n- Go 1.22+\n- A GitHub personal access token with `repo` (or `public_repo`) scope\n\n## Setup\n\n```bash\ngit clone https://github.com/fikrimohammad/secret-scraper.git\ncd secret-scraper\n\ncp files/config/config.yaml.sample files/config/config.yaml\n# Edit config.yaml and set your GitHub access token\n```\n\n### `files/config/config.yaml`\n\n```yaml\ngithub:\n  access_token: YOUR_GITHUB_TOKEN\n\nsecret_scraper:\n  - secret_provider: anthropic\n    secret_type: anthropic_api_key\n    secret_query_keyword: sk-ant-api03\n    secret_regex_pattern: sk-ant-api03-[a-zA-Z0-9\\-_]+\n  - secret_provider: anthropic\n    secret_type: anthropic_admin_key\n    secret_query_keyword: sk-ant-admin01\n    secret_regex_pattern: sk-ant-admin01-[a-zA-Z0-9\\-_]+\n```\n\nAdd more entries to scan for other secret types (AWS, OpenAI, etc.).\n\n## Running\n\n```bash\ngo run ./cmd/main.go\n```\n\nThe server starts on port `3000`.\n\n## API\n\n### `POST /v1/scraper/scrape_secret`\n\n**Request body:**\n\n```json\n{\n  \"secret_provider\": \"anthropic\",\n  \"secret_type\": \"anthropic_api_key\",\n  \"max_limit_per_iterations\": 10,\n  \"max_iterations\": 5\n}\n```\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `secret_provider` | string | required | Provider name (e.g. `anthropic`) |\n| `secret_type` | string | required | Secret type (e.g. `anthropic_api_key`) |\n| `max_limit_per_iterations` | int | 10 | Results per GitHub search page |\n| `max_iterations` | int | 10 | Number of pages to scan |\n\n**Response:**\n\n```json\n{\n  \"data\": [\n    {\n      \"provider\": \"anthropic\",\n      \"type\": \"anthropic_api_key\",\n      \"value\": \"sk-ant-api03-...\"\n    }\n  ]\n}\n```\n\n## Dependencies\n\n| Package | Purpose |\n|---------|---------|\n| `gofiber/fiber/v3` | HTTP framework |\n| `google/go-github/v84` | GitHub API client |\n| `gopkg.in/yaml.v3` | Config parsing |\n\n## License\n\nMIT","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffikrimohammad%2Fsecret-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffikrimohammad%2Fsecret-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffikrimohammad%2Fsecret-scraper/lists"}