{"id":13714124,"url":"https://github.com/app-sre/github-mirror","last_synced_at":"2026-01-26T13:34:11.292Z","repository":{"id":37962387,"uuid":"238664886","full_name":"app-sre/github-mirror","owner":"app-sre","description":"GitHub API mirror that caches the responses and implements conditional requests","archived":false,"fork":false,"pushed_at":"2026-01-24T04:14:13.000Z","size":804,"stargazers_count":37,"open_issues_count":5,"forks_count":32,"subscribers_count":12,"default_branch":"master","last_synced_at":"2026-01-24T11:59:42.940Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/app-sre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2020-02-06T10:40:58.000Z","updated_at":"2026-01-22T00:18:24.000Z","dependencies_parsed_at":"2023-01-21T18:45:15.239Z","dependency_job_id":"8e47b300-702c-4332-b7a6-6682eb00f87c","html_url":"https://github.com/app-sre/github-mirror","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/app-sre/github-mirror","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/app-sre%2Fgithub-mirror","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/app-sre%2Fgithub-mirror/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/app-sre%2Fgithub-mirror/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/app-sre%2Fgithub-mirror/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/app-sre","download_url":"https://codeload.github.com/app-sre/github-mirror/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/app-sre%2Fgithub-mirror/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28779341,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T11:46:04.308Z","status":"ssl_error","status_checked_at":"2026-01-26T11:46:02.664Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T23:01:52.778Z","updated_at":"2026-01-26T13:34:11.286Z","avatar_url":"https://github.com/app-sre.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# GitHub Mirror\n\nGitHub API mirror that caches the responses and implements\n[conditional requests](https://docs.github.com/en/rest/using-the-rest-api/best-practices-for-using-the-rest-api?apiVersion=2022-11-28#use-conditional-requests-if-appropriate).\n\nWith conditional requests, all the calls are forwarded to the Github API, but\nwhen the GitHub API replies with a 304 HTTP code, meaning that the resource\nhas not changed, we serve the client with the previously cached response.\n\nThat reduces the number of API calls that consume quota, helping you not to\nhit the API GitHub API rate limit.\n\nThe mirror acts only on GET requests, bypassing other HTTP methods.\n\nThe default cache backend is in-memory, and we also support Redis. The\nin-memory cache is shared among all the threads, but not shared between\nprocesses. Every time the server is started, the cache is initialized empty.\nUsing Redis prevents the cache from being lost when the github mirror server\nis restarted.\n\n## Quick Start\n\nRun the Docker container:\n\n```\n~$ docker run --rm -it -p 8080:8080 quay.io/redhat-services-prod/app-sre-tenant/github-mirror-master/github-mirror-master\n[2021-02-20 12:40:16 +0000] [1] [INFO] Starting gunicorn 20.0.4\n[2021-02-20 12:40:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)\n[2021-02-20 12:40:16 +0000] [1] [INFO] Using worker: threads\n[2021-02-20 12:40:16 +0000] [8] [INFO] Booting worker with pid: 8\n```\n\nUse it as the Github API url:\n\n```\n\u003e\u003e\u003e import requests\n\u003e\u003e\u003e\n\u003e\u003e\u003e requests.get('http://localhost:8080/repos/app-sre/github-mirror')\n\u003cResponse [200]\u003e\n\u003e\u003e\u003e\n\u003e\u003e\u003e requests.get('http://localhost:8080/repos/app-sre/github-mirror')\n\u003cResponse [200]\u003e\n\u003e\u003e\u003e\n```\n\nAfter those two requests, the server log will show:\n\n```\n2020-02-16 21:08:07,948 [GET] CACHE_MISS https://api.github.com/repos/app-sre/github-mirror\n2020-02-16 21:08:13,585 [GET] CACHE_HIT https://api.github.com/repos/app-sre/github-mirror\n```\n\nThe second request was served from the cache and it did not consume the API\ncalls quota.\n\nIf you're using PyGithub, you have to pass the `base_url` when creating the\nclient instance:\n\n```\n\u003e\u003e\u003e from github import Github\n\u003e\u003e\u003e gh_cli = Github(base_url='http://localhost:8080')\n```\n\n## Redis Cache Backend\n\nTo enable the Redis backend, set the environment variable:\n\n```\nCACHE_TYPE=redis\n```\n\nIn addition to that, you can provide the following optional configuration:\n\n- `PRIMARY_ENDPOINT` is the primary endpoint or host address of the Redis\n  service. If not set, it defaults to `localhost`.\n- `READER_ENDPOINT` is the read-only replica endpoint and can be used to\n  increase the read availability of the Redis service. If not set, it defaults\n  to the same address as the primary endpoint.\n- `REDIS_PORT` is the port which the Redis service binds to. The default port\n  is `6379`.\n- `REDIS_PASSWORD` is the authentication token to access a password protected\n  Redis server. If not set, the default is no authentication.\n- `REDIS_SSL` should be set to `True` if you are encrypting the traffic to the\n  Redis server. If not set, the default assumes no encryption.\n\nYou will find more details about the Redis cache backend implementation in the\n[Redis Cache Backend doc](docs/redis_cache_backend.md).\n\n## Metrics\n\nThe service has a `/metrics` endpoint, exposing metrics in the Prometheus\nformat:\n\n```\n\u003e\u003e\u003e response = requests.get('http://localhost:8080/metrics')\n\u003e\u003e\u003e print(response.content.decode())\n...\nhttp_request_total 2.0\n...\nrequest_latency_seconds_count{cache=\"MISS\",method=\"GET\",status=\"200\"} 1.0\n...\nrequest_latency_seconds_count{cache=\"HIT\",method=\"GET\",status=\"200\"} 1.0\n...\n```\n\nWith that, as the Github API rate limit is per hour, you can have the total\nHIT/MISS per hour with:\n\n```\nsum(increase(request_latency_seconds_count{endpoint=\"github-mirror\",cache=\"ONLINE_HIT\"}[1h]))\n```\n\nand\n\n```\nsum(increase(request_latency_seconds_count{endpoint=\"github-mirror\",cache=\"ONLINE_MISS\"}[1h]))\n```\n\nPlotting on Grafana, we have:\n\n![](docs/images/grafana_hits_misses.png)\n\nMany more metrics are available. Check the `/metrics` endpoint for details.\n\n## User Validation\n\nTo enable the user validation, the `GITHUB_USERS` environment variable\nshould be available to the server. The `GITHUB_USERS` is a colon-separated\nlist of authorized users to have their requests to the Github Mirror served.\nExample:\n\n```\nGITHUB_USERS=app-sre-bot,quay-io-bot\n```\n\nThe user validation, when enabled, will not allow unauthenticated requests\nto the Github Mirror.\n\nPlease notice that, in order to validate the user, one additional get request\nis made to the Github API, to the `/user` endpoint, using the provided\nauthorization token. That call will also go through the caching mechanism, so\nthe rate limit will be preserved when possible.\n\n## Offline Mode\n\nThere's a built-in mechanism to detect when the Github API is offline.\n\nTo do so, we have a separate thread that keeps checking the url\n`https://www.githubstatus.com/api/v2/components.json` every second.\nWhen we don't get a success response, or `API Requests` component is `major_outage`,\nwe consider the Github API offline.\n\nWhen that happens, all the requests are served from the cache until we detect\nthat the Github API is back online.\n\nRequests served from the cache when we are in offline mode will be accounted\nfor in separate metrics:\n\n```\nrequest_latency_seconds_count{endpoint=\"github-mirror\",cache=\"OFFLINE_HIT\"}\n```\n\nand\n\n```\nrequest_latency_seconds_count{endpoint=\"github-mirror\",cache=\"OFFLINE_MISS\"}\n```\n\n## Contributing\n\nFor contributing to the project, please follow the\n[Development Guide](docs/devel_guide.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapp-sre%2Fgithub-mirror","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapp-sre%2Fgithub-mirror","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapp-sre%2Fgithub-mirror/lists"}