{"id":15568630,"url":"https://github.com/jayvdb/https-everywhere-py","last_synced_at":"2025-03-16T07:32:22.397Z","repository":{"id":46165662,"uuid":"243063214","full_name":"jayvdb/https-everywhere-py","owner":"jayvdb","description":"Privacy for Pythons","archived":false,"fork":false,"pushed_at":"2022-10-31T09:49:04.000Z","size":106,"stargazers_count":9,"open_issues_count":33,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-09T19:15:27.542Z","etag":null,"topics":["hstspreload","https-client","https-everywhere","https-redirect","python-requests","python-requests-adapter"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jayvdb.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-25T17:51:27.000Z","updated_at":"2023-11-07T12:53:44.000Z","dependencies_parsed_at":"2022-09-24T15:00:15.041Z","dependency_job_id":null,"html_url":"https://github.com/jayvdb/https-everywhere-py","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayvdb%2Fhttps-everywhere-py","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayvdb%2Fhttps-everywhere-py/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayvdb%2Fhttps-everywhere-py/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayvdb%2Fhttps-everywhere-py/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jayvdb","download_url":"https://codeload.github.com/jayvdb/https-everywhere-py/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243806049,"owners_count":20350775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hstspreload","https-client","https-everywhere","https-redirect","python-requests","python-requests-adapter"],"created_at":"2024-10-02T17:19:49.999Z","updated_at":"2025-03-16T07:32:21.935Z","avatar_url":"https://github.com/jayvdb.png","language":"Python","readme":"# https-everywhere : Privacy for Pythons\n\n![Codecov](https://img.shields.io/codecov/c/gh/jayvdb/https-everywhere-py)\n![AppVeyor CI](https://img.shields.io/appveyor/build/jayvdb/https-everywhere-py)\n![Cirrus CI](https://img.shields.io/cirrus/github/jayvdb/https-everywhere-py)\n\nThis project primarily provides [requests](https://github.com/psf/requests/) adapters for Chrome [HSTS Preload](http://hstspreload.org/) and [HTTPS Everywhere](https://github.com/EFForg/https-everywhere) rulesets.\n\nAt this stage, the focus is on correct efficient loading of the approx 25,000 rulesets of HTTPS Everywhere for use with any requests.\nEmphasis is on converting those rulesets to simpler or more common rules to reduce memory requirements.\n\nCurrent list of data problems can be found in `https_everywhere/_fixme.py`.  Many of these have patches sent upstream to the main HTTPS Everywhere project.\n\n## Usage\n\n```py\nfrom https_everywhere.session import HTTPSEverywhereSession\n\ns = HTTPSEverywhereSession()\nr = s.get(\"http://freerangekitten.com/\")\nr.raise_for_status()\n\nassert r.url == \"https://freerangekitten.com/\"\nassert len(r.history) == 1\nassert r.history[0].status_code == 302\nassert r.history[0].reason == \"HTTPS Everywhere\"\n```\n\nThe log will emit\n```console\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://ww2\\.epeat\\.com/\"\n[W 200226 09:40:55 _rules:640] Rejecting ruleset EPEAT (partial) as it has no usable rules\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://(?:dashboard(?:-cdn)?|g-pixel|pixel|segment-pixel)\\.invitemedia\\.com/\"\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://((?:a[lt]|s|sca)\\d*|www)\\.listrakbi\\.com/\"\n[W 200226 09:40:55 _rules:640] Rejecting ruleset ListrakBI.com as it has no usable rules\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://demo\\.neobookings\\.com/\"\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://(www\\.)?partners\\.peer1\\.ca/\"\n[W 200226 09:40:55 _rules:640] Rejecting ruleset Peer1.ca (partial) as it has no usable rules\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://support\\.pickaweb\\.co\\.uk/(assets/)\"\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://www\\.svenskaspel\\.se/\"\n[W 200226 09:40:55 _rules:632] Rejecting rule with pattern \"^http://cdn\\.therepublic\\.com/\"\n[W 200226 09:40:55 _rules:640] Rejecting ruleset The Republic (partial) as it has no usable rules\n```\n\n### Adapters\n\nThere are many adapters in https_everywhere.adapter which can be used depending on use cases.\n\nAdapters can be mounted on 'http://', or a narrower mount point.\n\n* HTTPBlockAdapter - Mount on 'http://' to block HTTP traffic\n* HTTPRedirectBlockAdapter - Mount on 'https://' to block HTTPS responses redirecting to HTTP\n* HTTPSEverywhereOnlyAdapter - Apply HTTPS Everywhere rules\n* ChromePreloadHSTSAdapter - Upgrade to HTTPS for sites on Chrome preload list\n* MozillaPreloadHSTSAdapter - Upgrade to HTTPS for sites on Mozilla preload list\n* HTTPSEverywhereAdapter - Chrome preload hsts and https everywhere rules combined\n* ForceHTTPSAdapter - Just use HTTPS, always, everywhere\n* PreferHTTPSAdapter - Check HTTP if there are any redirects, before switching to HTTPS.\n* UpgradeHTTPSAdapter - Force HTTPS, but fall back to HTTP when HTTPS problems occur.\n* SafeUpgradeHTTPSAdapter - First check HTTP if there are any redirects, force HTTPS, and fallback to HTTP.\n\n## Testing\n\nTo test\n\n```sh\ngit clone https://github.com/jayvdb/https-everywhere-py\ngit clone https://github.com/EFForg/https-everywhere  # possibly use --depth 1\ncd https-everywhere-py\ntox\n```\n(Note: `test_rules` takes a long time to begin.)\n\n## Not implemented\n\n- custom local ruleset channels\n- cookie support\n- credentials in urls, such as `http://eff:techprojects@chart.googleapis.com/`,\n  which interfers with many rules, and also prevents exclusions from being applied\n- efficient memory structure for target mapping\n- rules with `@platform='mixedcontent'`; approx 800 rulesets ignored\n- rules with `@default_off`; approx 300 rulesets ignored, but all are mixedcontent\n- ruleset targets containing wildcards in the middle of the domain names (`foo.*.com`), which doesnt exist in the default channel\n- ruleset targets containing a wildcard at beginning and end (`*.foo.*`), which doesnt exist in the default channel\n- overlapping rules, which only applies to voxmedia.com in the default channel when filtered to exclude rules with `@default_off` and `@platform`.\n- rules for IPs; there are two 1.0.0.1 and 1.1.1.1 in the default channel. See https://en.wikipedia.org/wiki/1.1.1.1\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayvdb%2Fhttps-everywhere-py","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjayvdb%2Fhttps-everywhere-py","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayvdb%2Fhttps-everywhere-py/lists"}