{"id":37065204,"url":"https://github.com/essteer/masquer","last_synced_at":"2026-01-14T07:37:58.029Z","repository":{"id":240354613,"uuid":"799995437","full_name":"essteer/masquer","owner":"essteer","description":"A tool to generate random user-agent and referer data for HTTP requests","archived":false,"fork":false,"pushed_at":"2025-07-26T10:20:39.000Z","size":186,"stargazers_count":2,"open_issues_count":6,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-27T23:10:40.573Z","etag":null,"topics":["header","referer","requests","user-agent","webscraping"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/masquer/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/essteer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-13T13:58:12.000Z","updated_at":"2025-07-15T10:08:24.000Z","dependencies_parsed_at":"2024-05-30T18:36:48.040Z","dependency_job_id":"8ed286c3-9b6f-464c-bf4b-5039643b5aa0","html_url":"https://github.com/essteer/masquer","commit_stats":null,"previous_names":["essteer/masquerade","essteer/masquer"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/essteer/masquer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/essteer%2Fmasquer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/essteer%2Fmasquer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/essteer%2Fmasquer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/essteer%2Fmasquer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/essteer","download_url":"https://codeload.github.com/essteer/masquer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/essteer%2Fmasquer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28413457,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T05:26:33.345Z","status":"ssl_error","status_checked_at":"2026-01-14T05:21:57.251Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["header","referer","requests","user-agent","webscraping"],"created_at":"2026-01-14T07:37:57.500Z","updated_at":"2026-01-14T07:37:58.009Z","avatar_url":"https://github.com/essteer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\" id=\"title\"\u003eMasquer\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/essteer/masquer/actions/workflows/test.yaml\"\u003e\u003cimg src=\"https://github.com/essteer/masquer/actions/workflows/test.yaml/badge.svg\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/masquer/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/masquer?style=flat\u0026logo=PyPI\u0026logoColor=white\u0026label=PyPI\u0026labelColor=555\u0026color=3776AB\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/masquer/\"\u003e\u003cimg src=\"https://img.shields.io/badge/Python-3.9_~_3.13-3776AB.svg?style=flat\u0026logo=Python\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://snyk.io/test/github/essteer/masquer\"\u003e\u003cimg src=\"https://snyk.io/test/github/essteer/masquer/badge.svg?name=Snyk\u0026style=flat\u0026logo=Snyk\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://astral.sh\"\u003e\u003cimg src=\"https://img.shields.io/badge/Astral-261230.svg?style=flat\u0026logo=Astral\u0026labelColor=555\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://hub.docker.com/r/essteer/masquer\"\u003e\u003cimg src=\"https://img.shields.io/badge/Docker-2496ED.svg?flat\u0026logo=Docker\u0026labelColor=555\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/tiangolo/fastapi\"\u003e\u003cimg src=\"https://img.shields.io/badge/FastAPI-009688?style=flat\u0026logo=FastAPI\u0026labelColor=555\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://masquer.fly.dev/docs\"\u003e\u003cimg src=\"https://img.shields.io/badge/Fly.io-24175B.svg?style=flat\u0026labelColor=555\u0026logo=flydotio\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/pypa/hatch\"\u003e\u003cimg src=\"https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\nA tool to generate random user-agent and referer data for HTTP requests.\n\u003c/p\u003e\n\n## Overview\n\nUse `masquer` to obtain any combination of a random user-agent, referer or header data template, then use this with a library like [`requests`](https://github.com/psf/requests) to control the session data you send to other services.\n\nThe user-agent data is drawn from [this list](https://www.useragents.me/) of the most common desktop user-agents, and referer data is taken from [this list](https://gs.statcounter.com/search-engine-market-share/desktop/worldwide) of search engines with the largest global market share.\n\nWeighted random selections are made from those lists to approximate authentic header data patterns.\n\nA basic header template with common attributes — like [`\"Upgrade-Insecure-Requests\": \"1\"`](https://stackoverflow.com/questions/31950470/what-is-the-upgrade-insecure-requests-http-header/32003517#32003517) — is also provided and defaults to the most common referer and user-agent data from the above lists.\n\n### Note on privacy\n\nControlling header data in this way can help to preserve privacy and hinder third-party tracking behaviour by blending part of your web profile with the most common configurations. \n\nIt does not provide anonymity — that is a much more complex topic, and the open-source [Privacy Guides](https://www.privacyguides.org/en/) are a good place to start.\n\n## Documentation\n\nThe sections that follow describe different ways to use `masquer`.\n\nFor development purposes see the `docs` directory for notes on development, deployment and Docker.\n\n- [API](#api)\n- [Python package](#python-package)\n  - [Installation](#installation)\n  - [Operation](#operation)\n  - [Examples](#examples)\n- [Git repository](#git-repository)\n- [Docker image](#docker-image)\n\n## API\n\n[![FastAPI](https://img.shields.io/badge/FastAPI-masquer-009688?style=flat\u0026logo=FastAPI\u0026logoColor=white)](https://masquer.fly.dev/docs)\n\nAn API for `masquer` is in deployment at `https://masquer.fly.dev/masq` \u0026mdash; try it out with the interactive [Swagger UI](https://masquer.fly.dev/docs) or [ReDoc](https://masquer.fly.dev/redoc) documentation.\n\nThe API returns a JSON array, making it compatible with any language that can make HTTP requests and parse JSON.\n\nHere is an example using `curl` from the command line to get a random user-agent and referer:\n\n```console\n$ curl -X GET 'https://masquer.fly.dev/api/v1/masq?ua=true\u0026rf=true' -H 'accept: application/json'\n[\n  {\n    \"Referer\":\"https://www.google.com\",\n    \"User-Agent\":\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.3\"\n  }\n]\n```\n\nThe optional `count` parameter specifies the number of objects to return in the response. The default value is `1`.\n\nRefer to the [API docs](`https://masquer.fly.dev/docs`) for other examples, or see [more details below](#examples) in the package documentation. The `tests/requests/` directory also contains HTTP request examples for use in development and production.\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"#title\"\u003e\u003cimg src=\"https://img.shields.io/badge/▲%20Top%20▲-0466c8.svg?style=flat\"\u003e\u003c/a\u003e\n\u003c/h3\u003e\n\n## Python package\n\n[![PyPI](https://img.shields.io/badge/PyPI-masquer-3775A9.svg?style=flat\u0026logo=PyPI\u0026logoColor=white)](https://pypi.org/project/masquer/)\n\n### Installation\n\nTo use the `masquer` Python package, create and activate a virtual environment then install `masquer` using a package manager.\n\nThe below example uses [Astral's](https://docs.astral.sh/uv) `uv` \u0026mdash; substitute `pip` by dropping \"`uv`\" or use another package manager as preferred:\n\n```console\n$ uv venv\n$ source .venv/bin/activate\n$ uv pip install masquer\n```\n\nThe core tool obtained this way has no dependencies.\n\n### Operation\n\nInteract with `masquer` via the `masq` method:\n\n```python\n\u003e\u003e\u003e from masquer import masq\n```\n\nThe `masq` function accepts up to three boolean parameters...\n\n```python\n\u003e\u003e\u003e useragent = masq(\n  ua = True,  # user-agent, defaults to True\n  rf = False,  # referer, defaults to False\n  hd = False  # header-data, defaults to False\n)\n```\n\n...and returns the response as a dict object:\n\n```python\n{\n  'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.'\n}\n```\n\n### Examples\n\n#### User-agent\n\nBy default only the `ua` parameter is set to `True`, so each of the following methods may be used to obtain just one randomly selected user-agent:\n\n```python\n\u003e\u003e\u003e masq()\n{\"User-Agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.3\"}\n\u003e\u003e\u003e masq(True)\n{\"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.\"}\n\u003e\u003e\u003e masq(ua=True)\n{\"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3\"}\n```\n\n#### Referer\n\nBy the same logic, the following methods will each return one randomly selected referer:\n\n```python\n\u003e\u003e\u003e masq(False, True)\n{\"Referer\": \"https://www.google.com/\"}\n\u003e\u003e\u003e masq(ua=False, rf=True)\n{\"Referer\": \"https://www.google.com/\"}\n\u003e\u003e\u003e masq(ua=False, rf=True, hd=False)\n{\"Referer\": \"https://bing.com/\"}\n```\n\n#### Header-data with default user-agent and referer\n\nThe default header-data template supplies the most common user-agent and refer values as fixed values, and can be accessed via the following methods:\n\n```python\n\u003e\u003e\u003e masq(False, False, True)\n\u003e\u003e\u003e masq(ua=False, hd=True)\n\u003e\u003e\u003e masq(ua=False, rf=False, hd=True)\n```\n\nEach of the above function calls would return the following:\n\n```python\n\u003e\u003e\u003e default_header = masq(ua=False, rf=False, hd=True)\n\u003e\u003e\u003e default_header\n{\n  \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\", \n  \"Accept-Encoding\": \"gzip, deflate, br\", \n  \"Accept-Language\": \"en-US,en;q=0.5;\", \n  \"Referer\": \"https://www.google.com/\", \n  \"Sec-Fetch-Dest\": \"document\", \n  \"Sec-Fetch-Mode\": \"navigate\", \n  \"Sec-Fetch-Site\": \"none\", \n  \"Sec-Fetch-User\": \"?1\", \n  \"Upgrade-Insecure-Requests\": \"1\", \n  \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.3\"\n}\n```\n\n#### Header-data with random user-agent and referer\n\nTo get the header-data with randomly selected user-agent and/or referer data, pass those arguments as `True` in addition to the `hd` parameter as per the below examples:\n\n```python\n\u003e\u003e\u003e random_header = masq(rf=True, hd=True)  # ua=True by default\n\u003e\u003e\u003e random_header\n{\n  # ...\n  \"Referer\": \"https://duckduckgo.com\",\n  # ...\n  \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.\"\n}\n```\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"#title\"\u003e\u003cimg src=\"https://img.shields.io/badge/▲%20Top%20▲-0466c8.svg?style=flat\"\u003e\u003c/a\u003e\n\u003c/h3\u003e\n\n## Git repository\n\n[![GitHub](https://img.shields.io/badge/GitHub-masquer-181717.svg?flat\u0026logo=GitHub\u0026logoColor=white)](https://github.com/essteer/masquer)\n\nClone the `masquer` repo for the full source code, including the FastAPI app used to host the [API](#api) introduced above. \n\nThe repo includes the JSON source files used to generate the header data, a script to update the JSON content to the latest data, and a test suite.\n\n```console\n$ git clone git@github.com:essteer/masquer\n```\n\nThe package code within the `src/masquer` directory has no dependencies beyond built-in Python modules, so can be run right away in a virtual environment.\n\nThe `update.sh` shell script in the root directory updates the `header.json` and `referer.json` files in the `assets` directory to the latest versions, then uses this data to update the `assets.py` file used by the `masquer` package.\n\nTo use the `update.sh` script first install Beautiful Soup into the virtual environment:\n\n```console\n$ source .venv/bin/activate\n$ uv pip install beautifulsoup4==4.12.3\n```\n\nThen grant execution permissions to `update.sh` and run it \u0026mdash; the output should appear similar to that displayed below:\n\n```console\n$ chmod +x update.sh\n$ ./update.sh\n2024-09-17 14:34:03 - INFO - update.py:29 - Fetched user-agent data\n2024-09-17 14:34:04 - INFO - update.py:74 - Fetched referer data\n2024-09-17 14:34:04 - INFO - update.py:133 - Saved user-agent and referer JSON data to assets.py\n2024-09-17 14:34:04 - INFO - update.sh - Asset update OK\n```\n\nIf interested in making changes to the repo, see the `development notes` for additional details.\n\n### FastAPI\n\n[![FastAPI](https://img.shields.io/badge/FastAPI-masquer-009688?style=flat\u0026logo=FastAPI\u0026logoColor=white)](https://masquer.fly.dev/docs)\n\nThe Git repo includes the [FastAPI version](#api) of `masquer` that is introduced above and hosted at [`https://masquer.fly.dev`](https://masquer.fly.dev/docs) \u0026mdash; the relevant code is located at `src/api`.\n\nTo self-host the API, install the `FastAPI` optional dependency as declared in the `pyproject.toml` file. \n\n```console\n$ source .venv/bin/activate\n$ uv pip install fastapi==0.111.0\n```\n\nFastAPI runs on localhost port `8000` by default \u0026mdash; to amend this change the relevant uvicorn argument in `src/api/main.py`:\n\n```python\nif __name__ == \"__main__\":\n    import uvicorn\n\n    uvicorn.run(app, host=\"127.0.0.1\", port=8000)\n```\n\nRun the app from the root directory via:\n\n```console\n$ fastapi run src/api/main.py\n```\n\nThen follow the instructions FastAPI provides in the terminal.\n\nTo view the API's interactive documentation, run the app and navigate to `http://127.0.0.1:8000/docs` or `http://127.0.0.1:8000/redoc`.\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"#title\"\u003e\u003cimg src=\"https://img.shields.io/badge/▲%20Top%20▲-0466c8.svg?style=flat\"\u003e\u003c/a\u003e\n\u003c/h3\u003e\n\n## Docker image\n\n[![Docker](https://img.shields.io/badge/Docker-masquer-2496ED.svg?flat\u0026logo=Docker\u0026labelColor=555\u0026logoColor=white)](https://hub.docker.com/r/essteer/masquer)\n\nA Docker image for `masquer` has been developed for production use, and is publicly available.\n\nTo run `masquer` from a container, first pull the image from [DockerHub](https://hub.docker.com/r/essteer/masquer):\n\n```console\n$ docker pull essteer/masquer\n```\n\nLaunch a container using the `masquer` image as follows:\n\n```console\n$ docker run -d --name masquer -p 8000:8000 essteer/masquer\n```\n\nThen interact as per the [API instructions](#api) above.\n\n\u003ch3 align=\"center\"\u003e\n  \u003ca href=\"#title\"\u003e\u003cimg src=\"https://img.shields.io/badge/▲%20Top%20▲-0466c8.svg?style=flat\"\u003e\u003c/a\u003e\n\u003c/h3\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fessteer%2Fmasquer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fessteer%2Fmasquer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fessteer%2Fmasquer/lists"}