{"id":13414803,"url":"https://github.com/alecxe/scrapy-fake-useragent","last_synced_at":"2025-05-15T13:05:35.650Z","repository":{"id":24471897,"uuid":"27875640","full_name":"alecxe/scrapy-fake-useragent","owner":"alecxe","description":"Random User-Agent middleware based on fake-useragent","archived":false,"fork":false,"pushed_at":"2023-09-18T18:29:37.000Z","size":56,"stargazers_count":694,"open_issues_count":9,"forks_count":98,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-04-14T20:58:16.244Z","etag":null,"topics":["python","scrapy","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alecxe.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGELOG.rst","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-12-11T14:38:20.000Z","updated_at":"2025-04-02T02:40:09.000Z","dependencies_parsed_at":"2024-06-18T13:55:25.749Z","dependency_job_id":"86456946-e838-40fc-853d-0a4af221ad44","html_url":"https://github.com/alecxe/scrapy-fake-useragent","commit_stats":{"total_commits":70,"total_committers":10,"mean_commits":7.0,"dds":0.5857142857142856,"last_synced_commit":"f67b70eeb4e642d06475c831af530cb31b44d69d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecxe%2Fscrapy-fake-useragent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecxe%2Fscrapy-fake-useragent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecxe%2Fscrapy-fake-useragent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alecxe%2Fscrapy-fake-useragent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alecxe","download_url":"https://codeload.github.com/alecxe/scrapy-fake-useragent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254346624,"owners_count":22055808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scrapy","web-scraping"],"created_at":"2024-07-30T21:00:36.987Z","updated_at":"2025-05-15T13:05:35.630Z","avatar_url":"https://github.com/alecxe.png","language":"Python","funding_links":[],"categories":["Apps","Scrapy Middleware"],"sub_categories":["Avoid Ban"],"readme":".. image:: https://travis-ci.org/alecxe/scrapy-fake-useragent.svg?branch=master\n    :target: https://travis-ci.org/alecxe/scrapy-fake-useragent\n\n.. image:: https://codecov.io/gh/alecxe/scrapy-fake-useragent/branch/master/graph/badge.svg\n  :target: https://codecov.io/gh/alecxe/scrapy-fake-useragent\n\n.. image:: https://img.shields.io/pypi/pyversions/scrapy-fake-useragent.svg\n     :target: https://pypi.python.org/pypi/scrapy-fake-useragent\n     :alt: PyPI version\n\n.. image:: https://badge.fury.io/py/scrapy-fake-useragent.svg\n     :target: http://badge.fury.io/py/scrapy-fake-useragent\n     :alt: PyPI version\n\n.. image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master\n     :target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=master\n     :alt: Requirements Status\n\n.. image:: https://img.shields.io/badge/license-MIT-blue.svg\n     :target: https://github.com/alecxe/scrapy-fake-useragent/blob/master/LICENSE.txt\n     :alt: Package license\n\nscrapy-fake-useragent\n=====================\n\nRandom User-Agent middleware for Scrapy scraping framework based on\n`fake-useragent \u003chttps://pypi.python.org/pypi/fake-useragent\u003e`__, which picks up ``User-Agent`` strings \nbased on `usage statistics \u003chttp://www.w3schools.com/browsers/browsers_stats.asp\u003e`__\nfrom a `real world database \u003chttp://useragentstring.com/\u003e`__, but also has the option to configure a generator\nof fake UA strings, as a backup, powered by \n`Faker \u003chttps://faker.readthedocs.io/en/stable/providers/faker.providers.user_agent.html\u003e`__.\n\nIt also has the possibility of extending the\ncapabilities of the middleware, by adding your own providers.\n\nChanges\n----------\n\nPlease see `CHANGELOG`_.\n\nInstallation\n-------------\n\nThe simplest way is to install it via `pip`:\n\n    pip install scrapy-fake-useragent\n\nConfiguration\n-------------\n\nTurn off the built-in ``UserAgentMiddleware`` and ``RetryMiddleware`` and add\n``RandomUserAgentMiddleware`` and ``RetryUserAgentMiddleware``.\n\nIn Scrapy \u003e=1.0:\n\n.. code:: python\n\n    DOWNLOADER_MIDDLEWARES = {\n        'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,\n        'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,\n        'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,\n        'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,\n    }\n\nIn Scrapy \u003c1.0:\n\n.. code:: python\n\n    DOWNLOADER_MIDDLEWARES = {\n        'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,\n        'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': None,\n        'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,\n        'scrapy_fake_useragent.middleware.RetryUserAgentMiddleware': 401,\n    }\n\nRecommended setting (1.3.0+):\n\n.. code:: python\n\n    FAKEUSERAGENT_PROVIDERS = [\n        'scrapy_fake_useragent.providers.FakeUserAgentProvider',  # this is the first provider we'll try\n        'scrapy_fake_useragent.providers.FakerProvider',  # if FakeUserAgentProvider fails, we'll use faker to generate a user-agent string for us\n        'scrapy_fake_useragent.providers.FixedUserAgentProvider',  # fall back to USER_AGENT value\n    ]\n    USER_AGENT = '\u003cyour user agent string which you will fall back to if all other providers fail\u003e'\n\n----------------\n\nAdditional configuration information\n====================================\n\nEnabling providers\n---------------------------\n\nThe package comes with a thin abstraction layer of User-Agent providers, which for purposes of backwards compatibility defaults to:\n\n.. code:: python\n\n    FAKEUSERAGENT_PROVIDERS = [\n        'scrapy_fake_useragent.providers.FakeUserAgentProvider'\n    ]\n\nThe package has also ``FakerProvider`` (powered by `Faker library \u003chttps://faker.readthedocs.io/\u003e`__) and ``FixedUserAgentProvider`` implemented and available for use if needed.\n\nEach provider is enabled individually, and used in the order they are defined.\nIn case a provider fails execute (for instance, it can `happen \u003chttps://github.com/hellysmile/fake-useragent/issues/99\u003e`__ to fake-useragent because of it's dependency\nwith an online service), the next one will be used.\n\nExample of what ``FAKEUSERAGENT_PROVIDERS`` setting may look like in your case:\n\n.. code:: python\n\n    FAKEUSERAGENT_PROVIDERS = [\n        'scrapy_fake_useragent.providers.FakeUserAgentProvider',\n        'scrapy_fake_useragent.providers.FakerProvider',\n        'scrapy_fake_useragent.providers.FixedUserAgentProvider',\n        'mypackage.providers.CustomProvider'\n    ]\n\n\nConfiguring fake-useragent\n---------------------------\n\nParameter: ``FAKE_USERAGENT_RANDOM_UA_TYPE`` defaulting to ``random``.\n\nOther options, as example: \n\n* ``firefox`` to mimic only Firefox browsers\n* ``msie`` to mimic Internet Explorer only\n* etc.\n\nYou can also set the ``FAKEUSERAGENT_FALLBACK`` option, which is a ``fake-useragent`` specific fallback. For example:\n\n.. code:: python\n\n    FAKEUSERAGENT_FALLBACK = 'Mozilla/5.0 (Android; Mobile; rv:40.0)'\n\nWhat it does is, if the selected ``FAKE_USERAGENT_RANDOM_UA_TYPE`` fails to retrieve a UA, it will use\nthe type set in ``FAKEUSERAGENT_FALLBACK``.\n\nConfiguring faker\n---------------------------\n\nParameter: ``FAKER_RANDOM_UA_TYPE`` defaulting to ``user_agent`` which is the way of selecting totally random User-Agents values.\nOther options, as example:\n\n* ``chrome``\n* ``firefox``\n* ``safari``\n* etc. (please refer to `Faker UserAgent provider documentation \u003chttps://faker.readthedocs.io/en/master/providers/faker.providers.user_agent.html\u003e`_ for the available options)\n\nConfiguring FixedUserAgent\n---------------------------\n\nIt also comes with a fixed provider (only provides one user agent), reusing the Scrapy's default ``USER_AGENT`` setting value.\n\nUsage with `scrapy-proxies`\n---------------------------\n\nTo use with middlewares of random proxy such as `scrapy-proxies \u003chttps://github.com/aivarsk/scrapy-proxies\u003e`_, you need:\n\n1. set ``RANDOM_UA_PER_PROXY`` to True to allow switch per proxy\n\n2. set priority of ``RandomUserAgentMiddleware`` to be greater than ``scrapy-proxies``, so that proxy is set before handle UA\n\nLicense\n----------\n\nThe package is under MIT license. Please see `LICENSE`_.\n\n.. |GitHub version| image:: https://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent.svg\n   :target: http://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent\n.. |Requirements Status| image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master\n   :target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=master\n.. _LICENSE: https://github.com/alecxe/scrapy-fake-useragent/blob/master/LICENSE.txt\n.. _CHANGELOG: https://github.com/alecxe/scrapy-fake-useragent/blob/master/CHANGELOG.rst\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falecxe%2Fscrapy-fake-useragent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falecxe%2Fscrapy-fake-useragent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falecxe%2Fscrapy-fake-useragent/lists"}