{"id":13414814,"url":"https://github.com/cnu/scrapy-random-useragent","last_synced_at":"2025-03-14T22:32:14.019Z","repository":{"id":25060965,"uuid":"28481093","full_name":"cnu/scrapy-random-useragent","owner":"cnu","description":"Scrapy Middleware to set a random User-Agent for every Request.","archived":false,"fork":false,"pushed_at":"2019-08-16T21:29:30.000Z","size":13,"stargazers_count":201,"open_issues_count":9,"forks_count":48,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-09-27T04:49:16.515Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cnu.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-25T12:29:23.000Z","updated_at":"2024-09-24T17:48:05.000Z","dependencies_parsed_at":"2022-08-23T12:01:59.310Z","dependency_job_id":null,"html_url":"https://github.com/cnu/scrapy-random-useragent","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnu%2Fscrapy-random-useragent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnu%2Fscrapy-random-useragent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnu%2Fscrapy-random-useragent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cnu%2Fscrapy-random-useragent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cnu","download_url":"https://codeload.github.com/cnu/scrapy-random-useragent/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243292800,"owners_count":20268126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T21:00:37.122Z","updated_at":"2025-03-14T22:32:13.750Z","avatar_url":"https://github.com/cnu.png","language":"Python","funding_links":[],"categories":["Apps","Scrapy Middleware"],"sub_categories":["Avoid Ban"],"readme":"Scrapy Random User-Agent\n========================\n\nDoes your scrapy spider get identified and blocked by servers because\nyou use the default user-agent or a generic one?\n\nUse this ``random_useragent`` module and set a random user-agent for\nevery request. You are limited only by the number of different\nuser-agents you set in a text file.\n\nInstalling\n----------\n\nInstalling it is pretty simple.\n\n.. code-block:: python\n\n    pip install scrapy-random-useragent\n\nUsage\n-----\n\nIn your ``settings.py`` file, update the ``DOWNLOADER_MIDDLEWARES``\nvariable like this.\n\n.. code-block:: python\n\n    DOWNLOADER_MIDDLEWARES = {\n        'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,\n        'random_useragent.RandomUserAgentMiddleware': 400\n    }\n\nThis disables the default ``UserAgentMiddleware`` and enables the\n``RandomUserAgentMiddleware``.\n\nThen, create a new variable ``USER_AGENT_LIST`` with the path to your\ntext file which has the list of all user-agents\n(one user-agent per line).\n\n.. code-block:: python\n\n    USER_AGENT_LIST = \"/path/to/useragents.txt\"\n\nNow all the requests from your crawler will have a random user-agent\npicked from the text file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnu%2Fscrapy-random-useragent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcnu%2Fscrapy-random-useragent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcnu%2Fscrapy-random-useragent/lists"}