{"id":13414815,"url":"https://github.com/TeamHG-Memex/scrapy-crawl-once","last_synced_at":"2025-03-14T22:32:09.244Z","repository":{"id":53733318,"uuid":"83734016","full_name":"TeamHG-Memex/scrapy-crawl-once","owner":"TeamHG-Memex","description":"Scrapy middleware which allows to crawl only new content","archived":false,"fork":false,"pushed_at":"2022-10-31T19:28:53.000Z","size":15,"stargazers_count":79,"open_issues_count":6,"forks_count":23,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-09-20T06:13:17.129Z","etag":null,"topics":["scrapy"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TeamHG-Memex.png","metadata":{"files":{"readme":"README.rst","changelog":"CHANGES.rst","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-02T23:07:01.000Z","updated_at":"2024-08-24T21:17:44.000Z","dependencies_parsed_at":"2023-01-22T04:02:01.702Z","dependency_job_id":null,"html_url":"https://github.com/TeamHG-Memex/scrapy-crawl-once","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeamHG-Memex%2Fscrapy-crawl-once","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeamHG-Memex%2Fscrapy-crawl-once/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeamHG-Memex%2Fscrapy-crawl-once/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TeamHG-Memex%2Fscrapy-crawl-once/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TeamHG-Memex","download_url":"https://codeload.github.com/TeamHG-Memex/scrapy-crawl-once/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243658055,"owners_count":20326459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scrapy"],"created_at":"2024-07-30T21:00:37.151Z","updated_at":"2025-03-14T22:32:08.882Z","avatar_url":"https://github.com/TeamHG-Memex.png","language":"Python","funding_links":[],"categories":["Apps","Scrapy Middleware","Python"],"sub_categories":["Other Useful Extensions"],"readme":"scrapy-crawl-once\n=================\n\n.. image:: https://img.shields.io/pypi/v/scrapy-crawl-once.svg\n   :target: https://pypi.python.org/pypi/scrapy-crawl-once\n   :alt: PyPI Version\n\n.. image:: https://travis-ci.org/TeamHG-Memex/scrapy-crawl-once.svg?branch=master\n   :target: http://travis-ci.org/TeamHG-Memex/scrapy-crawl-once\n   :alt: Build Status\n\n.. image:: http://codecov.io/github/TeamHG-Memex/scrapy-crawl-once/coverage.svg?branch=master\n   :target: http://codecov.io/github/TeamHG-Memex/scrapy-crawl-once?branch=master\n   :alt: Code Coverage\n\nThis package provides a Scrapy_ middleware which allows to avoid re-crawling\npages which were already downloaded in previous crawls.\n\n.. _Scrapy: https://scrapy.org/\n\nLicense is MIT.\n\nInstallation\n------------\n\n::\n\n    pip install scrapy-crawl-once\n\nUsage\n-----\n\nTo enable it, modify your settings.py::\n\n    SPIDER_MIDDLEWARES = {\n        # ...\n        'scrapy_crawl_once.CrawlOnceMiddleware': 100,\n        # ...\n    }\n\n    DOWNLOADER_MIDDLEWARES = {\n        # ...\n        'scrapy_crawl_once.CrawlOnceMiddleware': 50,\n        # ...\n    }\n\nBy default it does nothing. To avoid crawling a particular page\nmultiple times set ``request.meta['crawl_once'] = True``. When a response\nis received and a callback is successful, the fingerprint of such request\nis stored to a database. When spider schedules a new request middleware\nfirst checks if its fingerprint is in the database, and drops the request\nif it is there.\n\nOther ``request.meta`` keys:\n\n* ``crawl_once_value`` - a value to store in DB. By default, timestamp\n  is stored.\n* ``crawl_once_key`` - request unique id; by default request_fingerprint\n  is used.\n\nSettings\n--------\n\n* ``CRAWL_ONCE_ENABLED`` - set it to False to disable middleware.\n  Default is True.\n* ``CRAWL_ONCE_PATH`` - a path to a folder with crawled requests database.\n  By default ``.scrapy/crawl_once/`` path inside a project dir is used;\n  this folder contains ``\u003cspider_name\u003e.sqlite`` files with databases of\n  seen requests.\n* ``CRAWL_ONCE_DEFAULT`` - default value for ``crawl_once`` meta key\n  (False by default). When True, all requests are handled by\n  this middleware unless disabled explicitly using\n  ``request.meta['crawl_once'] = False``.\n\nAlternatives\n------------\n\nhttps://github.com/scrapy-plugins/scrapy-deltafetch is a similar package; it\ndoes almost the same. Differences:\n\n* scrapy-deltafetch chooses whether to discard a request or not based on\n  yielded items; scrapy-crawl-once uses an explicit\n  ``request.meta['crawl_once']`` flag.\n* scrapy-deltafetch uses bsddb3, scrapy-crawl-once uses sqlite.\n\nAnother alternative is a built-in `Scrapy HTTP cache`_. Differences:\n\n* scrapy cache stores all pages on disc, scrapy-crawl-once only keeps request\n  fingerprints;\n* scrapy cache allows a more fine grained invalidation consistent with how\n  browsers work;\n* with scrapy cache all pages are still processed (though not all pages are\n  downloaded).\n\n.. _Scrapy HTTP cache: https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpcache\n\nContributing\n------------\n\n* source code: https://github.com/TeamHG-Memex/scrapy-crawl-once\n* bug tracker: https://github.com/TeamHG-Memex/scrapy-crawl-once/issues\n\nTo run tests, install tox_ and run ``tox`` from the source checkout.\n\n.. _tox: https://tox.readthedocs.io/en/latest/\n\n----\n\n.. image:: https://hyperiongray.s3.amazonaws.com/define-hg.svg\n    :target: https://www.hyperiongray.com/?pk_campaign=github\u0026pk_kwd=scrapy-crawl-once\n    :alt: define hyperiongray\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTeamHG-Memex%2Fscrapy-crawl-once","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FTeamHG-Memex%2Fscrapy-crawl-once","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FTeamHG-Memex%2Fscrapy-crawl-once/lists"}