{"id":15460925,"url":"https://github.com/machawk1/warcreader","last_synced_at":"2025-07-21T16:34:45.584Z","repository":{"id":68435992,"uuid":"154358513","full_name":"machawk1/warcreader","owner":"machawk1","description":"Read HTTP responses from web archieve files","archived":false,"fork":false,"pushed_at":"2018-10-23T16:11:05.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-24T09:45:21.242Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/machawk1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-23T16:07:47.000Z","updated_at":"2020-10-28T15:09:09.000Z","dependencies_parsed_at":"2023-07-02T19:05:24.099Z","dependency_job_id":null,"html_url":"https://github.com/machawk1/warcreader","commit_stats":{"total_commits":27,"total_committers":2,"mean_commits":13.5,"dds":"0.11111111111111116","last_synced_commit":"8d46a6fb0a2a755c21f6012294794ce23b4ff305"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fwarcreader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fwarcreader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fwarcreader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/machawk1%2Fwarcreader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/machawk1","download_url":"https://codeload.github.com/machawk1/warcreader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246634405,"owners_count":20809235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T23:40:18.261Z","updated_at":"2025-04-01T11:34:43.616Z","avatar_url":"https://github.com/machawk1.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# WarcReader\r\n\r\nWarcReader is as Python library for reading HTTP responses from [Web ARChive \r\n(WARC) files](https://en.wikipedia.org/wiki/Web_ARChive).\r\n\r\nIts main goal is to be as fast as possible, not to provide advanced functions\r\nto work with WARC files.\r\n\r\n## Authors\r\n\r\nMilos Svana (milos.svana(at)gmail.com)\r\n\r\nThis library was created as a part of my Bachelor's thesis at the\r\n[Knowledge Technology Research Group](http://knot.fit.vutbr.cz/), \r\nFaculty of Information technology, Brno University of Technology.\r\n\r\nThis library is released under Apache 2.0 licence\r\n\r\n## Documentation\r\n\r\n### Installation\r\n\r\nYou can use `pip` or `pip3` utility to install the library:\r\n\r\n```\r\npip install warcreader\r\n```\r\n\r\nor you can just download the repository contents and copy the `warcreader` directory\r\nto your project\r\n\r\n### WarcFile\r\n\r\n`WarcFile` class represents a WARC archive to be read.\r\n\r\nAccepts one parameter on initialization. Its value should be an opened file\r\ncontaining the WARC archive. It can be an instance of `file` type created by \r\n`open()` function or any other file-like object like `gzip.GzipFile` or \r\n`lzma.LZMAFile` instance.\r\n\r\n**The file has to be opened in binary mode** \r\n(letter 'b' has to be added to the `mode` parameter string)\r\n\r\n#### Iteration trough WARC records\r\n\r\n`WarcFile` instances are iterable. They return next HTTP response as `Webpage` \r\ninstance on each iteration.\r\n\r\n#### get_warcinfo()\r\n\r\nThis function returns the `warcinfo` record as a single string (bytes string in Python 3)\r\ninclucing WARC headers. Returns `None` if this type of record is not found. Only\r\nsearches for the `warcinfo` record in the beginning of the file. If other type of record \r\nis found, the search is stopped.\r\n\r\n### Webpage\r\n\r\n`Webpage` class represents one HTTP response from WARC archive. Does not\r\nprovide any methods, only the following attributes:\r\n\r\n- `uri` - absolute URI of the HTTP response\r\n- `content_type` - value of `Content-Type` field of HTTP header. `None` if this field is not found\r\n- `payload` - contents of the HTTP response like HTML source core of the the web page\r\n- `warc_record` - raw warc record as read from the archive\r\n\r\n### Example\r\n\r\n```python\r\nfrom warcreader import WarcFile\r\nfrom gzip import GzipFile\r\n\r\nwarc_gzip = GzipFile('/path/to/warc/file', 'rb')\r\nwarc_file = WarcFile(warc_gzip)\r\nfor webpage in warc_file:\r\n\tprint(webpage.uri)\r\n```\r\n\r\n## Benchmarks\r\n\r\n### Testing setup\r\n\r\nTested on Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz only using one core. Data were are stored on a disk array containing 12 4TB hard drives in RAID 6 and SSD cache. \r\n\r\n### Test script \r\n\r\n```python\r\nfrom warcreader import WarcFile\r\nfrom gzip import GzipFile\r\nfrom sys import argv\r\n\r\nif __name__ == '__main__':\r\n    with GzipFile(argv[1], mode='rb') as gzip_file:\r\n        warc_file = WarcFile(gzip_file)\r\n        for webpage in warc_file:\r\n            print(webpage.uri)\r\n```\r\n\r\n### Commoncrawl (CC-2015-48)\r\n\r\n| File name                                     | File size    | Time Python 2.7  | Time Python 3 |\r\n|-----------------------------------------------|--------------|------------------|---------------|\r\n| 1448398444047.40_20151124205404-00010-warc.gz | 861MB        | 2m2.715s         | 3m43.404s     |\r\n| 1448398444047.40_20151124205404-00021-warc.gz | 873MB        | 2m8.732s         | 3m59.925s     |\r\n| 1448398444047.40_20151124205404-00032-warc.gz | 880MB        | 2m7.905s         | 4m26.469s     | \r\n| 1448398444047.40_20151124205404-00043-warc.gz | 880MB        | 2m3.966s         | 3m50.878s     |\r\n| 1448398444047.40_20151124205404-00054-warc.gz | 870MB        | 2m13.064s        | 4m10.171s     |\r\n\r\n\r\n### Clueweb9\r\n\r\n| File name                                     | File size    | Time Python 2.7  | Time Python 3 |\r\n|-----------------------------------------------|--------------|------------------|---------------|\r\n| cw_en0035_27.warc.gz                          | 161MB        | 0m37.090s        | 0m43.223s     |\r\n| cw_en0035_32.warc.gz                          | 151MB        | 0m27.869s        | 0m31.620s     |\r\n| cw_en0035_37.warc.gz                          | 153MB        | 0m30.470s        | 0m33.357s     | \r\n| cw_en0035_42.warc.gz                          | 155MB        | 0m32.795s        | 0m35.594s     |\r\n| cw_en0035_47.warc.gz                          | 138MB        | 0m29.109s        | 0m32.739s     |\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmachawk1%2Fwarcreader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmachawk1%2Fwarcreader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmachawk1%2Fwarcreader/lists"}