{"id":19237486,"url":"https://github.com/webrecorder/cdxj-indexer","last_synced_at":"2025-04-07T16:17:50.045Z","repository":{"id":37271633,"uuid":"98106620","full_name":"webrecorder/cdxj-indexer","owner":"webrecorder","description":"CDXJ Indexing of WARC/ARCs","archived":false,"fork":false,"pushed_at":"2024-12-10T21:38:25.000Z","size":80,"stargazers_count":25,"open_issues_count":11,"forks_count":13,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-03-28T03:24:26.131Z","etag":null,"topics":["warc","web-archiving"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/webrecorder.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"webrecorder"}},"created_at":"2017-07-23T15:18:22.000Z","updated_at":"2025-01-24T06:11:36.000Z","dependencies_parsed_at":"2024-06-21T03:01:39.826Z","dependency_job_id":"d0ac7f2a-4330-4688-b91a-1ca8476f464d","html_url":"https://github.com/webrecorder/cdxj-indexer","commit_stats":{"total_commits":39,"total_committers":6,"mean_commits":6.5,"dds":0.3076923076923077,"last_synced_commit":"7d9e56b26adab142c08e841090aebf2c0670d223"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webrecorder%2Fcdxj-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webrecorder%2Fcdxj-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webrecorder%2Fcdxj-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/webrecorder%2Fcdxj-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/webrecorder","download_url":"https://codeload.github.com/webrecorder/cdxj-indexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247394816,"owners_count":20932010,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["warc","web-archiving"],"created_at":"2024-11-09T16:26:58.127Z","updated_at":"2025-04-07T16:17:50.024Z","avatar_url":"https://github.com/webrecorder.png","language":"Python","funding_links":["https://github.com/sponsors/webrecorder"],"categories":[],"sub_categories":[],"readme":"CDXJ Indexer\n~~~~~~~~~~~~\n\nA command-line tool for generating CDXJ (and  CDX) indexes from WARC and ARC files.\nThe indexer is a new tool redesigned for fast and flexible indexing. (Based on the indexing functionality from `pywb \u003chttps://github.com/ikreymer/pywb\u003e`_)\n\nInstall with ``pip install cdxj-indexer`` or install locally with ``python setup.py install``\n\n\nThe indexer supports classic CDX index format as well as the more flexible CDXJ. With CDXJ, the indexer supports custom fields and ``request`` record access for WARC files. See the examples below and the command line ``-h`` option for latest features. (This is a work in progress).\n\n\nUsage examples\n~~~~~~~~~~~~~~~~~~~~\n\nGenerate CDXJ index:\n\n.. code:: console\n\n    \u003e cdxj-indexer /path/to/archive-file.warc.gz\n    com,example)/ 20170730223850 {\"url\": \"http://example.com/\", \"mime\": \"text/html\", \"status\": \"200\", \"digest\": \"G7HRM7BGOKSKMSXZAHMUQTTV53QOFSMK\", \"length\": \"1219\", \"offset\": \"771\", \"filename\": \"example-20170730223917.warc.gz\"}\n\n\nCDX Index (11 field):\n\n.. code:: console\n\n    \u003e cdxj-indexer -11 /path/to/archive-file.warc.gz\n    CDX N b a m s k r M S V g\n    com,example)/ 20170730223850 http://example.com/ text/html 200 G7HRM7BGOKSKMSXZAHMUQTTV53QOFSMK - - 1219 771 example-20170730223917.warc.gz\n\n\nMore advanced use cases: add additional http headers as fields. ``http:`` prefix specifies current record headers, while ``req.http:`` specifies corresponding request record headers. The following adds the Date, Referer headers, and the request method to the index:\n\n.. code:: console\n\n    \u003e cdxj-indexer -f req.http:method,http:date,req.http:referer /path/to/archive-file.warc.gz\n    com,example)/ 20170801032435 {\"url\": \"http://example.com/\", \"mime\": \"text/html\", \"status\": \"200\", \"digest\": \"A6DESOVDZ3WLYF57CS5E4RIC4ARPWRK7\", \"length\": \"1207\", \"offset\": \"834\", \"filename\": \"temp-20170801032445.warc.gz\", \"req.http:method\": \"GET\", \"http:date\": \"Tue, 01 Aug 2017 03:24:35 GMT\", \"referrer\": \"https://webrecorder.io/temp-NU34HBNO/temp/recording-session/record/http://example.com/\"}\n    org,iana)/domains/example 20170801032437 {\"url\": \"http://www.iana.org/domains/example\", \"mime\": \"text/html\", \"status\": \"302\", \"digest\": \"RP3Y66FDBYBZKSFYQ4VJ4RMDA5BPDJX2\", \"length\": \"675\", \"offset\": \"2652\", \"filename\": \"temp-20170801032445.warc.gz\", \"req.http:method\": \"GET\", \"http:date\": \"Tue, 01 Aug 2017 02:35:05 GMT\", \"referrer\": \"http://example.com/\"}\n\n\nThe CDXJ Indexer extends the ``Indexer`` functionality in `warcio \u003chttps://github.com/webrecorder/warcio\u003e`_ and should be flexible to extend.\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebrecorder%2Fcdxj-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwebrecorder%2Fcdxj-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwebrecorder%2Fcdxj-indexer/lists"}