https://github.com/centic9/commoncrawldocumentdownload

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

cdx-files commoncrawl java mime-types warc

Last synced: 07 Apr 2025