https://github.com/govau/warcraider

Convert WARC files into Avro for big data processing
https://github.com/govau/warcraider

avro bigquery crawler rust warc

Last synced: 13 days ago
JSON representation

Convert WARC files into Avro for big data processing

• Host: GitHub
• URL: https://github.com/govau/warcraider
• Owner: govau
• License: mit
• Created: 2019-06-09T02:13:13.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2020-02-11T12:04:30.000Z (almost 5 years ago)
• Last Synced: 2024-11-20T19:12:27.537Z (3 months ago)
• Topics: avro, bigquery, crawler, rust, warc
• Language: HTML
• Homepage: https://data.gov.au/dataset/ds-dga-99f43557-1d3d-40e7-bc0c-665a4275d625/details
• Size: 181 KB
• Stars: 0
• Watchers: 4
• Forks: 0
• Open Issues: 9

Awesome Lists containing this project