{"id":27415817,"url":"https://github.com/helgeho/hadoopconcatgz","last_synced_at":"2025-04-14T09:39:36.813Z","repository":{"id":67842534,"uuid":"65206923","full_name":"helgeho/HadoopConcatGz","owner":"helgeho","description":"A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz","archived":false,"fork":false,"pushed_at":"2018-02-07T14:31:16.000Z","size":53,"stargazers_count":9,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-08T03:34:33.286Z","etag":null,"topics":["hadoop","spark","warc","web-archiving","webarchive"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/helgeho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-08-08T13:36:13.000Z","updated_at":"2021-04-07T00:20:09.000Z","dependencies_parsed_at":null,"dependency_job_id":"c2747443-dde6-42a1-ab0c-61677bdf25a6","html_url":"https://github.com/helgeho/HadoopConcatGz","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helgeho%2FHadoopConcatGz","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helgeho%2FHadoopConcatGz/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helgeho%2FHadoopConcatGz/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/helgeho%2FHadoopConcatGz/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/helgeho","download_url":"https://codeload.github.com/helgeho/HadoopConcatGz/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248855263,"owners_count":21172523,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hadoop","spark","warc","web-archiving","webarchive"],"created_at":"2025-04-14T09:39:36.379Z","updated_at":"2025-04-14T09:39:36.804Z","avatar_url":"https://github.com/helgeho.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A Splitable Hadoop InputFormat for Concatenated GZIP Files\n\nGZIP compressed files are usually considered to be non-splitable.\nThis makes them a rather impractical input format for distributed processing / [Hadoop](http://hadoop.apache.org), which stores its files in a distributed filesystem and exploits data locality to gain efficiency.\nHowever, a GZIP file may be comprised of multiple concatenated GZIP records, which constitutes a valid GZIP file again.\nSuch compressed files can be split by identifying the start of a new record and this way, can be used by Hadoop much more efficiently.\n\nOne application that heavily uses concatenated GZIP files is Web archives.\nThe standardized data format [WARC](https://en.wikipedia.org/wiki/Web_ARChive) is typically stored in GZIP files with every record being a separate GZIP record.\n\nWe provide both, one general input format for concatenated GZIP files (*ConcatGzipInputFormat*) as well as a specialized one for \\*.warc.gz files, i.e., WARC records stored in concatenated GZIP files (*WarcGzInputFormat*). The *WarcGzInputFormat* also serves as an example to show how to easily implement custom input formats based on concatenated GZIP files.\n\n### [ConcatGzipInputFormat](src/main/java/de/l3s/concatgz/io/ConcatGzipInputFormat.java)\n\nIf configured as input format in your Hadoop job, the key in your mapper will be of *Text* and the value of type *FileBackedBytesWritable*, which is based on Google's *[com.google.common.io.FileBackedOutputStream](https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/FileBackedOutputStream.html)*.\nThe key contains the filname as well as the position of the current value in the form *filename:position*.\nThe value contains the current GZIP record as bytes, which can be accessed by calling *value.getBytes()* as *[com.google.common.io.ByteSource](https://google.github.io/guava/releases/19.0/api/docs/com/google/common/io/ByteSource.html)*.\nTo decompress the contents you can read the bytes through input stream:\n\n```java\nByteArrayInputStream in = value.getBytes().openBufferedStream();\nGZIPInputStream decompressed = new GZIPInputStream(in);\n...\ndecompressed.read();\n...\ndecompressed.close();\n```\n\n### [WarcGzInputFormat](src/main/java/de/l3s/concatgz/io/warc/WarcGzInputFormat.java)\n\nThis input format gives you only a value of type *[WarcWritable](src/main/java/de/l3s/concatgz/io/warc/WarcWritable.java)*, with the key being of type *NullWritable*.\n*WarcWritable* provides access to the raw (compressed) bytes, the filename, the offset as well as a wrapped *[WarcRecord](src/main/java/de/l3s/concatgz/data/WarcRecord.java)* with convenient getter methods to read headers, contents and parsed HTTP responses.\n\n```java\n...\nString file = value.getFilename();\nlong offset = value.getOffset();\nWarcRecord warc = value.getRecord();\n...\nArchiveRecord record = warc.getRecord();\nArchiveRecordHeader header = warc.getHeader();\nif (warc.isHttp()) {\n    Map\u003cString, String\u003e httpHeaders = warc.getHttpHeaders();\n    byte[] body = warc.getHttpBody();\n    String bodyString = warc.getHttpStringBody();\n    String mime = warc.getHttpMimeType();\n    ...\n}\n...\n```\n\n### Output\n\nTo facilitate the output with Hadoop and allow for easy creation of GZIP records and concatenated GZIP files, we provide some helper classes:\n\n[*ImmediateOutput*](src/main/java/de/l3s/concatgz/io/ImmediateOutput.java) enables you to write out data directly to HDFS without using Hadoop output mechanisms.\nIt takes care of a unique naming per task and overwrites files in case of failed and rerun tasks.\nThis is how you configure your Hadoop job to use *ImmediateOutput*:\n```java\nJob job = Job.getInstance(config);\nImmediateOutput.initialize(job);\nImmediateOutput.setPath(job, outPath);\nImmediateOutput.setExtension(job, \".gz\");\nImmediateOutput.setReplication(job, (short) 2);\n```\n\nNow, to use it in your mapper / reducer, you need to create an instance as follows:\n\n```java\nprivate ImmediateOutput output;\n\n@Override\nprotected void setup(Context context) throws IOException, InterruptedException {\n    output = new ImmediateOutput(context, true);\n}\n\n@Override\npublic void cleanup(Context context) throws IOException, InterruptedException {\n    output.close();\n}\n```\n\n[*GZipBytes*](src/main/java/de/l3s/concatgz/util/GZipBytes.java) lets you output any kind of data into a GZIP compressed byte array, that you can write out using your *ImmediateOutput*:\n\n```java\nGZipBytes gzip = new GZipBytes();\n...\nOutputStream out = gzip.open();\nout.write(...);\n...\nbyte[] bytes = out.close();\noutput.write(bytes, \"location/relative/to/outPath/raw\");\n...\nDataOutputStream data = gzip.openData();\ndata.writeInt(123);\n...\nbytes = data.close();\noutput.write(bytes, \"location/relative/to/outPath/data\");\n...\nPrintStream print = gzip.openPrint();\nprint.println(\"text\");\n...\nbytes = print.close();\noutput.write(bytes, \"location/relative/to/outPath/text\");\n...\n```\n\n## Build\n\nThe easiest way to build a JAR file of this project that you can add to your Hadoop classpath is to use Maven:\n\n`mvn package`\n\n## Maven\n\nIf you want to add this library as a dependency to your project, you can get it from the [Internet Archive](http://archive.org)'s build server under http://builds.archive.org/maven2/de/l3s/hadoop-concat-gz (thanks for hosting this!).\nThe artifact is also available through [Maven Central](https://search.maven.org/#artifactdetails|com.github.helgeho|hadoop-concat-gz|1.2|jar) (may not be the latest release).\n\nIn order to get it from the Internet Archive build server in your [Spark](http://spark.apache.org) project, add this as a new resolver:\n`resolvers ++= Seq(\"internetarchive\" at \"http://builds.archive.org/maven2\")`\n\nNow you can add the dependency as follows (please check for the newest version):\n```scala\nlibraryDependencies ++= Seq(\"com.github.helgeho\" % \"hadoop-concat-gz\" % \"1.2\")\n```\n\nFinally, load your WARC dataset into an RDD:\n```scala\nval rdd = sc.newAPIHadoopFile(INPUT_PATH, classOf[ConcatGzipInputFormat], classOf[Text], classOf[FileBackedBytesWritable\n```\n\n## License\n\nThe MIT License (MIT)\n\nCopyright (c) 2016 Helge Holzmann (L3S)\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelgeho%2Fhadoopconcatgz","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhelgeho%2Fhadoopconcatgz","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhelgeho%2Fhadoopconcatgz/lists"}