{"id":39349166,"url":"https://github.com/pixelsdb/pixels-hive","last_synced_at":"2026-01-18T02:26:55.206Z","repository":{"id":40694636,"uuid":"470801287","full_name":"pixelsdb/pixels-hive","owner":"pixelsdb","description":"The Pixels serde (connector) for Hive.","archived":false,"fork":false,"pushed_at":"2025-08-14T16:48:07.000Z","size":71,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-14T18:33:20.508Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pixelsdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-17T00:59:11.000Z","updated_at":"2025-08-14T16:48:10.000Z","dependencies_parsed_at":"2024-04-11T08:49:26.220Z","dependency_job_id":null,"html_url":"https://github.com/pixelsdb/pixels-hive","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/pixelsdb/pixels-hive","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelsdb%2Fpixels-hive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelsdb%2Fpixels-hive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelsdb%2Fpixels-hive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelsdb%2Fpixels-hive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pixelsdb","download_url":"https://codeload.github.com/pixelsdb/pixels-hive/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pixelsdb%2Fpixels-hive/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28526569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-18T02:26:55.083Z","updated_at":"2026-01-18T02:26:55.181Z","avatar_url":"https://github.com/pixelsdb.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pixels-hive\nThis is the Pixels `Hive SerDe` (short for “Serializer and Deserializer.”).\nHive uses SerDe and the corresponding FileFormat to read and write tables.\n\nThe flow of deserialization is:\n```\nHDFS files –\u003e InputFileFormat –\u003e \u003ckey, value\u003e –\u003e Deserializer –\u003e Row object\n```\nWhile the flow of serialization is:\n```\nRow object –\u003e Serializer –\u003e \u003ckey, value\u003e –\u003e OutputFileFormat –\u003e HDFS files\n```\n\n## Compatibility\nPixels Serde has been tested on Hive 2.1 and 2.3. Other Hive versions that are compatible\nwith the SerDe API in Hive 2.3 should also work well with Pixels SerDe.\n\nCurrently, Pixels Hive SerDe only supports HDFS as the underlying storage.\n\n## Build\n\nThis project can be opened as a maven project in Intellij and built using maven.\n\nHowever, [Pixels](https://github.com/pixelsdb/pixels) is the parent of this project,\ntherefore use `mvn install` to install Pixels modules into your local maven repository,\nbefore building this project.\n\n## Use Pixels in Hive\n\nBuild pixels-hive using `mvn package`. Find the `pixels-hive-*-full.jar` under the `target` directory.\nThis is the SerDe that can be used in Hive.\n\nEnsure that Pixels and other prerequisites are installed following the instructions\n[HERE](https://github.com/pixelsdb/pixels/blob/master/docs/INSTALL.md). Presto is not needed as we are using Hive\ninstead as the query engine.\n\n### Load Pixels SerDe\nThere are two options to load Pixels SerDe in Hive:\n\n* In Hive cli:\n```sql\nhive\u003e add jar {PATH}/pixels-hive-*-full.jar\n```\n* You can also put `pixels-hive-*-full.jar` in the path `HIVE_HOME/auxlib/`, where `HIVE_HOME` is the installation directory of Hive.\nThen Hive will load the jar when launching.\n\n### Create Table\nCreate tables in Hive with the syntax in [Hive Language Manual](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Create/Drop/Alter/UseDatabase).\nWithin the `CREATE TABLE` statement, use the following `ROW FORMAT` and `TABLEPROPERTY`:\n```SQL\nROW FORMAT SERDE\n  'io.pixelsdb.pixels.hive.PixelsSerDe' \nSTORED AS INPUTFORMAT \n  'io.pixelsdb.pixels.hive.mapred.PixelsInputFormat' \nOUTPUTFORMAT \n  'io.pixelsdb.pixels.hive.mapred.PixelsOutputFormat'\nLOCATION\n  'hdfs://hostname:9000/path/to/pixels/files'\nTBLPROPERTIES ('bind.pixels.table'='schema_name.table_name')\n```\n`LOCATION` is necessary. It should be the same as the `order_path`\nor `compact_path` in Pixels metadata. `bind.pixels.table` specifies\nwhich table in Pixels to be bind with this Hive table. Replace the `schema_name`\nand `table_name` with the right schema and table name in Pixels.\n\n**Note**: It is a better to create an EXTERNAL table. Internal table will delete the data\nwhen the table is dropped.\n\n### Load Data\nLoad data by `pixels-cli`. Then it is ready to execute queries in Hive.\nCurrently, we have only implemented `PixelsInputFormat` for Hive,\ntherefore data can not be loaded using Hive's `LOAD` command.\nAn example of `pixels-cli` usage shown in the [TPC-H Evaluation](https://github.com/pixelsdb/pixels/blob/master/docs/TPC-H.md).\n\n### Run Queries\nBefore executing a query, set `hive.input.format` in the session:\n```sh\nset hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;\n```\nThe default `hive.input.format` is `CombineHiveInputFormat`, which\nwill not invoke our dynamic splitting algorithm that generates the input\nsplits for MapReduce jobs. Pixels SerDe currently declines to read\nand process splits generated by `CombineHiveInputSplit`.\n\nIn addition, it is better to set a smaller value for the max\nreducers in a job, such as:\n```sh\nset hive.exec.reducers.max=16\n```\nPixels is an efficient columnar store, especially for wide tables.\nIt is likely that only a very small portion of data is read from each split.\nThe default reducer number in Hive is estimated by the total size\nof the input splits `(num_reducer=min(hive.exec.reducers.max, (total_input_size/N))`, \nwhich is generally much larger than the actual number of reducers we need. \nThe default max reducers in Hive 2.x is 1099.\n\n**Note:** the logs of pixels-hive are included in the Hive log file that is located at `\\tmp\\{user_name}\\hive.log`\n by default.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpixelsdb%2Fpixels-hive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpixelsdb%2Fpixels-hive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpixelsdb%2Fpixels-hive/lists"}