{"id":13569390,"url":"https://github.com/lihuigang/hive-bitmap-udf","last_synced_at":"2025-04-04T05:32:22.703Z","repository":{"id":39450722,"uuid":"462996128","full_name":"lihuigang/hive-bitmap-udf","owner":"lihuigang","description":"在hive中使用Roaring64Bitmap实现精确去重功能","archived":false,"fork":false,"pushed_at":"2024-07-08T00:34:38.000Z","size":48,"stargazers_count":63,"open_issues_count":3,"forks_count":28,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-08-02T14:06:19.364Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lihuigang.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-24T03:34:27.000Z","updated_at":"2024-07-09T03:10:01.000Z","dependencies_parsed_at":"2023-12-30T15:25:36.091Z","dependency_job_id":"efbfb315-15d9-4166-b325-cb91ea0501c6","html_url":"https://github.com/lihuigang/hive-bitmap-udf","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lihuigang%2Fhive-bitmap-udf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lihuigang%2Fhive-bitmap-udf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lihuigang%2Fhive-bitmap-udf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lihuigang%2Fhive-bitmap-udf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lihuigang","download_url":"https://codeload.github.com/lihuigang/hive-bitmap-udf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223100121,"owners_count":17087387,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T14:00:39.420Z","updated_at":"2025-04-04T05:32:22.686Z","avatar_url":"https://github.com/lihuigang.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"# hive-bitmap-udf\n\n在hive、spark中使用Roaring64Bitmap实现精确去重功能\n主要目的：\n1. 提升 hive、spark 中精确去重性能，代替hive或Spark 中的 count(distinct uuid)；\n2. 节省 hive 存储 ，使用 bitmap 对数据压缩 ，减少了存储成本；\n3. 提供在 hive、spark 中 bitmap 的灵活运算 ，比如：交集、并集、差集运算 ，计算后的 bitmap 也可以直接写入 hive 表中；\n\n如果方便的话，还请各位帮忙点个star，为开源项目加油！\n## 1. 项目编译\n```angular2html\njava 版本：1.8\n```\n```angular2html\nmvn clean package\n```\n编译完成后使用jar包：hive-bitmap-udf.jar\n## 2. 在hive中创建UDF\n将 hive-bitmap-udf.jar 上传至HDFS系统或者放到本地，在spark或者hive中注册使用\n```\nadd jar hdfs://node:9000/hive-bitmap-udf.jar;\n\nCREATE TEMPORARY FUNCTION to_bitmap AS 'com.hive.bitmap.udf.ToBitmapUDAF';\nCREATE TEMPORARY FUNCTION bitmap_union AS 'com.hive.bitmap.udf.BitmapUnionUDAF';\n\nCREATE TEMPORARY FUNCTION bitmap_count AS 'com.hive.bitmap.udf.BitmapCountUDF';\nCREATE TEMPORARY FUNCTION bitmap_and AS 'com.hive.bitmap.udf.BitmapAndUDF';\nCREATE TEMPORARY FUNCTION bitmap_or AS 'com.hive.bitmap.udf.BitmapOrUDF';\nCREATE TEMPORARY FUNCTION bitmap_xor AS 'com.hive.bitmap.udf.BitmapXorUDF';\nCREATE TEMPORARY FUNCTION bitmap_to_array AS 'com.hive.bitmap.udf.BitmapToArrayUDF';\nCREATE TEMPORARY FUNCTION bitmap_from_array AS 'com.hive.bitmap.udf.BitmapFromArrayUDF';\nCREATE TEMPORARY FUNCTION bitmap_contains AS 'com.hive.bitmap.udf.BitmapContainsUDF';\n\n```\n\n## 3. UDF说明\n\n|        UDF        |             描述              |                案例                |     结果类型      |\n|:-----------------:|:---------------------------:|:--------------------------------:|:-------------:|\n|     to_bitmap     | 将num（int或bigint） 转化为 bitmap |          to_bitmap(num)          |    bitmap     |\n|   bitmap_union    |   多个bitmap合并为一个bitmap（并集）   |       bitmap_union(bitmap)       |    bitmap     |\n|   bitmap_count    |      计算bitmap中存储的num个数      |       bitmap_count(bitmap)       |     bigint      |\n|    bitmap_and     |        计算两个bitmap交集         |   bitmap_and(bitmap1,bitmap2)    |    bitmap     |\n|     bitmap_or     |        计算两个bitmap并集         |    bitmap_or(bitmap1,bitmap2)    |    bitmap     |\n|    bitmap_xor     |        计算两个bitmap差集         |   bitmap_xor(bitmap1,bitmap2)    |    bitmap     |\n| bitmap_from_array |  array 转化为bitmap         \t  |     bitmap_from_array(array)     |    bitmap     |\n|  bitmap_to_array  |       bitmap转化为array        |     bitmap_to_array(bitmap)      | array\u003cbigint\u003e |\n|  bitmap_contains  |   bitmap是否包含另一个bitmap全部元素   | bitmap_contains(bitmap1,bitmap2) |    boolean    |\n|  bitmap_contains  |       bitmap是否包含某个元素        |   bitmap_contains(bitmap,num)    |    boolean    |\n| bitmap_intersect  |         多个bitmap的交集         |     bitmap_intersect(bitmap)    |   bitmap  |\n\n## 4. 下载地址\nhttps://github.com/lihuigang/hive-bitmap-udf/releases\n## 5. 在 hive 中创建 bitmap 类型表,导入数据并查询\n```\nCREATE TABLE IF NOT EXISTS `hive_bitmap_table`\n( \n    k      int      comment 'id',\n    bitmap binary   comment 'bitmap'\n) comment 'hive bitmap 类型表' \nSTORED AS ORC;\n\n-- 数据写入\ninsert into table  hive_bitmap_table select  1 as id,to_bitmap(1) as bitmap;\ninsert into table hive_bitmap_table select  2 as id,to_bitmap(2) as bitmap;\n\n-- 查询\n\nselect bitmap_union(bitmap) from hive_bitmap_table;\nselect bitmap_count(bitmap_union(bitmap)) from hive_bitmap_table;\n\nselect bitmap_contains(bitmap,1) from hive_bitmap_table;\nselect bitmap_contains(bitmap,bitmap_from_array(array(1,2))) from hive_bitmap_table;\n\n\n\n```\n\n## 5. 在 hive 中使用 bitmap 实现精确去重\n```\nCREATE TABLE IF NOT EXISTS `hive_table`\n( \n    k      int      comment 'id',\n    uuid   bigint   comment '用户id'\n) comment 'hive 普通类型表' \nSTORED AS ORC;\n\n-- 普通查询（计算去重人数）\n\nselect count(distinct uuid) from hive_table;\n\n-- bitmap查询（计算去重人数）\n\nselect bitmap_count(to_bitmap(uuid)) from hive_table;\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flihuigang%2Fhive-bitmap-udf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flihuigang%2Fhive-bitmap-udf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flihuigang%2Fhive-bitmap-udf/lists"}