{"id":13488787,"url":"https://github.com/xiaohei-info/Spark_DB_Connector","last_synced_at":"2025-03-28T01:37:54.904Z","repository":{"id":105574912,"uuid":"85906834","full_name":"xiaohei-info/Spark_DB_Connector","owner":"xiaohei-info","description":"Use Scala API to read/write data from different databases,HBase,MySQL,etc.","archived":false,"fork":false,"pushed_at":"2018-02-28T07:02:37.000Z","size":213,"stargazers_count":24,"open_issues_count":4,"forks_count":8,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-18T09:16:24.662Z","etag":null,"topics":["connector","database","hbase","mysql","scala","spark-database"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xiaohei-info.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-03-23T04:23:13.000Z","updated_at":"2024-09-02T05:38:51.000Z","dependencies_parsed_at":"2024-01-16T09:03:08.459Z","dependency_job_id":"92115b28-0958-4bb2-ab41-c5847b35afa8","html_url":"https://github.com/xiaohei-info/Spark_DB_Connector","commit_stats":null,"previous_names":["chubbyjiang/spark_db_connector"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaohei-info%2FSpark_DB_Connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaohei-info%2FSpark_DB_Connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaohei-info%2FSpark_DB_Connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xiaohei-info%2FSpark_DB_Connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xiaohei-info","download_url":"https://codeload.github.com/xiaohei-info/Spark_DB_Connector/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222157855,"owners_count":16940457,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["connector","database","hbase","mysql","scala","spark-database"],"created_at":"2024-07-31T18:01:21.811Z","updated_at":"2025-03-28T01:37:54.895Z","avatar_url":"https://github.com/xiaohei-info.png","language":"Scala","funding_links":[],"categories":["Scala"],"sub_categories":[],"readme":"# Spark Database Connector\n\n## New Feature\n\n- List写入HBase支持Kerberos认证\n- 升级HBase Client API为1.2.0版本\n\n隐藏处理各种数据库的连接细节，使用Scala API在Spark中简易地处理数据库连接的读写操作。\n\n相关测试环境信息:\n\n- Scala 2.11.8/2.10.5\n- Spark 1.6.0\n- HBase 0.98.4\n- Jdbc Driver 5.1.35\n\n目前支持的有:\n\n- HBase\n- MySQL\n\n添加Maven引用:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003einfo.xiaohei.www\u003c/groupId\u003e\n    \u003cartifactId\u003espark-database-connector_2.11\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nScala 2.10版本使用:\n\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003einfo.xiaohei.www\u003c/groupId\u003e\n    \u003cartifactId\u003espark-database-connector_2.10\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## HBase\n\n### 设置HBase host\n\n通过以下三种任意方式设置HBase host地址\n\n**1、在spark-submit中设置命令：**\n\n```shell\nspark-submit --conf spark.hbase.host=your-hbase-host\n```\n\n**2、在Scala代码中设置：**\n\n```scala\nval sparkConf = new SparkConf()\nsparkConf.set(\"spark.hbase.host\", \"your-hbase-host\")\nval sc = new SparkContext(sparkConf)\n```\n\n**3、在JVM参数中设置：**\n\n```shell\njava -Dspark.hbase.host=your-hbase-host -jar ....\n```\n\n**设置hbase-site.xml文件读取路径(可选)**\n\n如果有读取hbase-site.xml文件的需求时,可以通过设置下面的选项进行指定:\n\n```shell\nspark.hbase.config=your-hbase-config-path\n```\n\n设置该选项的方式同上\n注意:需要将hbase-site.xml文件添加到当前项目可识别的resource路径中,否则将无法读取,使用默认配置\n\n### 向HBase写入数据\n\n**导入隐式转换：**\n\n```scala\nimport info.xiaohei.spark.connector.hbase._\n```\n\n#### Spark RDD写入HBase\n\n任何Spark RDD对象都能直接操作写入HBase，例如：\n\n```scala\nval rdd = sc.parallelize(1 to 100)\n            .map(i =\u003e (s\"rowkey-${i.toString}\", s\"column1-${i.toString}\", \"column2\"))\n```\n\n这个RDD包含了100个三元组类型的数据，写入HBase时，第一个元素为rowkey，剩下的元素依次为各个列的值：\n\n```scala\nrdd.toHBase(\"mytable\")\n      .insert(\"col1\", \"col2\")\n      .inColumnFamily(\"columnFamily\")\n      .save()\n```\n\n(1)使用RDD的toHBase函数传入要写入的表名   \n(2)insert函数传入要插入的各个列名   \n(3)inColumnFamily函数传入这些列所在的列族名   \n(4)最后save函数将该RDD保存在HBase中\n\n如果col2和col1的列族不一样，可以在insert传入列名时单独指定：\n\n```scala\nrdd.toHBase(\"mytable\")\n      .insert(\"col1\", \"otherColumnFamily:col2\")\n      .inColumnFamily(\"defaultColumnFamily\")\n      .save()\n```\n\n列族名和列名之间要用冒号(:)隔开，其他列需要指定列名时使用的方式一致\n\n#### Scala集合/序列写入HBase\n\n```scala\nval dataList  = Seq[(String, String)](\n      (\"00001475304346643896037\", \"kgJkm0euSbe\"),\n      (\"00001475376619355219953\", \"kiaR40qzI8o\"),\n      (\"00001475458728618943637\", \"kgCoW0hgzXO\"),\n      (\"00001475838363931738019\", \"kqiHu0WNJC0\")\n\n    )\n\n//创建隐式变量\nimplicit val hbaseConf = HBaseConf.createConf(\"hbase-host\")\n//如果实在spark程序操作可以通过以下的方式\nimplicit val hbaseConf = HBaseConf.createFromSpark(sc)\n\ndataList.toHBase(\"mytable\")\n\t.insert(\"col1\", \"col2\")\n\t.inColumnFamily(\"columnFamily\")\n\t.save()\n```\n\n使用方式和RDD写入HBase的操作类似，**注意,隐式变量不能在spark的foreachPartition等算子中定义**\n\n以上的方式将使用HTable的put list批量将集合中的数据一次全部put到HBase中，如果写入HBase时想使用缓存区的方式，需要另外添加几个参数：\n\n```scala\ndataList.toHBase(\"mytable\"\n      //该参数指定写入时的autoFlush为false\n      , Some(false, false)\n      //该参数指定写入缓冲区的大小\n      , Some(5 * 1024 * 1024))\n      .insert(\"col1\", \"col2\")\n      .inColumnFamily(\"columnFamily\")\n      .save()\n```\n\n使用该方式时，集合中的每个数据都会被put一次，但是关闭了自动刷写，所以只有当缓冲区满了之后才会批量向HBase写入\n\n#### 写入时为Rowkey添加salt前缀\n\n```scala\nrdd.toHBase(\"mytable\")\n      .insert(\"col1\", \"otherColumnFamily:col2\")\n      .inColumnFamily(\"defaultColumnFamily\")\n      //添加salt\n      .withSalt(saltArray)\n      .save()\n```\n\nsaltArray是一个字符串数组,简单的例如0-9的字符串表示,由使用者自己定义\n\n使用withSalt函数之后,在写入HBase时会为rowkey添加一个saltArray中的随机串,**注意:为了更好的支持HBase部分键扫描(rowkey左对齐),数组中的所有元素长度都应该相等**\n\n取随机串的方式有两种:\n* 1.计算当前的rowkey的hashCode的16进制表示并对saltArray的长度取余数,得到saltArray中的一个随机串作为salt前缀添加到rowkey\n* 2.使用随机数生成器获得不超过saltArray长度的数字作为下标取数组中的值\n\n当前使用的是第一种方式\n\n### 读取HBase数据\n\n**导入隐式转换：**\n\n```scala\nimport info.xiaohei.spark.connector.hbase._\n```\n\n读取HBase的数据操作需要通过sc来进行：\n\n```scala\nval hbaseRdd = sc.fromHBase[(String, String, String)](\"mytable\")\n      .select(\"col1\", \"col2\")\n      .inColumnFamily(\"columnFamily\")\n      .withStartRow(\"startRow\")\n      .withEndRow(\"endRow\")\n      //当rowkey中有随机的salt前缀时,将salt数组传入即可自动解析\n      //得到的rowkey将会是原始的,不带salt前缀的\n      .withSalt(saltArray)\n```\n\n(1)使用sc的fromHBase函数传入要读取数据的表名，该函数需要指定读取数据的类型信息   \n(2)select函数传入要读取的各个列名   \n(3)inColumnFamily函数传入这些列所在的列族名   \n(4)withStartRow和withEndRow将设置rowkey的扫描范围，可选操作\n(5)之后就可以在hbaseRdd上执行Spark RDD的各种算子操作\n\n上面的例子中，fromHBase的泛型类型为三元组，但是select中只读取了两列值，因此，该三元组中第一个元素将是rowkey的值，其他元素按照列的顺序依次类推   \n\n当你不需要读取rowkey的值时，只需要将fromHBase的泛型类型改为二元组\n\n即读取的列数为n，泛型类型为n元组时，列名和元组中的各个元素相对应\n读取的列数为n，泛型类型为n+1元组时，元组的第一个元素为rowkey\n\n当各个列位于不同列族时，设置列族的方式同写入HBase一致\n\n### SQL On HBase\n\n借助SQLContext的DataFrame接口，在组件中可以轻易实现SQL On HBase的功能。\n\n上例中的hbaseRdd是从HBase中读取出来的数据，在此RDD的基础上进行转换操作：\n\n```scala\n//创建org.apache.spark.sql.Row类型的RDD\nval rowRdd = hbaseRdd.map(r =\u003e Row(r._1, r._2, r._3))\nval sqlContext = new SQLContext(sc)\nval df = sqlContext.createDataFrame(\n      rowRdd,\n      StructType(Array(StructField(\"col1\", StringType), StructField(\"col2\", StringType), StructField(\"col3\", StringType)))\n    )\ndf.show()\n\ndf.registerTempTable(\"mytable\")\nsqlContext.sql(\"select col1 from mytable\").show()\n```\n\n### 使用case class查询/读取HBase的数据\n\n使用内置的隐式转换可以处理基本数据类型和元组数据,当有使用case class的需求时,需要额外做一些准备工作\n\n定义如下的case class:\n\n```scala\ncase class MyClass(name: String, age: Int)\n```\n\n如果想达到以下的效果:\n\n```scala\nval classRdd = sc.fromHBase[MyClass](\"tableName\")\n    .select(\"name\",\"age\")\n    .inColumnFamily(\"info\")\n\nclassRdd.map{\n    c =\u003e\n        (c.name,c.age)\n}\n```\n\n或者以下的效果:\n\n```scala\n//classRdd的类型为RDD[MyClass]\nclassRdd.toHBase(\"tableName\")\n    .insert(\"name\",\"age\")\n    .inColumnFamily(\"info\")\n    .save()\n```\n\n需要另外实现能够解析自定义case class的隐式方法:\n\n```scala\nimplicit def myReaderConversion: DataReader[MyClass] = new CustomDataReader[(String, Int), MyClass] {\n    override def convert(data: (String, Int)): MyClass = MyClass(data._1, data._2)\n  }\n\nimplicit def myWriterConversion: DataWriter[MyClass] = new CustomDataWriter[MyClass, (String, Int)] {\n    override def convert(data: MyClass): (String, Int) = (data.name, data.age)\n  }\n```\n\n该隐式方法返回一个DataReader/DataWriter 重写CustomDataReader/CustomDataWriter中的convert方法\n将case class转换为一个元组或者将元组转化为case class即可\n\n## 带有Kerberos认证的HBase\n\n除了上述过程中写HBase需要的配置外,还需要指定以下三个配置:\n\n- spark.hbase.krb.principal:认证的principal用户名\n- spark.hbase.krb.keytab:keytab文件路径(各个节点都存在且路径保持一致)\n- spark.hbase.config:hbase-site.xml文件路径\n\n写入HBase时将会使用提供给的krb信息进行认证\n\n当前仅支持无缝读取启用了Kerberos认证的HBase\n写入时有一定限制，如要使用RDD的foreachPartition入库:\n\n```scala\nrdd.foreachPartition{\n    data =\u003e\n        data.toList.toHBase(\"table\").insert(\"columns\")//...\n}\n```\n\n**注意,foreachPartition中的toList操作将会把分区中的所有数据加载到内存中，如果数据量过大可能会造成OOM，增加Executor的内存即可**\n\nTODO:RDD的读写接口目前还未实现Kerberos认证\n\n## MySQL\n\n除了可以将RDD/集合写入HBase之外，还可以在普通的程序中进行MySQL的相关操作\n\n### 在conf中设置相关信息\n\n**1、Spark程序中操作**\n\n在SparkConf中设置以下的信息：\n\n```scala\nsparkConf\n  .set(\"spark.mysql.host\", \"your-host\")\n  .set(\"spark.mysql.username\", \"your-username\")\n  .set(\"spark.mysql.password\", \"your-passwd\")\n  .set(\"spark.mysql.port\", \"db-port\")\n  .set(\"spark.mysql.db\", \"database-name\")\n\n//创建MySqlConf的隐式变量\nimplicit val mysqlConf = MysqlConf.createFromSpark(sc)\n```\n\n关于这个隐式变量的说明：在RDD的foreachPartition或者mapPartitions等操作时，因为涉及到序列化的问题，默认的对MySqlConf的隐式转化操作会出现异常问题，所以需要显示的声明一下这个变量，其他不涉及网络序列化传输的操作可以省略这步\n\nHBase小节中的设置属性的方法在这里也适用\n\n**2、普通程序中操作**\n\n创建MysqlConf，并设置相关属性：\n\n```scala\n//创建MySqlConf的隐式变量\nimplicit val mysqlConf = MysqlConf.createConf(\n      \"your-host\",\n      \"username\",\n      \"password\",\n      \"port\",\n      \"db-name\"\n    )\n\n```\n\n在普通程序中操作时一定要显示声明MysqlConf这个隐式变量\n\n### 写入MySQL\n\n导入隐式转换：\n\n```scala\nimport info.xiaohei.spark.connector.mysql._\n```\n\n之后任何Iterable类型的数据都可以直接写入MySQL中：\n\n```scala\nlist.toMysql(\"table-name\")\n  //插入的列名\n  .insert(\"columns\")\n  //where条件，如age=1\n  .where(\"where-conditions\")\n  .save()\n```\n\n\n### 在Spark程序中从MySQL读取数据\n\n```scala\nval res = sc.fromMysql[(Int,String,Int)](\"table-name\")\n  .select(\"id\",\"name\",\"age\")\n  .where(\"where-conditions\")\n  .get\n```\n\n### 在普通程序中从MySQL读取数据\n\n```scala\n//普通程序读取关系型数据库入口\nval dbEntry = new RelationalDbEntry\n\nval res = dbEntry.fromMysql[(Int,String,Int)](\"table-name\")\n  .select(\"id\",\"name\",\"age\")\n  .where(\"where-conditions\")\n  .get\n```\n\n创建数据库入口之后的操作和spark中的流程一致\n\n### case class解析\n\n如果需要使用自定义的case class解析/写入MySQL,例如:\n\n```scala\ncase class Model(id: Int, name: String, age: Int)\n```\n\n基本流程和hbase小节中差不多,定义隐式转换:\n\n```scala\nimplicit def myExecutorConversion: DataExecutor[Model] = new CustomDataExecutor[Model, (Int, String, Int)]() {\n    override def convert(data: Model): (Int, String, Int) = (data.id, data.name, data.age)\n}\n\nimplicit def myMapperConversion: DataMapper[Model] = new CustomDataMapper[(Int, String, Int), Model]() {\n    override def convert(data: (Int, String, Int)): Model = Model(data._1, data._2, data._3)\n }\n```\n\n之后可以直接使用:\n\n```scala\nval entry = new RelationalDbEntry\nval res = entry.fromMysql[Model](\"test\")\n  .select(\"id\", \"name\", \"age\")\n  .get\nres.foreach(x =\u003e println(s\"id:${x.id},name:${x.name},age:${x.age}\"))\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaohei-info%2FSpark_DB_Connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxiaohei-info%2FSpark_DB_Connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxiaohei-info%2FSpark_DB_Connector/lists"}