{"id":18080600,"url":"https://github.com/anskarl/parsimonious","last_synced_at":"2025-04-12T14:22:03.786Z","repository":{"id":53836553,"uuid":"352927223","full_name":"anskarl/parsimonious","owner":"anskarl","description":"Parsimonious is a helper library for encoding/decoding Apache Thrift and Twitter Scrooge classes to Spark Dataframes and Jackson JSON.","archived":false,"fork":false,"pushed_at":"2024-05-02T15:54:47.000Z","size":169,"stargazers_count":11,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-26T08:51:39.311Z","etag":null,"topics":["deserialization","jackson","json","serialization","spark","thrift"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anskarl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.TXT","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-30T08:30:55.000Z","updated_at":"2024-05-02T15:54:52.000Z","dependencies_parsed_at":"2024-05-02T16:55:11.956Z","dependency_job_id":"417f4f04-3cbd-4d44-b8ba-0b51c0da5337","html_url":"https://github.com/anskarl/parsimonious","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anskarl%2Fparsimonious","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anskarl%2Fparsimonious/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anskarl%2Fparsimonious/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anskarl%2Fparsimonious/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anskarl","download_url":"https://codeload.github.com/anskarl/parsimonious/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248578883,"owners_count":21127720,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deserialization","jackson","json","serialization","spark","thrift"],"created_at":"2024-10-31T13:08:59.837Z","updated_at":"2025-04-12T14:22:03.767Z","avatar_url":"https://github.com/anskarl.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"```\n┌─┐┌─┐┬─┐┌─┐┬┌┬┐┌─┐┌┐┌┬┌─┐┬ ┬┌─┐\n├─┘├─┤├┬┘└─┐│││││ ││││││ ││ │└─┐\n┴  ┴ ┴┴└─└─┘┴┴ ┴└─┘┘└┘┴└─┘└─┘└─┘\n```\n_Parsimonious_ is a helper library for encoding/decoding Apache Thrift and Twitter Scrooge (experimental) classes to Spark Dataframes, \nJackson JSON and Apache Flink state serializers. \n\n  - The implementation for Spark is based on [airbnb-spark-thrift](https://github.com/airbnb/airbnb-spark-thrift/tree/nwparker/convV2).\n  - The implementation for Flink state serializers is inspired by [findify flink-protobuf](https://github.com/findify/flink-protobuf).\n\nImportant features:\n\n  - Supports all Thrift types, including unions and binary types. \n  - Supports both Apache Thrift and Twitter Scrooge (experimental) generated classes.\n  - Supports nested and recursive structures. Please note, for Spark, recursive structures are being serialized to bytes.\n  - For JSON, when a Thrift map does not have string type as a key (e.g., a struct) then _Parsimonious_ will convert it to a sequence of key, value tuples (the opposite during decoding to Thrift is also supported).\n  \n## Example usage\n\nMore detailed examples can be found in unit tests (including nested and recursive structures).\n\nAssume that we have an Apache Thrift struct named `BasicDummy`\n\n```thrift\nnamespace java com.github.anskarl.parsimonious\n\nstruct BasicDummy {\n    1: required string reqStr\n    2: optional string str\n    3: optional i16 int16\n    4: optional i32 int32\n    5: optional i64 int64\n    6: optional double dbl\n    7: optional byte byt\n    8: optional bool bl\n    9: optional binary bin\n}\n```\n\n### Apache Thrift with Apache Spark\n\nCreate a Spark Dataframe:\n\n```scala\nimport org.apache.spark.sql.types.StructType\nimport org.apache.spark.rdd.RDD\nimport org.apache.spark.sql.{DataFrame, Row}\nimport com.github.anskarl.parsimonious._\nimport com.github.anskarl.parsimonious.spark.ThriftRowConverter\n\n// Assume that we have SparkSession initialized as `spark`\n// To extract the schema (i.e., Apache Spark org.apache.spark.sql.types.StructType)\nval sparkSchema: StructType = ThriftRowConverter.extractSchema(classOf[BasicDummy])\n\n// Example collection of BasicDummy instances\nval exampleData = \n    for (index \u003c- 1 to 100) \n        yield new BasicDummy().setReqStr(s\"index: ${index}\").setInt32(index).setBl(index % 10 == 0)\n\n// Convert BasicDummy to org.apache.spark.sql.Row\nval rowSeq: Seq[Row] = exampleData.map(_.toRow)\n\n// Create RDD[Row]\nval rowRDD: RDD[Row] = spark.sparkContext.parallelize(rowSeq)\n\n// Create the corresponding dataframe\nval df: DataFrame = spark.createDataFrame(rowRDD, sparkSchema)\n```\n\nSee the schema and preview the first 5 rows:\n\n```scala\n// Print dataframe schema\ndf.printSchema()\n\n// Will print:\n// root\n// |-- reqStr: string (nullable = false)\n// |-- str: string (nullable = true)\n// |-- int16: short (nullable = true)\n// |-- int32: integer (nullable = true)\n// |-- int64: long (nullable = true)\n// |-- dbl: double (nullable = true)\n// |-- byt: byte (nullable = true)\n// |-- bl: boolean (nullable = true)\n// |-- bin: binary (nullable = true)\n\ndf.show(numRows = 5, truncate = false)\n// +---------+----+-----+-----+-----+----+----+-----+----+\n// |reqStr   |str |int16|int32|int64|dbl |byt |bl   |bin |\n// +---------+----+-----+-----+-----+----+----+-----+----+\n// |index: 1 |null|null |1    |null |null|null|false|null|\n// |index: 2 |null|null |2    |null |null|null|false|null|\n// |index: 3 |null|null |3    |null |null|null|false|null|\n// |index: 4 |null|null |4    |null |null|null|false|null|\n// |index: 5 |null|null |5    |null |null|null|false|null|\n// +---------+----+-----+-----+-----+----+----+-----+----+\n```\n\nCollect the rows and convert back to `BasicDummy`:\n\n```scala\n// Collect rows\nval dfRows: Array[Row] = df.collect()\n\n// Convert the collected rows back to BasicDummy instances\nval decodedInputSeq: Seq[BasicDummy] = dfRows\n    .map(row =\u003e row.as(classOf[BasicDummy])\n    .toSeq\n\n// Print the first 5 `BasicDummy`\ndecodedInputSeq.take(5).foreach(println)\n\n// BasicDummy(reqStr:index: 1, int32:1, bl:false)\n// BasicDummy(reqStr:index: 2, int32:2, bl:false)\n// BasicDummy(reqStr:index: 3, int32:3, bl:false)\n// BasicDummy(reqStr:index: 4, int32:4, bl:false)\n// BasicDummy(reqStr:index: 5, int32:5, bl:false)\n```\n\n### Apache Thrift with Jackson for JSON support\n\nEncode/Decode Apache Thrift POJO class to/from Jackson node:\n\n```scala\nimport com.github.anskarl.parsimonious._\nimport com.github.anskarl.parsimonious.json._\nimport scala.collection.JavaConverters._\n\n// create POJO\nval basicDummy = new BasicDummy()\n  basicDummy.setReqStr(\"required 101\")\n  basicDummy.setStr(\"optional 101\")\n  basicDummy.setInt16(101.toShort)\n  basicDummy.setInt32(101)\n  basicDummy.setInt64(101L)\n  basicDummy.setDbl(101.101)\n  basicDummy.setByt(8.toByte)\n  basicDummy.setBl(false)\n  basicDummy.setBin(\"101\".getBytes(\"UTF-8\"))\n  basicDummy.setListNumbersI32(List(1,2,3).map(java.lang.Integer.valueOf).asJava)\n  basicDummy.setListNumbersDouble(List(1.1,2.2,3.3).map(java.lang.Double.valueOf).asJava)\n  basicDummy.setSetNumbersI32(Set(1,2,3).map(java.lang.Integer.valueOf).asJava)\n  basicDummy.setSetNumbersDouble(Set(1.1,2.2,3.3).map(java.lang.Double.valueOf).asJava)\n  basicDummy.setEnm(EnumDummy.MAYBE)\n  basicDummy.setListStruct(List(new PropertyValue(\"prop1\", \"val1\"), new PropertyValue(\"prop2\", \"val2\")).asJava)\n  basicDummy.setMapPrimitives(\n    Map(\n      java.lang.Integer.valueOf(1) -\u003e java.lang.Double.valueOf(1.1),\n      java.lang.Integer.valueOf(2) -\u003e java.lang.Double.valueOf(2.2)\n    ).asJava\n  )\nbasicDummy.setMapStructKey(Map(\n    new PropertyValue(\"prop1\", \"val1\") -\u003e java.lang.Double.valueOf(1.1),\n    new PropertyValue(\"prop2\", \"val2\") -\u003e java.lang.Double.valueOf(2.2)\n  ).asJava)\nbasicDummy.setMapPrimitivesStr(Map(\"one\" -\u003e java.lang.Double.valueOf(1.0), \"two\" -\u003e java.lang.Double.valueOf(2.0)).asJava)\n\n\n// .. etc\n\n// Encode to Json\nval encoded: ObjectNode = ThriftJsonConverter.convert(basicDummy)\nprintln(encoded.toPrettyString)\n```\nWill print the following:\n```json\n{\n    \"reqStr\" : \"required 101\",\n    \"str\" : \"optional 101\",\n    \"int16\" : 101,\n    \"int32\" : 101,\n    \"int64\" : 101,\n    \"dbl\" : 101.101,\n    \"byt\" : 8,\n    \"bl\" : false,\n    \"bin\" : \"MTAx\",\n    \"listNumbersI32\" : [ 1, 2, 3 ],\n    \"listNumbersDouble\" : [ 1.1, 2.2, 3.3 ],\n    \"setNumbersI32\" : [ 1, 2, 3 ],\n    \"setNumbersDouble\" : [ 1.1, 2.2, 3.3 ],\n    \"enm\" : \"MAYBE\",\n    \"listStruct\" : [ {\n      \"property\" : \"prop1\",\n      \"value\" : \"val1\"\n    }, {\n      \"property\" : \"prop2\",\n      \"value\" : \"val2\"\n    } ],\n    \"mapPrimitives\":[{\"key\":1,\"value\":1.1},{\"key\":2,\"value\":2.2}],\n    \"mapStructKey\" : [ {\n      \"key\" : {\n        \"property\" : \"prop1\",\n        \"value\" : \"val1\"\n      },\n      \"value\" : 1.1\n    }, {\n      \"key\" : {\n        \"property\" : \"prop2\",\n        \"value\" : \"val2\"\n      },\n      \"value\" : 2.2\n    } ],\n    \"mapPrimitivesStr\": {\"one\": 1.0, \"two\": 2.0}\n}\n```\nPlease note that the type of key in both `mapPrimitives` and `mapStructKey` is not string. \nIn such cases _Parsimonious_ converts them to lists of structs with the pair of fields \n`\"key\"` and `\"value\"`. That convention, is being performed since JSON does not support \nmaps having keys with type different of string.  \n\nTo decode from JSON back to Thrift POJO:\n\n```scala\n// Decode from Json\nval decoded: BasicDummy = JsonThriftConverter.convert(classOf[BasicDummy], encoded)\n```\n\n### Scrooge Thrift with Apache Spark (experimental)\n\nCreate a Spark Dataframe:\n\n```scala\nimport org.apache.spark.sql.types.StructType\nimport org.apache.spark.rdd.RDD\nimport org.apache.spark.sql.{DataFrame, Row}\nimport com.github.anskarl.parsimonious.scrooge._\nimport com.github.anskarl.parsimonious.scrooge.spark._\n\n// Assume that we have SparkSession initialized as `spark`\n// To extract the schema (i.e., Apache Spark org.apache.spark.sql.types.StructType)\nval sparkSchema: StructType = ScroogeRowConverter.extractSchema(classOf[BasicDummy])\n\n// Need to create once union builders, helper class to extract the schema and \n// create builder for Scrooge Union (com.twitter.scrooge.ThriftUnion)\nimplicit val unionBuilders = UnionBuilders.create(classOf[BasicDummy])\n\n\n// Example collection of BasicDummy instances\nval exampleData = for (index \u003c- 1 to 100) \n  yield new BasicDummy(reqStr = s\"index: ${index}\", int32 = index, bl = index % 10 == 0)\n\n// Convert BasicDummy to org.apache.spark.sql.Row\nval rowSeq: Seq[Row] = exampleData.map(_.toRow)\n\n// Create RDD[Row]\nval rowRDD: RDD[Row] = spark.sparkContext.parallelize(rowSeq)\n\n// Create the corresponding dataframe\nval df: DataFrame = spark.createDataFrame(rowRDD, sparkSchema)\n```\n\nSee the schema and preview the first 5 rows:\n```scala\n// Print dataframe schema\ndf.printSchema()\n\n// Will print:\n// root\n// |-- reqStr: string (nullable = false)\n// |-- str: string (nullable = true)\n// |-- int16: short (nullable = true)\n// |-- int32: integer (nullable = true)\n// |-- int64: long (nullable = true)\n// |-- dbl: double (nullable = true)\n// |-- byt: byte (nullable = true)\n// |-- bl: boolean (nullable = true)\n// |-- bin: binary (nullable = true)\n\ndf.show(numRows = 5, truncate = false)\n// +---------+----+-----+-----+-----+----+----+-----+----+\n// |reqStr   |str |int16|int32|int64|dbl |byt |bl   |bin |\n// +---------+----+-----+-----+-----+----+----+-----+----+\n// |index: 1 |null|null |1    |null |null|null|false|null|\n// |index: 2 |null|null |2    |null |null|null|false|null|\n// |index: 3 |null|null |3    |null |null|null|false|null|\n// |index: 4 |null|null |4    |null |null|null|false|null|\n// |index: 5 |null|null |5    |null |null|null|false|null|\n// +---------+----+-----+-----+-----+----+----+-----+----+\n```\n\nCollect the rows and convert back to `BasicDummy`:\n\n\n```scala\n// Collect rows\nval dfRows: Array[Row] = df.collect()\n\n// Convert the collected rows back to BasicDummy instances\nval decodedInputSeq: Seq[BasicDummy] = dfRows\n    .map(row =\u003e row.as(classOf[BasicDummy])\n    .toSeq\n```\n\n### Scrooge Thrift with Jackson for JSON support (experimental)\n\nEncode/Decode Scrooge generated classes to/from Jackson node:\n\n```scala\nimport com.fasterxml.jackson.databind.ObjectMapper\nimport com.fasterxml.jackson.databind.node.ObjectNode\nimport com.fasterxml.jackson.module.scala.DefaultScalaModule\nimport com.github.anskarl.parsimonious.{BasicDummy, EnumDummy, NestedDummy, PropertyValue}\nimport com.github.anskarl.parsimonious.scrooge._\nimport java.nio.ByteBuffer\n\nval mapper = new ObjectMapper().registerModule(DefaultScalaModule)\n\nval sampleBasicDummy = BasicDummy(\n    reqStr = \"required 101\",\n    str = Option(\"optional 101\"),\n    int16 = Option(16.toShort),\n    int32 = Option(32),\n    int64 = Option(64L),\n    dbl = Option(101.101),\n    byt = Option(8.toByte),\n    bl = Option(false),\n    bin = Option(ByteBuffer.wrap(\"101\".getBytes(\"UTF-8\"))),\n    listNumbersI32 = Option(List(1,2,3)),\n    listNumbersDouble = Option(List(1.1,2.2,3.3)),\n    setNumbersI32 = Option(Set(1,2,3)),\n    setNumbersDouble = Option(Set(1.1,2.2,3.3)),\n    enm = Option(EnumDummy.Maybe),\n    listStruct = Option(List(PropertyValue(\"prop1\", \"val1\"),PropertyValue(\"prop2\", \"val2\"))),\n    mapPrimitives = Option(Map(1 -\u003e 1.1, 2 -\u003e 2.2)),\n    mapStructKey = Option(Map(PropertyValue(\"prop1\", \"val1\") -\u003e 1.1, PropertyValue(\"prop2\", \"val2\") -\u003e 2.2)),\n    mapPrimitivesStr = Option(Map(\"one\" -\u003e 1.0, \"two\" -\u003e 2.0))\n  )\n\n// Encode to JSON\nval encoded = ScroogeJsonConverter.convert(sampleBasicDummy)\nprintln(encoded.toPrettyString)\n```\n\nWill print the following:\n```json\n{\n  \"reqStr\" : \"required 101\",\n  \"str\" : \"optional 101\",\n  \"int16\" : 16,\n  \"int32\" : 32,\n  \"int64\" : 64,\n  \"dbl\" : 101.101,\n  \"byt\" : 8,\n  \"bl\" : false,\n  \"bin\" : \"MTAx\",\n  \"listNumbersI32\" : [ 1, 2, 3 ],\n  \"listNumbersDouble\" : [ 1.1, 2.2, 3.3 ],\n  \"setNumbersI32\" : [ 1, 2, 3 ],\n  \"setNumbersDouble\" : [ 1.1, 2.2, 3.3 ],\n  \"enm\" : \"Maybe\",\n  \"listStruct\" : [ {\n    \"property\" : \"prop1\",\n    \"value\" : \"val1\"\n  }, {\n    \"property\" : \"prop2\",\n    \"value\" : \"val2\"\n  } ],\n  \"mapPrimitives\" : [ {\n    \"key\" : 1,\n    \"value\" : 1.1\n  }, {\n    \"key\" : 2,\n    \"value\" : 2.2\n  } ],\n  \"mapStructKey\" : [ {\n    \"key\" : {\n      \"property\" : \"prop1\",\n      \"value\" : \"val1\"\n    },\n    \"value\" : 1.1\n  }, {\n    \"key\" : {\n      \"property\" : \"prop2\",\n      \"value\" : \"val2\"\n    },\n    \"value\" : 2.2\n  } ],\n  \"mapPrimitivesStr\" : {\n    \"one\" : 1.0,\n    \"two\" : 2.0\n  }\n}\n```\nRecall to Apache Thrift to JSON conversion, the type of key in both `mapPrimitives` and `mapStructKey` is not string.\nIn such cases _Parsimonious_ converts them to lists of structs with the pair of fields\n`\"key\"` and `\"value\"`. That convention, is being performed since JSON does not support\nmaps having keys with type different of string.\n\nTo decode from JSON back to Scrooge class:\n```scala\n\n// Need to create once union builders, helper class to extract the schema and \n// create builder for Scrooge Union (com.twitter.scrooge.ThriftUnion)\nimplicit val unionBuilders = UnionBuilders.create(classOf[BasicDummy])\n\n// Decode to Scrooge class\nval decoded: BasicDummy = JsonScroogeConverter.convert(classOf[BasicDummy], encodedJson)\n```\n\n### Flink state serializers for Apache Thrift\n\nParsimonious provides [state serializer](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/) for Apache Thrift generated classes. To enable Thrift compatible \nstate serializer you need to create [TypeInformation](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#flinks-typeinformation-class) for Thrift generated classes.\n\nFor example, to specify type information for class `BasicDummy`:\n\n```scala\nimport com.github.anskarl.parsimonious.flink._\n\nval typeInformation = ThriftTypeInformation(classOf[BasicDummy])\n```\n\n### Flink state serializers for Twitter Scrooge\n\nSimilarly, parsimonious provides [state serializer](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/serialization/custom_serialization/) for Twitter Scrooge generated classes. To enable Scrooge compatible\nstate serializer you need to create [TypeInformation](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#flinks-typeinformation-class) for Scrooge generated classes. \n\nFor example, to specify type information for class `BasicDummy`:\n\n```scala\nimport com.github.anskarl.parsimonious.scrooge.flink._\n\nval typeInformation = ScroogeTypeInformation(classOf[BasicDummy])\n```\n\n## Configure Parsimonious\nPlease note that the type of key in both `mapPrimitives` and `mapStructKey` is not string.\nIn such cases _Parsimonious_ converts them to lists of structs with the pair of fields\n`\"key\"` and `\"value\"`. That convention, is being performed since JSON does not support\nmaps having keys with type different of string.\n\n\nYou can configure the parameters of Parsimonious modules by specifying an implicit value of `ParsimoniousConfig` class. \nThe parameters are the following:\n\n  - In JSON, when the type of key in some Thrift `map` collection is not a string, _Parsimonious_ converts them to \n  lists of structs with the pair of fields. Each struct is composed of a pair fields (key and value). The default field \n  names are `\"key\"` and `\"value\"`, but they can change by specifying `keyName` and `valueName` respectively in\n  `ParsimoniousConfig` class.\n  - You can also change the serialization protocol by specifying the `TProtocolFactoryType` in field \n  `protocolFactoryType`.  _Parsimonious_ supports the following protocols:\n    1. `TCompactProtocolFactoryType` (default), that creates TProtocolFactory that produces TCompactProtocols.\n    2. `TBinaryProtocolFactoryType`, that creates TProtocolFactory that produces TBinaryProtocols.\n    3. `TJSONProtocolFactoryType`, that creates TProtocolFactory that produces TJSONProtocols.\n  - **IMPORTANT NOTE**: For Flink state serializers, Parsimonious ignores the `protocolFactoryType` and uses only `TCompactProtocolFactoryType`.\n\nFor example, the following instance `ParsimoniousConfig`, specifies `k` and `v` for key-value fields when mapping \nThrift map collection in which the key is not a type of string. Also, thrift protocol will be the type of \n`TBinaryProtocol`: \n\n```scala\nimport com.github.anskarl.parsimonious.common._\n\nimplicit val config = ParsimoniousConfig(\n  keyName = \"k\",\n  valueName = \"v\",\n  protocolFactoryType = TBinaryProtocolFactoryType\n)\n```\n\n## Dependencies\n\nVersion variants published in `oss.sonatype.org`\n\n  - scala_version: `2.12`, `2.13`\n  - thrift_version: `thrift_0.10`, `thrift_0.13`\n  - spark_profile: `spark2` (i.e., 2.4.x), `spark3` (i.e., 3.1.x)\n  - flink_profile: `flink1_13`, `flink1_14` and `flink1_15`\n  - parsimonious_version: e.g., `0.4.0`\n\n#### Maven\n\n***thrift-jackson***:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.anskarl\u003c/groupId\u003e\n  \u003cartifactId\u003eparsimonious-thrift-jackson_[scala_version]\u003c/artifactId\u003e\n  \u003cversion\u003e[thrift_version]-[parsimonious_version]\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n***thrift-spark***:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.anskarl\u003c/groupId\u003e\n  \u003cartifactId\u003eparsimonious-thrift-spark_[scala_version]\u003c/artifactId\u003e\n  \u003cversion\u003e[thrift_version]_[spark_profile]-[parsimonious_version]\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n***scrooge-jackson***:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.anskarl\u003c/groupId\u003e\n  \u003cartifactId\u003eparsimonious-scrooge-jackson_[scala_version]\u003c/artifactId\u003e\n  \u003cversion\u003e[thrift_version]-[parsimonious_version]\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n***scrooge-spark***:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.anskarl\u003c/groupId\u003e\n  \u003cartifactId\u003eparsimonious-scrooge-spark_[scala_version]\u003c/artifactId\u003e\n  \u003cversion\u003e[thrift_version]_[spark_profile]-[parsimonious_version]\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n***thrift-flink***:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.anskarl\u003c/groupId\u003e\n  \u003cartifactId\u003eparsimonious-thrift-flink_2.12\u003c/artifactId\u003e\n  \u003cversion\u003e[thrift_version]_[flink_profile]-[parsimonious_version]\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n\n***scrooge-flink***:\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.anskarl\u003c/groupId\u003e\n  \u003cartifactId\u003eparsimonious-scrooge-flink_2.12\u003c/artifactId\u003e\n  \u003cversion\u003e[thrift_version]_[flink_profile]-[parsimonious_version]\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### SBT\n\nSee the [official docs](https://www.scala-sbt.org/1.x/docs/Resolvers.html) to configure Sonatype (snapshots) \nresolver in your SBT project. For example:\n```\nresolvers += Resolver.sonatypeRepo(\"public\") //  (or “snapshots”, “releases”) \n```\n\n***thrift-jackson***:\n```\n\"com.github.anskarl\" %% \"parsimonious-thrift-jackson\" % \"[thrift_version]-[parsimonious_version]\"\n```\n\n***thrift-spark***:\n```\n\"com.github.anskarl\" %% \"parsimonious-thrift-spark\" % \"[thrift_version]_[spark_profile]-[parsimonious_version]\"\n```\n\n***scrooge-jackson***:\n```\n\"com.github.anskarl\" %% \"parsimonious-scrooge-jackson\" % \"[thrift_version]-[parsimonious_version]\"\n```\n\n***scrooge-spark***:\n```\n\"com.github.anskarl\" %% \"parsimonious-scrooge-spark\" % \"[thrift_version]_[spark_profile]-[parsimonious_version]\"\n```\n\n***thrift-flink***:\n```\n\"com.github.anskarl\" % \"parsimonious-thrift-flink_2.12\" % \"[thrift_version]_[flink_profile]-[parsimonious_version]\"\n```\n\n***scrooge-flink***:\n```\n\"com.github.anskarl\" % \"parsimonious-scrooge-flink_2.12\" % \"[thrift_version]_[flink_profile]-[parsimonious_version]\"\n```\n\n\n## Build from sources\n\nTo build parsimonious from sources you will need an SBT version 1.6+. The build can be parameterized for the following environment variables:\n\n  - `THRIFT_VERSION`: e.g., 0.13.0. Default is `0.10.0`. Please note that for Scrooge modules is always version `0.10.0`.\n  - `SPARK_PROFILE`: can be either `spark2` or `spark3` (default is `spark3`).\n    * In `spark2` parsimonious is build for Spark v2.4.6, Hadoop v2.10.0 and Parquet v1.10.1.\n    * In `spark3` parsimonious is build for Spark v3.1.2, Hadoop v3.3.1 and Parquet v1.12.2.\n  - `FLINK_PROFILE`: can be one of `flink1_13`,`flink1_14` and `flink1_15`\n\nFor all variants of `THRIFT_VERSION` and `SPARK_PROFILE`, parsimonious can be cross-build for Scala 2.12 and 2.13.\nFlink modules are only build for Scala 2.12 (2.13 is not currently supported by Flink).\n\nThere are several sbt command aliases to build any of the parsimonious modules separately or all together. \nFor a complete list see the `.sbtrc` file. \n\n##### All modules using default settings\n\nThe following command will build all modules, using default version of Thrift (0.10), Spark (3.2) and Flink (1.13)\n\n```shell\nsbt build-all\n```\n\n##### All modules with custom Thrift, Spark and Flink versions\n\nThe following command will build all modules for Thrift version 0.13.0, Spark 2 (2.4.x) and Flink 1.15\n\n```shell\nTHRIFT_VERSION=0.13.0 SPARK_PROFILE=spark2 FLINK_PROFILE=flink1_15 sbt build-all\n```\n\n##### Specific module using default settings\n\nThe following command will build build-scrooge-jackson module (translates Scrooge classes to/from JSON),\nusing default version of Thrift (0.10) and Spark (3.2).\n\n```shell\nsbt build-scrooge-jackson\n```\n\n##### Specific module custom Thrift and Spark version\n\nThe following command will build build-spark module (translates Apache Thrift classes to/from Spark Dataframe),\nusing Thrift version 0.13 and Spark 2 (2.4.x).\n\n```shell\nTHRIFT_VERSION=0.13.0 SPARK_PROFILE=spark2 sbt build-spark\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanskarl%2Fparsimonious","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanskarl%2Fparsimonious","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanskarl%2Fparsimonious/lists"}