{"id":28532140,"url":"https://github.com/databricks-industry-solutions/smolder","last_synced_at":"2025-07-07T13:31:22.181Z","repository":{"id":40385212,"uuid":"314386512","full_name":"databricks-industry-solutions/smolder","owner":"databricks-industry-solutions","description":"HL7 Apache Spark Datasource","archived":false,"fork":false,"pushed_at":"2024-10-23T22:49:50.000Z","size":76,"stargazers_count":66,"open_issues_count":1,"forks_count":25,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-06-09T15:43:55.506Z","etag":null,"topics":["datasource","hl7","hl7v2","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks-industry-solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE-OF-CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-11-19T22:37:52.000Z","updated_at":"2025-05-28T19:01:54.000Z","dependencies_parsed_at":"2024-10-24T01:55:27.200Z","dependency_job_id":"d39c9338-2e5a-4520-b124-9c7aaaa689af","html_url":"https://github.com/databricks-industry-solutions/smolder","commit_stats":null,"previous_names":["databricks-industry-solutions/smolder"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/databricks-industry-solutions/smolder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fsmolder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fsmolder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fsmolder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fsmolder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks-industry-solutions","download_url":"https://codeload.github.com/databricks-industry-solutions/smolder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks-industry-solutions%2Fsmolder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264085408,"owners_count":23555195,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datasource","hl7","hl7v2","spark"],"created_at":"2025-06-09T15:31:09.378Z","updated_at":"2025-07-07T13:31:22.175Z","avatar_url":"https://github.com/databricks-industry-solutions.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"smolder-light-background.svg\" width=\"300px\"/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  A library for burning through electronic health record data using Apache Spark\u0026trade;\n\u003c/p\u003e\n\nSmolder provides an Apache Spark\u0026trade; SQL data source for loading EHR data from\n[HL7v2](https://www.hl7.org/implement/standards/product_brief.cfm?product_id=244)\nmessage formats. Additionally, Smolder provides helper functions that can be used\non a Spark SQL DataFrame to parse HL7 message text, and to extract segments,\nfields, and subfields, from a message.\n\n\n# Project Support\nPlease note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs).  They are provided AS-IS and we do not make any guarantees of any kind.  Please do not submit a support ticket relating to any issues arising from the use of these projects.\n\nAny issues discovered through the use of this project should be filed as GitHub Issues on the Repo.  They will be reviewed as time permits, but there are no formal SLAs for support.\n\n# Building and Testing\n\nThis project is built using [sbt](https://www.scala-sbt.org/1.0/docs/Setup.html) and Java 8.\n\nStart an sbt shell using the `sbt` command.\n\n\u003e **FYI**: The following SBT projects are built on Spark 3.2.1/Scala 2.12.8 by default. To change the Spark version and\nScala version, set the environment variables `SPARK_VERSION` and `SCALA_VERSION`.\n\nTo compile the main code:\n```\ncompile\n```\n\nTo run all Scala tests:\n```\ntest\n```\n\nTo test a specific suite:\n```\ntestOnly *HL7FileFormatSuite\n```\n\nTo create a JAR that can be run as part of an [Apache Spark job or\nshell](http://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management), run:\n```\npackage\n```\n\nThe JAR can be found under `target/scala-\u003cmajor-version\u003e`.\n\n# Getting Started\n\nTo load HL7 messages into an [Apache Spark SQL\nDataFrame](http://spark.apache.org/docs/latest/sql-programming-guide.html),\nsimply invoke the `hl7` reader:\n\n```\nscala\u003e val df = spark.read.format(\"hl7\").load(\"path/to/hl7/messages\")\ndf: org.apache.spark.sql.DataFrame = [message: string, segments: array\u003cstruct\u003cid:string,fields:array\u003cstring\u003e\u003e\u003e]\n```\n\nThe schema returned contains the message header in the `message` column. The\nmessage segments are nested in the `segments` column, which is an array. This\narray contains two nested fields: the string `id` for the segment (e.g., `PID`\nfor a [patient identification segment](http://www.hl7.eu/refactored/segPID.html)\nand an array of segment `fields`.\n\n## Parsing message text from a DataFrame\n\nSmolder can also be used to parse raw message text. This might happen if you had\nan HL7 message feed land in an intermediate source first (e.g., a Kafka stream).\nTo do this, we can use Smolder's `parse_hl7_message` helper function. First, we\nstart with a DataFrame containing HL7 message text:\n\n```\nscala\u003e val textMessageDf = ...\ntextMessageDf: org.apache.spark.sql.DataFrame = [value: string]\n\nscala\u003e textMessageDf.show()\n+--------------------+                                                          \n|               value|\n+--------------------+\n|MSH|^~\\\u0026|||||2020...|\n+--------------------+\n```\n\nThen, we can import the `parse_hl7_message` message from the\n`com.databricks.labs.smolder.functions` object and apply that to the column we\nwant to parse:\n\n```\nscala\u003e import com.databricks.labs.smolder.functions.parse_hl7_message\nimport com.databricks.labs.smolder.functions.parse_hl7_message\n\nscala\u003e val parsedDf = textMessageDf.select(parse_hl7_message($\"value\").as(\"message\"))\nparsedDf: org.apache.spark.sql.DataFrame = [message: struct\u003cmessage: string, segments: array\u003cstruct\u003cid:string,fields:array\u003cstring\u003e\u003e\u003e\u003e]\n```\n\nThis yields the same schema as our `hl7` data source.\n\n## Extracting fields from an HL7 message segment\n\nWhile Smolder provides an easy-to-use schema for HL7 messages, we also provide\nhelper functions in `com.databricks.labs.smolder.functions` to extract subfields\nof a message segment. For instance, let's say we want to get the patient's name,\nwhich is the 5th field in the patient ID (PID) segment. We can extract this with\nthe `segment_field` function:\n\n```\nscala\u003e import com.databricks.labs.smolder.functions.segment_field\nimport com.databricks.labs.smolder.functions.segment_field\n\nscala\u003e val nameDf = df.select(segment_field(\"PID\", 4).alias(\"name\"))\nnameDf: org.apache.spark.sql.DataFrame = [name: string]\n\nscala\u003e nameDf.show()\n+-------------+\n|         name|\n+-------------+\n|Heller^Keneth|\n+-------------+\n```\n\nIf we then wanted to get the patient's first name, we can use the `subfield`\nfunction:\n\n```\nscala\u003e import com.databricks.labs.smolder.functions.subfield\nimport com.databricks.labs.smolder.functions.subfield\n\nscala\u003e val firstNameDf = nameDf.select(subfield($\"name\", 1).alias(\"firstname\"))\nfirstNameDf: org.apache.spark.sql.DataFrame = [firstname: string]\n\nscala\u003e firstNameDf.show()\n+---------+\n|firstname|\n+---------+\n|   Keneth|\n+---------+\n```\n\n# License and Contributing\n\nSmolder is made available under an [Apache 2.0 license](LICENSE), and we welcome\ncontributions from the community. Please see our [contibutor guidance](CONTRIBUTING.md)\nfor information about how to contribute to the project. To ensure that contributions\nto Smolder are properly licensed, we follow the [Developer Certificate of Origin\n(DCO)](http://developercertificate.org/) for all contributions to the project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fsmolder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks-industry-solutions%2Fsmolder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks-industry-solutions%2Fsmolder/lists"}