Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/data-tools/big-data-types
A library to transform Scala product types and Schemes from different systems into other Schemes. Any implemented type automatically gets methods to convert it into the rest of the types and vice versa. E.g: a Spark Schema can be transformed into a BigQuery table.
https://github.com/data-tools/big-data-types
apache-spark bigquery bigquery-tables cassandra circe database-types scala schemas spark typeclass typeclass-derivation typesafe
Last synced: 4 months ago
JSON representation
A library to transform Scala product types and Schemes from different systems into other Schemes. Any implemented type automatically gets methods to convert it into the rest of the types and vice versa. E.g: a Spark Schema can be transformed into a BigQuery table.
- Host: GitHub
- URL: https://github.com/data-tools/big-data-types
- Owner: data-tools
- License: apache-2.0
- Created: 2020-11-04T16:25:39.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2024-09-28T21:50:28.000Z (4 months ago)
- Last Synced: 2024-09-29T15:11:57.963Z (4 months ago)
- Topics: apache-spark, bigquery, bigquery-tables, cassandra, circe, database-types, scala, schemas, spark, typeclass, typeclass-derivation, typesafe
- Language: Scala
- Homepage: https://data-tools.github.io/big-data-types/
- Size: 3.65 MB
- Stars: 12
- Watchers: 4
- Forks: 3
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# Big Data Types
[](https://github.com/data-tools/big-data-types/actions/workflows/ci-tests.yml)
[](https://github.com/data-tools/big-data-types/actions/workflows/bigquery-integration.yml)
[](https://codecov.io/gh/data-tools/big-data-types)
[](https://scala-steward.org)
A type-safe library to transform Case Classes into Database schemas and to convert implemented types into other types
# Documentation
Check the [Documentation website](https://data-tools.github.io/big-data-types) to learn more about how to use this library
# Available conversions:
| From \ To | | Scala Types | BigQuery | Spark | Cassandra | Circe (JSON) |
|:------------:|:---------------------------------------------------------------------------------------------------------------------:|:-----------:|:------------------:|:------------------:|:------------------:|:------------:|
| Scala | 
| - | :white_check_mark: | :white_check_mark: | :white_check_mark: | |
| BigQuery | 
| | - | :white_check_mark: | :white_check_mark: | |
| Spark | 
| | :white_check_mark: | - | :white_check_mark: | |
| Cassandra | 
| | :white_check_mark: | :white_check_mark: | - | |
| Circe (JSON) | 
| | :white_check_mark: | :white_check_mark: | :white_check_mark: | - |
Versions for Scala  ,
and  are available in Maven
# Quick Start
The library has different modules that can be imported separately
- BigQuery
```
libraryDependencies += "io.github.data-tools" %% "big-data-types-bigquery" % "{version}"
```
- Spark
```
libraryDependencies += "io.github.data-tools" %% "big-data-types-spark" % "{version}"
```
- Cassandra
```
libraryDependencies += "io.github.data-tools" %% "big-data-types-cassandra" % "{version}"
```
- Circe (JSON)
```
libraryDependencies += "io.github.data-tools" %% "big-data-types-circe" % "{version}"
```
- Core
- To get support for abstract SqlTypes, it is included in the others, so it is not needed if you are using one of the others
```
libraryDependencies += "io.github.data-tools" %% "big-data-types-core" % "{version}"
```
In order to transform one type into another, both modules have to be imported.
## How it works
The library internally uses a generic ADT ([SqlType](https://github.com/data-tools/big-data-types/blob/main/core/src/main/scala_3/org/datatools/bigdatatypes/basictypes/SqlType.scala))
that can store any schema representation, and from there, it can be converted into any other.
Transformations are done through 2 different type-classes.
### Quick examples
Case Classes to other types
```scala
//Spark
val s: StructType = SparkSchemas.schema[MyCaseClass]
//BigQuery
val bq: List[Field] = SqlTypeToBigQuery[MyCaseClass].bigQueryFields // just the schema
BigQueryTable.createTable[MyCaseClass]("myDataset", "myTable") // Create a table in a BigQuery real environment
//Cassandra
val c: CreateTable = CassandraTables.table[MyCaseClass]
```
There are also `extension methods` that make easier the transformation between types when there are instances
```scala
//from Case Class instance
val foo: MyCaseClass = ???
foo.asBigQuery // List[Field]
foo.asSparkSchema // StructType
foo.asCassandra("TableName", "primaryKey") // CreateTable
```
**Conversion between types** works in the same way
```scala
// From Spark to others
val foo: StructType = myDataFrame.schema
foo.asBigQuery // List[Field]
foo.asCassandra("TableName", "primaryKey") // CreateTable
//From BigQuery to others
val foo: Schema = ???
foo.asSparkFields // List[StructField]
foo.asSparkSchema // StructType
foo.asCassandra("TableName", "primaryKey") // CreateTable
//From Cassandra to others
val foo: CreateTable = ???
foo.asSparkFields // List[StructField]
foo.asSparkSchema // StructType
foo.asBigQuery // List[Field]
foo.asBigQuery.schema // Schema
```