{"id":13794977,"url":"https://github.com/databricks/spark-csv","last_synced_at":"2025-05-15T04:00:21.578Z","repository":{"id":24075199,"uuid":"27461793","full_name":"databricks/spark-csv","owner":"databricks","description":"CSV Data Source for Apache Spark 1.x","archived":false,"fork":false,"pushed_at":"2018-12-13T09:50:29.000Z","size":438,"stargazers_count":1052,"open_issues_count":204,"forks_count":441,"subscribers_count":420,"default_branch":"master","last_synced_at":"2025-05-08T05:19:08.157Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://databricks.com/","language":"Scala","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/databricks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-03T00:56:21.000Z","updated_at":"2025-02-19T15:22:53.000Z","dependencies_parsed_at":"2022-08-22T10:11:17.277Z","dependency_job_id":null,"html_url":"https://github.com/databricks/spark-csv","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fspark-csv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fspark-csv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fspark-csv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/databricks%2Fspark-csv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/databricks","download_url":"https://codeload.github.com/databricks/spark-csv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254270640,"owners_count":22042858,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T23:00:50.756Z","updated_at":"2025-05-15T04:00:21.538Z","avatar_url":"https://github.com/databricks.png","language":"Scala","readme":"# CSV Data Source for Apache Spark 1.x\n\n__NOTE: This functionality has been inlined in Apache Spark 2.x. This package is in maintenance mode and we only accept critical bug fixes.__\n\nA library for parsing and querying CSV data with Apache Spark, for Spark SQL and DataFrames.\n\n[![Build Status](https://travis-ci.org/databricks/spark-csv.svg?branch=master)](https://travis-ci.org/databricks/spark-csv)\n[![codecov.io](http://codecov.io/github/databricks/spark-csv/coverage.svg?branch=master)](http://codecov.io/github/databricks/spark-csv?branch=master)\n\n## Requirements\n\nThis library requires Spark 1.3+\n\n## Linking\nYou can link against this library in your program at the following coordinates:\n\n### Scala 2.10\n```\ngroupId: com.databricks\nartifactId: spark-csv_2.10\nversion: 1.5.0\n```\n### Scala 2.11\n```\ngroupId: com.databricks\nartifactId: spark-csv_2.11\nversion: 1.5.0\n```\n\n## Using with Spark shell\nThis package can be added to  Spark using the `--packages` command line option.  For example, to include it when starting the spark shell:\n\n### Spark compiled with Scala 2.11\n```\n$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.11:1.5.0\n```\n\n### Spark compiled with Scala 2.10\n```\n$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.5.0\n```\n\n## Features\nThis package allows reading CSV files in local or distributed filesystem as [Spark DataFrames](https://spark.apache.org/docs/1.6.0/sql-programming-guide.html).\nWhen reading files the API accepts several options:\n* `path`: location of files. Similar to Spark can accept standard Hadoop globbing expressions.\n* `header`: when set to true the first line of files will be used to name columns and will not be included in data. All types will be assumed string. Default value is false.\n* `delimiter`: by default columns are delimited using `,`, but delimiter can be set to any character\n* `quote`: by default the quote character is `\"`, but can be set to any character. Delimiters inside quotes are ignored\n* `escape`: by default the escape character is `\\`, but can be set to any character. Escaped quote characters are ignored\n* `parserLib`: by default it is \"commons\" can be set to \"univocity\" to use that library for CSV parsing.\n* `mode`: determines the parsing mode. By default it is PERMISSIVE. Possible values are:\n  * `PERMISSIVE`: tries to parse all lines: nulls are inserted for missing tokens and extra tokens are ignored.\n  * `DROPMALFORMED`: drops lines which have fewer or more tokens than expected or tokens which do\n   not match the schema\n  * `FAILFAST`: aborts with a RuntimeException if encounters any malformed line\n* `charset`: defaults to 'UTF-8' but can be set to other valid charset names\n* `inferSchema`: automatically infers column types. It requires one extra pass over the data and is false by default\n* `comment`: skip lines beginning with this character. Default is `\"#\"`. Disable comments by setting this to `null`.\n* `nullValue`: specifies a string that indicates a null value, any fields matching this string will be set as nulls in the DataFrame\n* `dateFormat`: specifies a string that indicates the date format to use when reading dates or timestamps. Custom date formats follow the formats at [`java.text.SimpleDateFormat`](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). This applies to both `DateType` and `TimestampType`. By default, it is `null` which means trying to parse times and date by `java.sql.Timestamp.valueOf()` and `java.sql.Date.valueOf()`.\n\nThe package also supports saving simple (non-nested) DataFrame. When writing files the API accepts several options:\n* `path`: location of files.\n* `header`: when set to true, the header (from the schema in the DataFrame) will be written at the first line.\n* `delimiter`: by default columns are delimited using `,`, but delimiter can be set to any character\n* `quote`: by default the quote character is `\"`, but can be set to any character. This is written according to `quoteMode`.\n* `escape`: by default the escape character is `\\`, but can be set to any character. Escaped quote characters are written.\n* `nullValue`: specifies a string that indicates a null value, nulls in the DataFrame will be written as this string.\n* `dateFormat`: specifies a string that indicates the date format to use writing dates or timestamps. Custom date formats follow the formats at [`java.text.SimpleDateFormat`](https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). This applies to both `DateType` and `TimestampType`. If no dateFormat is specified, then \"yyyy-MM-dd HH:mm:ss.S\".\n* `codec`: compression codec to use when saving to file. Should be the fully qualified name of a class implementing `org.apache.hadoop.io.compress.CompressionCodec` or one of case-insensitive shorten names (`bzip2`, `gzip`, `lz4`, and `snappy`). Defaults to no compression when a codec is not specified.\n* `quoteMode`: when to quote fields (`ALL`, `MINIMAL` (default), `NON_NUMERIC`, `NONE`), see [Quote Modes](https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/QuoteMode.html)\n\nThese examples use a CSV file available for download [here](https://github.com/databricks/spark-csv/raw/master/src/test/resources/cars.csv):\n\n```\n$ wget https://github.com/databricks/spark-csv/raw/master/src/test/resources/cars.csv\n```\n\n### SQL API\n\nCSV data source for Spark can infer data types:\n```sql\nCREATE TABLE cars\nUSING com.databricks.spark.csv\nOPTIONS (path \"cars.csv\", header \"true\", inferSchema \"true\")\n```\n\nYou can also specify column names and types in DDL.\n```sql\nCREATE TABLE cars (yearMade double, carMake string, carModel string, comments string, blank string)\nUSING com.databricks.spark.csv\nOPTIONS (path \"cars.csv\", header \"true\")\n```\n\n### Scala API\n__Spark 1.4+:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```scala\nimport org.apache.spark.sql.SQLContext\n\nval sqlContext = new SQLContext(sc)\nval df = sqlContext.read\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\") // Use first line of all files as header\n    .option(\"inferSchema\", \"true\") // Automatically infer data types\n    .load(\"cars.csv\")\n\nval selectedData = df.select(\"year\", \"model\")\nselectedData.write\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\")\n    .save(\"newcars.csv\")\n```\n\nYou can manually specify the schema when reading data:\n```scala\nimport org.apache.spark.sql.SQLContext\nimport org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType}\n\nval sqlContext = new SQLContext(sc)\nval customSchema = StructType(Array(\n    StructField(\"year\", IntegerType, true),\n    StructField(\"make\", StringType, true),\n    StructField(\"model\", StringType, true),\n    StructField(\"comment\", StringType, true),\n    StructField(\"blank\", StringType, true)))\n\nval df = sqlContext.read\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\") // Use first line of all files as header\n    .schema(customSchema)\n    .load(\"cars.csv\")\n\nval selectedData = df.select(\"year\", \"model\")\nselectedData.write\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\")\n    .save(\"newcars.csv\")\n```\n\nYou can save with compressed output:\n```scala\nimport org.apache.spark.sql.SQLContext\n\nval sqlContext = new SQLContext(sc)\nval df = sqlContext.read\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\") // Use first line of all files as header\n    .option(\"inferSchema\", \"true\") // Automatically infer data types\n    .load(\"cars.csv\")\n\nval selectedData = df.select(\"year\", \"model\")\nselectedData.write\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\")\n    .option(\"codec\", \"org.apache.hadoop.io.compress.GzipCodec\")\n    .save(\"newcars.csv.gz\")\n```\n\n__Spark 1.3:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```scala\nimport org.apache.spark.sql.SQLContext\n\nval sqlContext = new SQLContext(sc)\nval df = sqlContext.load(\n    \"com.databricks.spark.csv\",\n    Map(\"path\" -\u003e \"cars.csv\", \"header\" -\u003e \"true\", \"inferSchema\" -\u003e \"true\"))\nval selectedData = df.select(\"year\", \"model\")\nselectedData.save(\"newcars.csv\", \"com.databricks.spark.csv\")\n```\n\nYou can manually specify the schema when reading data:\n```scala\nimport org.apache.spark.sql.SQLContext\nimport org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType};\n\nval sqlContext = new SQLContext(sc)\nval customSchema = StructType(Array(\n    StructField(\"year\", IntegerType, true),\n    StructField(\"make\", StringType, true),\n    StructField(\"model\", StringType, true),\n    StructField(\"comment\", StringType, true),\n    StructField(\"blank\", StringType, true)))\n\nval df = sqlContext.load(\n    \"com.databricks.spark.csv\",\n    schema = customSchema,\n    Map(\"path\" -\u003e \"cars.csv\", \"header\" -\u003e \"true\"))\n\nval selectedData = df.select(\"year\", \"model\")\nselectedData.save(\"newcars.csv\", \"com.databricks.spark.csv\")\n```\n\n### Java API\n__Spark 1.4+:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```java\nimport org.apache.spark.sql.SQLContext\n\nSQLContext sqlContext = new SQLContext(sc);\nDataFrame df = sqlContext.read()\n    .format(\"com.databricks.spark.csv\")\n    .option(\"inferSchema\", \"true\")\n    .option(\"header\", \"true\")\n    .load(\"cars.csv\");\n\ndf.select(\"year\", \"model\").write()\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\")\n    .save(\"newcars.csv\");\n```\n\nYou can manually specify schema:\n```java\nimport org.apache.spark.sql.SQLContext;\nimport org.apache.spark.sql.types.*;\n\nSQLContext sqlContext = new SQLContext(sc);\nStructType customSchema = new StructType(new StructField[] {\n    new StructField(\"year\", DataTypes.IntegerType, true, Metadata.empty()),\n    new StructField(\"make\", DataTypes.StringType, true, Metadata.empty()),\n    new StructField(\"model\", DataTypes.StringType, true, Metadata.empty()),\n    new StructField(\"comment\", DataTypes.StringType, true, Metadata.empty()),\n    new StructField(\"blank\", DataTypes.StringType, true, Metadata.empty())\n});\n\nDataFrame df = sqlContext.read()\n    .format(\"com.databricks.spark.csv\")\n    .schema(customSchema)\n    .option(\"header\", \"true\")\n    .load(\"cars.csv\");\n\ndf.select(\"year\", \"model\").write()\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\")\n    .save(\"newcars.csv\");\n```\n\nYou can save with compressed output:\n```java\nimport org.apache.spark.sql.SQLContext\n\nSQLContext sqlContext = new SQLContext(sc);\nDataFrame df = sqlContext.read()\n    .format(\"com.databricks.spark.csv\")\n    .option(\"inferSchema\", \"true\")\n    .option(\"header\", \"true\")\n    .load(\"cars.csv\");\n\ndf.select(\"year\", \"model\").write()\n    .format(\"com.databricks.spark.csv\")\n    .option(\"header\", \"true\")\n    .option(\"codec\", \"org.apache.hadoop.io.compress.GzipCodec\")\n    .save(\"newcars.csv\");\n```\n\n__Spark 1.3:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```java\nimport org.apache.spark.sql.SQLContext\n\nSQLContext sqlContext = new SQLContext(sc);\n\nHashMap\u003cString, String\u003e options = new HashMap\u003cString, String\u003e();\noptions.put(\"header\", \"true\");\noptions.put(\"path\", \"cars.csv\");\noptions.put(\"inferSchema\", \"true\");\n\nDataFrame df = sqlContext.load(\"com.databricks.spark.csv\", options);\ndf.select(\"year\", \"model\").save(\"newcars.csv\", \"com.databricks.spark.csv\");\n```\n\nYou can manually specify schema:\n```java\nimport org.apache.spark.sql.SQLContext;\nimport org.apache.spark.sql.types.*;\n\nSQLContext sqlContext = new SQLContext(sc);\nStructType customSchema = new StructType(new StructField[] {\n    new StructField(\"year\", DataTypes.IntegerType, true, Metadata.empty()),\n    new StructField(\"make\", DataTypes.StringType, true, Metadata.empty()),\n    new StructField(\"model\", DataTypes.StringType, true, Metadata.empty()),\n    new StructField(\"comment\", DataTypes.StringType, true, Metadata.empty()),\n    new StructField(\"blank\", DataTypes.StringType, true, Metadata.empty())\n});\n\nHashMap\u003cString, String\u003e options = new HashMap\u003cString, String\u003e();\noptions.put(\"header\", \"true\");\noptions.put(\"path\", \"cars.csv\");\n\nDataFrame df = sqlContext.load(\"com.databricks.spark.csv\", customSchema, options);\ndf.select(\"year\", \"model\").save(\"newcars.csv\", \"com.databricks.spark.csv\");\n```\n\nYou can save with compressed output:\n```java\nimport org.apache.spark.sql.SQLContext;\nimport org.apache.spark.sql.SaveMode;\n\nSQLContext sqlContext = new SQLContext(sc);\n\nHashMap\u003cString, String\u003e options = new HashMap\u003cString, String\u003e();\noptions.put(\"header\", \"true\");\noptions.put(\"path\", \"cars.csv\");\noptions.put(\"inferSchema\", \"true\");\n\nDataFrame df = sqlContext.load(\"com.databricks.spark.csv\", options);\n\nHashMap\u003cString, String\u003e saveOptions = new HashMap\u003cString, String\u003e();\nsaveOptions.put(\"header\", \"true\");\nsaveOptions.put(\"path\", \"newcars.csv\");\nsaveOptions.put(\"codec\", \"org.apache.hadoop.io.compress.GzipCodec\");\n\ndf.select(\"year\", \"model\").save(\"com.databricks.spark.csv\", SaveMode.Overwrite,\n                                saveOptions);\n```\n\n### Python API\n\n__Spark 1.4+:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```python\nfrom pyspark.sql import SQLContext\nsqlContext = SQLContext(sc)\n\ndf = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')\ndf.select('year', 'model').write.format('com.databricks.spark.csv').save('newcars.csv')\n```\n\nYou can manually specify schema:\n```python\nfrom pyspark.sql import SQLContext\nfrom pyspark.sql.types import *\n\nsqlContext = SQLContext(sc)\ncustomSchema = StructType([ \\\n    StructField(\"year\", IntegerType(), True), \\\n    StructField(\"make\", StringType(), True), \\\n    StructField(\"model\", StringType(), True), \\\n    StructField(\"comment\", StringType(), True), \\\n    StructField(\"blank\", StringType(), True)])\n\ndf = sqlContext.read \\\n    .format('com.databricks.spark.csv') \\\n    .options(header='true') \\\n    .load('cars.csv', schema = customSchema)\n\ndf.select('year', 'model').write \\\n    .format('com.databricks.spark.csv') \\\n    .save('newcars.csv')\n```\n\nYou can save with compressed output:\n```python\nfrom pyspark.sql import SQLContext\nsqlContext = SQLContext(sc)\n\ndf = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')\ndf.select('year', 'model').write.format('com.databricks.spark.csv').options(codec=\"org.apache.hadoop.io.compress.GzipCodec\").save('newcars.csv')\n```\n\n__Spark 1.3:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```python\nfrom pyspark.sql import SQLContext\nsqlContext = SQLContext(sc)\n\ndf = sqlContext.load(source=\"com.databricks.spark.csv\", header = 'true', inferSchema = 'true', path = 'cars.csv')\ndf.select('year', 'model').save('newcars.csv', 'com.databricks.spark.csv')\n```\n\nYou can manually specify schema:\n```python\nfrom pyspark.sql import SQLContext\nfrom pyspark.sql.types import *\n\nsqlContext = SQLContext(sc)\ncustomSchema = StructType([ \\\n    StructField(\"year\", IntegerType(), True), \\\n    StructField(\"make\", StringType(), True), \\\n    StructField(\"model\", StringType(), True), \\\n    StructField(\"comment\", StringType(), True), \\\n    StructField(\"blank\", StringType(), True)])\n\ndf = sqlContext.load(source=\"com.databricks.spark.csv\", header = 'true', schema = customSchema, path = 'cars.csv')\ndf.select('year', 'model').save('newcars.csv', 'com.databricks.spark.csv')\n```\n\nYou can save with compressed output:\n```python\nfrom pyspark.sql import SQLContext\nsqlContext = SQLContext(sc)\n\ndf = sqlContext.load(source=\"com.databricks.spark.csv\", header = 'true', inferSchema = 'true', path = 'cars.csv')\ndf.select('year', 'model').save('newcars.csv', 'com.databricks.spark.csv', codec=\"org.apache.hadoop.io.compress.GzipCodec\")\n```\n\n### R API\n__Spark 1.4+:__\n\nAutomatically infer schema (data types), otherwise everything is assumed string:\n```R\nlibrary(SparkR)\n\nSys.setenv('SPARKR_SUBMIT_ARGS'='\"--packages\" \"com.databricks:spark-csv_2.10:1.4.0\" \"sparkr-shell\"')\nsqlContext \u003c- sparkRSQL.init(sc)\n\ndf \u003c- read.df(sqlContext, \"cars.csv\", source = \"com.databricks.spark.csv\", inferSchema = \"true\")\n\nwrite.df(df, \"newcars.csv\", \"com.databricks.spark.csv\", \"overwrite\")\n```\n\nYou can manually specify schema:\n```R\nlibrary(SparkR)\n\nSys.setenv('SPARKR_SUBMIT_ARGS'='\"--packages\" \"com.databricks:spark-csv_2.10:1.4.0\" \"sparkr-shell\"')\nsqlContext \u003c- sparkRSQL.init(sc)\ncustomSchema \u003c- structType(\n    structField(\"year\", \"integer\"),\n    structField(\"make\", \"string\"),\n    structField(\"model\", \"string\"),\n    structField(\"comment\", \"string\"),\n    structField(\"blank\", \"string\"))\n\ndf \u003c- read.df(sqlContext, \"cars.csv\", source = \"com.databricks.spark.csv\", schema = customSchema)\n\nwrite.df(df, \"newcars.csv\", \"com.databricks.spark.csv\", \"overwrite\")\n```\n\nYou can save with compressed output:\n```R\nlibrary(SparkR)\n\nSys.setenv('SPARKR_SUBMIT_ARGS'='\"--packages\" \"com.databricks:spark-csv_2.10:1.4.0\" \"sparkr-shell\"')\nsqlContext \u003c- sparkRSQL.init(sc)\n\ndf \u003c- read.df(sqlContext, \"cars.csv\", source = \"com.databricks.spark.csv\", inferSchema = \"true\")\n\nwrite.df(df, \"newcars.csv\", \"com.databricks.spark.csv\", \"overwrite\", codec=\"org.apache.hadoop.io.compress.GzipCodec\")\n```\n\n## Building From Source\nThis library is built with [SBT](http://www.scala-sbt.org/0.13/docs/Command-Line-Reference.html), which is automatically downloaded by the included shell script. To build a JAR file simply run `sbt/sbt package` from the project root. The build configuration includes support for both Scala 2.10 and 2.11.\n","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fspark-csv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabricks%2Fspark-csv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabricks%2Fspark-csv/lists"}