{"id":28466089,"url":"https://github.com/zilliztech/spark-milvus","last_synced_at":"2025-07-29T01:35:51.576Z","repository":{"id":225093376,"uuid":"752514213","full_name":"zilliztech/spark-milvus","owner":"zilliztech","description":"spark connector for Milvus","archived":false,"fork":false,"pushed_at":"2025-06-05T07:50:07.000Z","size":16398,"stargazers_count":14,"open_issues_count":8,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-06-05T08:40:16.629Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zilliztech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-04T03:59:55.000Z","updated_at":"2025-06-05T07:36:28.000Z","dependencies_parsed_at":"2025-06-05T08:42:31.499Z","dependency_job_id":null,"html_url":"https://github.com/zilliztech/spark-milvus","commit_stats":null,"previous_names":["zilliztech/spark-milvus"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/zilliztech/spark-milvus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fspark-milvus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fspark-milvus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fspark-milvus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fspark-milvus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zilliztech","download_url":"https://codeload.github.com/zilliztech/spark-milvus/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zilliztech%2Fspark-milvus/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267616720,"owners_count":24116162,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-28T02:00:09.689Z","response_time":68,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-07T06:08:27.932Z","updated_at":"2025-07-29T01:35:51.571Z","avatar_url":"https://github.com/zilliztech.png","language":"Java","funding_links":[],"categories":["Data Integration \u0026 Migration"],"sub_categories":[],"readme":"# Spark Milvus Connector\n\n## Introduction\nThe Spark Milvus Connector provides seamless integration between Apache Spark and Milvus, enabling users to leverage Spark's processing capabilities alongside Milvus's vector data storage and query functionalities. With this integration, numerous interesting applications become feasible, including:\n- Transfer and integrate data between Milvus and various storage systems or databases\n- Processing and analysing the data in Milvus\n- Utilize Spark MLlib and other AI libraries for efficient vector processing operations.\n\n## Development\n\n\u003cb\u003ePrerequisites\u003c/b\u003e\n\n- Java 8 or higher\n- Apache Maven\n- Scala 2.12.15 (Please make sure you are using this version)\n- Spark 3.3.2 (Please make sure you are using this version)\n\n\u003cb\u003eClone \u0026 Build\u003c/b\u003e\n\n```bash\n# make the shade milvus-sdk package\ngit clone https://github.com/milvus-io/milvus-sdk-java.git\n# apply the pom.xml diff\n# https://github.com/SimFG/milvus-sdk-java/tree/master-build-expr\ncd milvus-sdk-java\nmvn clean package -DskipTests\n\ngit clone https://github.com/zilliztech/spark-milvus.git\ncd spark-milvus\n# HINT: Follow the comment\n# 1. modify the pom.xml file in the current directory and replace the systemPath path of the milvus-sdk-java dependency\n# 2. package the spark-milvus\nmvn clean package -DskipTests\n# 3. modify the pom.xml file in the example directory and replace the systemPath path of the milvus-sdk-java and spark-milvus dependency\n# 4. package the example\ncd examples\nmvn clean package -DskipTests\ncd ..\n```\nAfter the packaging, a new spark-milvus-1.0.0-SNAPSHOT.jar is generated in spark-milvus/target/ directory and a new spark-milvus-examples-1.0.0-SNAPSHOT.jar is generated in spark-milvus/example/target/.\n\n\n## Usages\n\n\n\u003cb\u003eEnable Spark Milvus Connector in Spark environment\u003c/b\u003e\n\nTo enable spark-milvus connector in Spark, add jar in Spark start command or add it in class path.\n\n```shell\n# pyspark\n./bin/pyspark --jars spark-milvus-1.0.0-SNAPSHOT.jar\n# spark-shell\n./bin/spark-shell --jars spark-milvus-1.0.0-SNAPSHOT.jar\n# spark-submit\n./bin/spark-submit --jars spark-milvus-1.0.0-SNAPSHOT.jar ......\n```\n\n\u003cb\u003eAdd spark-milvus-connector dependency in a maven project\u003c/b\u003e\n\n```scala\n\u003cdependency\u003e\n    \u003cgroupId\u003ezilliztech\u003c/groupId\u003e\n    \u003cartifactId\u003espark-milvus\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0-SNAPSHOT\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n### Quick Start\nIn this demo, we create a sample dataframe and write it to milvus through spark milvus connector. A collection will be created automatically based on the schema and some options.\n\npython\n```python\nfrom pyspark.sql import SparkSession\n\nspark = SparkSession.builder \\\n    .appName(\"Milvus Integration\") \\\n    .master(\"local[*]\") \\\n    .getOrCreate()\ncolumns = [\"id\", \"text\", \"vec\"]\ndata = [(1, \"a\", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]),\n    (2, \"b\", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]),\n    (3, \"c\", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0]),\n    (4, \"d\", [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0])]\nsample_df = spark.sparkContext.parallelize(data).toDF(columns)\nsample_df.write \\\n    .mode(\"append\") \\\n    .option(\"milvus.host\", \"localhost\") \\\n    .option(\"milvus.port\", \"19530\") \\\n    .option(\"milvus.collection.name\", \"hello_spark_milvus\") \\\n    .option(\"milvus.collection.vectorField\", \"vec\") \\\n    .option(\"milvus.collection.vectorDim\", \"8\") \\\n    .option(\"milvus.collection.primaryKeyField\", \"id\") \\\n    .format(\"milvus\") \\\n    .save()\n```\n\nThis source file is `examples/python/quickstar.py`. In the project root directory, execute the following command to run it:\n```bash\nspark-submit --jars /home/zilliz/Repo/milvus-sdk-java/sdk-core/target/milvus-sdk-java-new-shaded-2.5.4.jar,/home/zilliz/Repo/spark-milvus/target/spark-milvus-1.0.0-SNAPSHOT.jar  examples/py/quickstart.py\n```\n\nScala\n\n```scala\nimport org.apache.spark.sql.{SaveMode, SparkSession}\n\nobject Hello extends App {\n\n  val spark = SparkSession.builder().master(\"local[*]\")\n    .appName(\"HelloSparkMilvus\")\n    .getOrCreate()\n\n  import spark.implicits._\n\n  // Create DataFrame\n  val sampleDF = Seq(\n    (1, \"a\", Seq(1.0,2.0,3.0,4.0,5.0)),\n    (2, \"b\", Seq(1.0,2.0,3.0,4.0,5.0)),\n    (3, \"c\", Seq(1.0,2.0,3.0,4.0,5.0)),\n    (4, \"d\", Seq(1.0,2.0,3.0,4.0,5.0))\n  ).toDF(\"id\", \"text\", \"vec\")\n\n  // set milvus options\n  val milvusOptions = Map(\n      \"milvus.host\" -\u003e \"localhost\",\n      \"milvus.port\" -\u003e \"19530\",\n      \"milvus.collection.name\" -\u003e \"hello_spark_milvus\",\n      \"milvus.collection.vectorField\" -\u003e \"vec\",\n      \"milvus.collection.vectorDim\" -\u003e \"5\",\n      \"milvus.collection.primaryKeyField\" -\u003e \"id\"\n    )\n    \n  sampleDF.write.format(\"milvus\")\n    .options(milvusOptions)\n    .mode(SaveMode.Append)\n    .save()\n}\n```\n\nThis source file is `examples/src/main/scala/QuickStart.scala`. Before running, you need to ensure that the example directory has been packaged and the milvus service has been started locally. In the project root directory, execute the following command to run:\n\n```bash\nspark-submit --jars /home/zilliz/Repo/milvus-sdk-java/sdk-core/target/milvus-sdk-java-new-shaded-2.5.4.jar,/home/zilliz/Repo/spark-milvus/target/spark-milvus-1.0.0-SNAPSHOT.jar --class \"QuickStart\" examples/target/spark-milvus-examples-1.0.0-SNAPSHOT.jar\n```\n\n### More demos\n\n[scala](examples/src/main/scala/)\n- [InsertDemo](examples/src/main/scala/InsertDemo.scala)\n- [ReadDemo](examples/src/main/scala/ReadDemo.scala)\n- [ReadMilvusBinlogDemo](examples/src/main/scala/ReadMilvusBinlogDemo.scala)\n- [Mysql2MilvusDemo](examples/src/main/scala/Mysql2MilvusDemo.scala)\n- [BulkInsertDemo](examples/src/main/scala/BulkInsertDemo.scala)\n\n[python](examples/py/)\n- [QuickStart](examples/py/quickstart.py)\n\n[Databricks notebooks](examples/databricks-notebook)\n\n\n## Key Features \u0026 Concepts\n\n### Milvus Options\nMilvus Options are the configs to create Milvus connections and manage all other Milvus behaviors.\nEach spark-milvus job may need several(not all) of them.\nSee [MilvusOptions](src/main/scala/zilliztech/spark/milvus/MilvusOptions.scala) for detail\n\n### Milvus Data format\n\n\u003cb\u003emilvus\u003c/b\u003e\n\n\"milvus\" is a new data format supporting seemlessly write Spark DataFrame data into Milvus Collections. It is achieved by batch calls to the Insert API of Milvus SDK. A new collection will be created based on the schema of the dataframe if it doesn't exist in Milvus. However, the automatically created collection can't support all features of collection schema. It is recommended to create a collection via SDK first and do the writing with spark-milvus. Please refer to demo\n\n```scala\nval milvusOptions = Map(\n  MILVUS_HOST -\u003e host,\n  MILVUS_PORT -\u003e port.toString,\n  MILVUS_COLLECTION_NAME -\u003e collectionName,\n)\nval df = spark.write\n  .format(\"milvus\")\n  .options(milvusOptions)\n  .mode(SaveMode.Append)\n  .save()\n```\n\n\u003cb\u003emilvusbinlog\u003c/b\u003e\n\nNew data format \"milvusbinlog\" is for reading Milvus built-in binlog data.\nBinlog is Milvus's specific data storage format based on parquet.\nUnfortunately it can't be read by a regular parquet library.\nWe implement this new dataformat enabling Spark to read it.\nIt is not suggested to use \"milvusbinlog\" directly unless you are familiar to the milvus storage hierarchy and need to dig into the data.\nInstead, it is encapsulated within the MilvusUtils functions for a more convenient and recommended approach.\nMilvusUtils is introduced in the next section.\n\n```scala\nval df = spark.read\n  .format(\"milvusbinlog\")\n  .load(path)\n  .withColumnRenamed(\"val\", \"embedding\")\n```\n\n\u003cb\u003emjson\u003c/b\u003e\n\nMilvus provides Bulkinsert function(https://milvus.io/docs/bulk_insert.md) which has a better writing performance than Insert. However, the json format required by Milvus is different from Spark's output.\nTo resolve this, we introduce \"mjson\" data format to generate data that meets Milvus requirement.\n\n```scala\nval df = spark.write\n  .format(\"mjson\")\n  .mode(SaveMode.Overwrite)\n  .save(path)\n```\n\n### MilvusUtils\n\nMilvusUtils contains several util functions that make codes easier.\nCurrently it is only supported in Scala.\nPython developers can write your own utils.\nHow to use them is shown in Advanced Usage.\n\n\u003cB\u003eMilvusUtils.readMilvusCollection\u003c/B\u003e\n\nMilvusUtils.readMilvusCollection wrap needed SDK callings, milvusbinlog reading and some union/join logic to load a whole collection into a Spark dataframe.\n\n```scala\nval collectionDF = MilvusUtils.readMilvusCollection(spark, milvusOptions)\n```\n\n\u003cb\u003eMilvusUtils.bulkInsertFromSpark\u003c/b\u003e\n\nMilvusUtils.bulkInsertFromSpark provides a quick way to import Spark output files into Milvus via bullkinserts.\n\n```scala\ndf.write.format(\"parquet\").save(outputPath)\nMilvusUtils.bulkInsertFromSpark(spark, milvusOptions, outputPath, \"parquet\")\n```\n\n## Contributing\nContributions to spark-milvus are welcome from everyone.\nFeel free to create an issue if you meet any problem in using spark-milvus.\nGo to [Milvus discord](https://discord.com/channels/1160323594396635310) to discuss, raise questions, share thoughts with other users and developers.\n\n## License\nspark-milvus is licensed under the Server Side Public License v1 (SSPLv1) and the GNU Affero General Public License v3 (AGPLv3).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzilliztech%2Fspark-milvus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzilliztech%2Fspark-milvus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzilliztech%2Fspark-milvus/lists"}