{"id":22172958,"url":"https://github.com/EMCECS/spark-ecs-connector","last_synced_at":"2025-07-26T14:31:23.544Z","repository":{"id":57719052,"uuid":"51792525","full_name":"EMCECS/spark-ecs-connector","owner":"EMCECS","description":"[Archived] ArchiveECS connector for Apache Spark","archived":false,"fork":false,"pushed_at":"2017-10-10T16:24:59.000Z","size":21648,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-06-01T03:09:19.307Z","etag":null,"topics":["apache-spark","ecs"],"latest_commit_sha":null,"homepage":"https://www.dellemc.com/en-us/storage/ecs/index.htm","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EMCECS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-02-15T23:12:40.000Z","updated_at":"2023-09-02T00:15:49.000Z","dependencies_parsed_at":"2022-08-27T19:40:22.258Z","dependency_job_id":null,"html_url":"https://github.com/EMCECS/spark-ecs-connector","commit_stats":null,"previous_names":["emcecs/spark-ecs-s3","emcvipr/spark-ecs-s3"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/EMCECS/spark-ecs-connector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fspark-ecs-connector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fspark-ecs-connector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fspark-ecs-connector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fspark-ecs-connector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EMCECS","download_url":"https://codeload.github.com/EMCECS/spark-ecs-connector/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EMCECS%2Fspark-ecs-connector/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266048633,"owners_count":23868743,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-spark","ecs"],"created_at":"2024-12-02T07:27:56.947Z","updated_at":"2025-07-26T14:31:23.225Z","avatar_url":"https://github.com/EMCECS.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Bucket Metadata Search with Spark SQL (2.x)\n\nThe [spark-ecs-connector](https://github.com/emcvipr/spark-ecs-connector) project makes it possible to view an ECS bucket as a Spark dataframe. \nEach row in the dataframe corresponds to an object in the bucket, and each column coresponds to a piece of object metadata.\n\n**How it Works**\n\nSpark SQL supports querying external data sources and rendering the results as a dataframe.   With the [PrunedFilteredScan](https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.sources.PrunedFilteredScan) trait, the external data source handles column pruning and predicate pushdown.  In other words, the WHERE clause is pushed to ECS by taking advantage of the bucket metadata search feature of ECS 2.2.\n\n![Screenshot](screenshot.png)\n\n# Using\n\n### Linking to your Spark 2.x Application\nThe library is published to Maven Central.  Link to the library using these dependency coordinates: \n```\ncom.emc.ecs:spark-ecs-connector_2.11:1.4.2\n```\n\n### Using in Zeppelin\n1. Install Zeppelin 0.7+.\n2. `export SPARK_LOCAL_IP=127.0.0.1`\n3. `bin/zeppelin.sh`\n\nCreate a notebook with the following commands.   Replace `***` with your S3 credentials.\n\n```\n%dep\nz.load(\"com.emc.ecs:spark-ecs-connector_2.11:1.4.2\")\n```\n\n```\nimport java.net.URI\nimport com.emc.ecs.spark.sql.sources.s3._\n\nval endpointUri = new URI(\"http://10.1.83.51:9020/\")\nval credential = (\"***ACCESS KEY ID***\", \"***SECRET ACCESS KEY***\")\n\nval df = sqlContext.read.bucket(endpointUri, credential, \"ben_bucket\", withSystemMetadata = false)\ndf.createOrReplaceTempView(\"ben_bucket\")\n```\n\n```\n%sql\nSELECT * FROM ben_bucket \nWHERE `image-viewcount` \u003e= 5000 AND `image-viewcount` \u003c= 10000\n```\n\n# Contributing\n## Building\nThe project use the Gradle build system and includes a script that automatically downloads Gradle.\n\nBuild and install the library to your local Maven repository as follows:\n```\n$ ./gradlew publishShadowPublicationToMavenLocal\n```\n\n## TODO\n1. Implement 'OR' pushdown.  ECS supports 'or', but not in combination with 'and'.\n2. Avoid sending a query containing a non-indexable key.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEMCECS%2Fspark-ecs-connector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FEMCECS%2Fspark-ecs-connector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FEMCECS%2Fspark-ecs-connector/lists"}