{"id":19542644,"url":"https://github.com/opensearch-project/opensearch-spark","last_synced_at":"2025-08-15T15:19:32.199Z","repository":{"id":185685017,"uuid":"664873197","full_name":"opensearch-project/opensearch-spark","owner":"opensearch-project","description":"Spark Accelerator framework ; It enables secondary indices to remote data stores. ","archived":false,"fork":false,"pushed_at":"2025-08-07T19:31:47.000Z","size":5064,"stargazers_count":37,"open_issues_count":219,"forks_count":47,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-08-07T21:21:49.711Z","etag":null,"topics":["compute","opensearch","secondary-index","spark"],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opensearch-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-07-11T00:24:44.000Z","updated_at":"2025-08-04T20:41:33.000Z","dependencies_parsed_at":"2024-11-07T03:21:04.949Z","dependency_job_id":"0d131aa2-7f5e-4fb5-a554-b114463baf9f","html_url":"https://github.com/opensearch-project/opensearch-spark","commit_stats":null,"previous_names":["opensearch-project/opensearch-spark"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/opensearch-project/opensearch-spark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-spark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-spark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-spark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-spark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opensearch-project","download_url":"https://codeload.github.com/opensearch-project/opensearch-spark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opensearch-project%2Fopensearch-spark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270586485,"owners_count":24611317,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-15T02:00:12.559Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["compute","opensearch","secondary-index","spark"],"created_at":"2024-11-11T03:15:25.698Z","updated_at":"2025-08-15T15:19:32.171Z","avatar_url":"https://github.com/opensearch-project.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenSearch Flint\n\nOpenSearch Flint is ... It consists of four modules:\n\n- `flint-core`: a module that contains Flint specification and client.\n- `flint-commons`: a module that provides a shared library of utilities and common functionalities, designed to easily extend Flint's capabilities.\n- `flint-spark-integration`: a module that provides Spark integration for Flint and derived dataset based on it.\n- `ppl-spark-integration`: a module that provides PPL query execution on top of Spark See [PPL repository](https://github.com/opensearch-project/piped-processing-language).\n\n## Documentation\n\nPlease refer to the [Flint Index Reference Manual](./docs/index.md) for more information.\n\n### PPL-Language\n\n* For additional details on PPL commands, see [PPL Commands Docs](docs/ppl-lang/README.md)\n\n* For additional details on Spark PPL Architecture, see [PPL Architecture](docs/ppl-lang/PPL-on-Spark.md)\n\n* For additional details on Spark PPL commands project, see [PPL Project](https://github.com/orgs/opensearch-project/projects/214/views/2)\n\n* Experiment ppl queries on local spark cluster [PPL on local spark ](docs/ppl-lang/local-spark-ppl-test-instruction.md)\n\n## Prerequisites\n\nVersion compatibility:\n\n| Flint version | JDK version | Spark version | Scala version | OpenSearch |\n|---------------|-------------|---------------|---------------|------------|\n| 0.1.0         | 11+         | 3.3.1         | 2.12.14       | 2.6+       |\n| 0.2.0         | 11+         | 3.3.1         | 2.12.14       | 2.6+       |\n| 0.3.0         | 11+         | 3.3.2         | 2.12.14       | 2.13+      |\n| 0.4.0         | 11+         | 3.3.2         | 2.12.14       | 2.13+      |\n| 0.5.0         | 11+         | 3.5.1         | 2.12.14       | 2.17+      |\n| 0.6.0         | 11+         | 3.5.1         | 2.12.14       | 2.17+      |\n| 0.7.0         | 11+         | 3.5.1         | 2.12.14       | 2.17+      |\n\n## Flint Extension Usage \n\nTo use this application, you can run Spark with Flint extension:\n\n```\nspark-sql --conf \"spark.sql.extensions=org.opensearch.flint.spark.FlintSparkExtensions\"\n```\n\n## PPL Extension Usage\n\nTo use PPL to Spark translation, you can run Spark with PPL extension:\n\n```\nspark-sql --conf \"spark.sql.extensions=org.opensearch.flint.spark.FlintPPLSparkExtensions\"\n```\n\n### Running With both Extension \n```\nspark-sql --conf \"spark.sql.extensions=org.opensearch.flint.spark.FlintPPLSparkExtensions,org.opensearch.flint.spark.FlintSparkExtensions\"\n```\n\n## Build\n\nTo build and run this application with Spark, you can run (requires Java 11):\n\n```\nsbt clean standaloneCosmetic/publishM2\n```\nthen add org.opensearch:opensearch-spark-standalone_2.12 when run spark application, for example,\n```\nbin/spark-shell --packages \"org.opensearch:opensearch-spark-standalone_2.12:1.0.0-SNAPSHOT\" \\\n                --conf \"spark.sql.extensions=org.opensearch.flint.spark.FlintSparkExtensions\" \\\n                --conf \"spark.sql.catalog.dev=org.apache.spark.opensearch.catalog.OpenSearchCatalog\"\n```\n\n### PPL Build \u0026 Run \n\nTo build and run this PPL in Spark, you can run (requires Java 11):\n\n```\nsbt clean sparkPPLCosmetic/publishM2\n```\n\nThen add org.opensearch:opensearch-spark-ppl_2.12 when run spark application, for example,\n\n```\nbin/spark-shell --packages \"org.opensearch:opensearch-spark-ppl_2.12:1.0.0-SNAPSHOT\" \\\n                --conf \"spark.sql.extensions=org.opensearch.flint.spark.FlintPPLSparkExtensions\" \\\n                --conf \"spark.sql.catalog.dev=org.apache.spark.opensearch.catalog.OpenSearchCatalog\"\n\n```\n\n### PPL Run queries on a local spark cluster\nSee ppl usage sample on local spark cluster [PPL on local spark ](docs/ppl-lang/local-spark-ppl-test-instruction.md)\n\n### Running integration tests on a local spark cluster\nSee integration test documentation [Docker Integration Tests](integ-test/script/README.md)\n\n## Code of Conduct\n\nThis project has adopted an [Open Source Code of Conduct](./CODE_OF_CONDUCT.md).\n\n## Security\n\nIf you discover a potential security issue in this project we ask that you notify OpenSearch Security directly via email to security@opensearch.org. Please do **not** create a public GitHub issue.\n\n## License\n\nSee the [LICENSE](./LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution.\n\n## Copyright\n\nCopyright OpenSearch Contributors. See [NOTICE](./NOTICE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensearch-project%2Fopensearch-spark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopensearch-project%2Fopensearch-spark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopensearch-project%2Fopensearch-spark/lists"}