{"id":13834862,"url":"https://github.com/chitralverma/scala-polars","last_synced_at":"2025-04-09T08:12:50.589Z","repository":{"id":131703109,"uuid":"584799586","full_name":"chitralverma/scala-polars","owner":"chitralverma","description":"Polars for Scala \u0026 Java projects!","archived":false,"fork":false,"pushed_at":"2025-03-17T22:49:28.000Z","size":4241,"stargazers_count":80,"open_issues_count":20,"forks_count":7,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-02T06:08:34.508Z","etag":null,"topics":["arrow","big-data","dataframe","dataframe-library","java","jni","polars","rust","scala"],"latest_commit_sha":null,"homepage":"https://chitralverma.github.io/scala-polars/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chitralverma.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-03T14:47:41.000Z","updated_at":"2025-02-24T01:28:44.000Z","dependencies_parsed_at":"2024-01-15T18:47:12.609Z","dependency_job_id":"f700ecb7-d22f-4f10-b35e-562ebc08596a","html_url":"https://github.com/chitralverma/scala-polars","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chitralverma%2Fscala-polars","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chitralverma%2Fscala-polars/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chitralverma%2Fscala-polars/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chitralverma%2Fscala-polars/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chitralverma","download_url":"https://codeload.github.com/chitralverma/scala-polars/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247999864,"owners_count":21031046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arrow","big-data","dataframe","dataframe-library","java","jni","polars","rust","scala"],"created_at":"2024-08-04T14:00:53.126Z","updated_at":"2025-04-09T08:12:50.559Z","avatar_url":"https://github.com/chitralverma.png","language":"Scala","funding_links":[],"categories":["Libraries/Packages/Scripts","数据科学"],"sub_categories":["Scala/Java"],"readme":"scala-polars\n============\n\n`scala-polars` is a library for using the awesome [Polars](https://www.pola.rs/) DataFrame library in\nScala and Java projects.\n\n## About\n\n### About Polars\n\nPolars is a blazing fast DataFrames library implemented in Rust using\n[Apache Arrow Columnar Format](https://arrow.apache.org/docs/format/Columnar.html) as the memory model.\n\n- Lazy / eager execution\n- Multithreaded\n- SIMD\n- Query optimization\n- Powerful expression API\n- Hybrid Streaming (larger than RAM datasets)\n- Rust | Python | NodeJS | ...\n\n### About scala-polars\n\nThis library has been written mostly in scala and leverages [JNI](https://en.wikipedia.org/wiki/Java_Native_Interface)\nto offload heavy data processing tasks to its native counterpart written completely in rust. The aim of this library is\nto provide an easy-to-use interface for Scala/ Java developers though which they can leverage the amazing Polars library\nin their existing projects.\n\nThe project is mainly divided into 2 submodules,\n\n- `core` - Contains the user facing interfaces written in scala that will be used to work with data. Internally this\n  module relies on native submodule.\n- `native` - This is an internal module written in rust which relies on the official rust implementation of Polars.\n\n### Examples\n\n- [Java Examples](examples/src/main/java/examples/java/)\n- [Scala Examples](examples/src/main/scala/examples/scala/)\n\n## Compatibility\n\n- JDK version `\u003e=8`\n- Scala version `2.12.x`, `2.13.x` and `3.3.x`. Default is `2.13.x`\n- Rust version `\u003e=1.58`\n\n## Building\n\n### Prerequisites\n\nThe following tooling is required to start building `scala-polars`,\n\n- JDK 8+ ([OpenJDK](https://openjdk.org/projects/jdk/)\n  or [Oracle Java SE](https://www.oracle.com/java/technologies/javase/))\n- [Rust](https://www.rust-lang.org/tools/install) (cargo, rustc etc.)\n- [sbt](https://www.scala-sbt.org/index.html)\n\n### How to Compile?\n\nsbt is the primary build tool for this project and all the required interlinking has been done in such a way that your\nIntelliJ IDE or an external build works in the same way. This means that whether you are in development mode or want to\nbuild to distribute, the process of the build remains the same and is more or less abstracted.\n\nThe build process that sbt triggers involves the following steps,\n\n- Compile the rust code present in the `native` module to a binary.\n- Compile the scala and java (if any) facade code.\n- Copy the built rust binary to the classpath of scala code during its build at a fixed location.\n\nAll of the above steps happen automatically when you run an sbt build job that triggers `compile` phase. Other than\nthis, during package phase, the scala, java code and the built rust binary is added to the built jar(s). To keep\neverything monolithic, the `native` module is not packaged as a jar, only `core` module is.\n\nThe above process might look complicated, and it actually is 😂, but since all the internally sbt wiring is already in\nplace, the user facing process is fairly straight-forward. This can be done by going through the following steps in\nsequence firstly ensure JDK 8+, sbt and the latest rust\ncompiler are installed, then follow the commands below as per the need.\n\n**Compilation**\n\n```shell\n# To compile the whole project (scala/ java/ rust) in one go\nsbt compile\n```\n\n**Local packaging/ installation**\n\n```shell\n# To package the project and install locally as slim jars with default scala version.\nsbt publishLocal\n\n# To package the project and install locally as slim jars for all supported scala versions.\nsbt +publishLocal\n```\n\n**Build Assembly (fat jar)**\n\n```shell\n# To package the project and install locally as fat jars with default scala version.\nsbt assembly\n\n# To package the project and install locally as slim jars for all supported scala versions.\nsbt +assembly\n```\n\n**Generate Native Binary Only**\n\n```shell\n# To compile only the native module containing rust code to binary.\nsbt generateNativeLibrary\n```\n\n## License\n\nApache License 2.0, see [LICENSE](LICENSE).\n\n## Community\n\nReach out to the Polars community on [Discord](https://discord.gg/4UfP5cfBE7).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchitralverma%2Fscala-polars","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchitralverma%2Fscala-polars","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchitralverma%2Fscala-polars/lists"}