Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/polaris-catalog/polaris
Apache Polaris, the interoperable, open source catalog for Apache Iceberg
https://github.com/polaris-catalog/polaris
apache iceberg polaris
Last synced: about 2 months ago
JSON representation
Apache Polaris, the interoperable, open source catalog for Apache Iceberg
- Host: GitHub
- URL: https://github.com/polaris-catalog/polaris
- Owner: apache
- License: apache-2.0
- Created: 2024-05-29T18:44:27.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-10-29T00:05:46.000Z (3 months ago)
- Last Synced: 2024-10-29T15:22:40.736Z (3 months ago)
- Topics: apache, iceberg, polaris
- Language: Python
- Homepage: https://polaris.apache.org/
- Size: 5.85 MB
- Stars: 1,118
- Watchers: 97
- Forks: 120
- Open Issues: 55
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-datalake - Polaris Catalog - Polaris Catalog is an open source catalog for Apache Iceberg. Polaris Catalog implements Iceberg’s open REST API for multi-engine interoperability with Apache Doris, Apache Flink, Apache Spark, PyIceberg, StarRocks and Trino. (Data Catalog)
README
# Apache Polaris (incubating)
Apache Polaris™ is an open-source, fully-featured catalog for Apache Iceberg™. It implements Iceberg's
[REST API](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml),
enabling seamless multi-engine interoperability across a wide range of platforms, including Apache Doris™, Apache Flink®,
Apache Spark™, StarRocks, and Trino.
Documentation is available at https://polaris.apache.org, including
[Polaris management API doc](https://polaris.apache.org/index.html#tag/polaris-management-service_other)
and [Apache Iceberg REST API doc](https://polaris.apache.org/index.html#tag/Configuration-API).
[Subscribe to the dev mailing list][dev-list-subscribe] to join discussions via email or browse [the archives](https://lists.apache.org/[email protected]). Check out the [CONTRIBUTING guide](CONTRIBUTING.md)
for contribution guidelines.
[](https://polaris-catalog.zulipchat.com/)
[](https://github.com/apache/polaris/actions/workflows/gradle.yml?query=branch%3Amain)
[dev-list-subscribe]: mailto:[email protected]
## Building and Running
Apache Polaris is organized into the following modules:
- `polaris-core` - The main Polaris entity definitions and core business logic
- `polaris-server` - The Polaris REST API server
- `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface
Apache Polaris is built using Gradle with Java 21+ and Docker 27+.
- `./gradlew build` - To build and run tests. Make sure Docker is running, as the integration tests depend on it.
- `./gradlew assemble` - To skip tests.
- `./gradlew test` - To run unit tests and integration tests.
- `./gradlew runApp` - To run the Polaris server locally on localhost:8181.
- `./regtests/run_spark_sql.sh` - To connect from Spark SQL. Here are some example commands to run in the Spark SQL shell:
```sql
create database db1;
show databases;
create table db1.table1 (id int, name string);
insert into db1.table1 values (1, 'a');
select * from db1.table1;
```
Apache Polaris supports the following optional build options:
- `-PeclipseLink=true` – Enables the EclipseLink extension.
- `-PeclipseLinkDeps=[groupId]:[artifactId]:[version],...` – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.
### More build and run options
Running in Docker
- `docker build -t localhost:5001/polaris:latest .` - To build the image.
- Optional build options:
- `docker build -t localhost:5001/polaris:latest --build-arg ECLIPSELINK=true .` - Enables the EclipseLink extension.
- `docker build -t localhost:5001/polaris:latest --build-arg ECLIPSELINK=true --build-arg ECLIPSELINK_DEPS=[groupId]:[artifactId]:[version],... .` – Enables the EclipseLink extension with one or more additional dependencies for EclipseLink (e.g. JDBC drivers) separated by commas.
- `docker run -p 8181:8181 localhost:5001/polaris:latest` - To run the image in standalone mode.
Running in Kubernetes
- `./run.sh` - To run Polaris as a mini-deployment locally. This will create one pod that bind itself to ports `8181` and `8182`.
- Optional run options:
- `./run.sh -b "ECLIPSELINK=true"` - Enables the EclipseLink extension.
- `./run.sh -b "ECLIPSELINK=true;ECLIPSELINK_DEPS=[groupId]:[artifactId]:[version],..."` – Enables the EclipseLink extension with one or more additional dependencies for EclipseLink (e.g. JDBC drivers) separated by commas.
- `kubectl port-forward svc/polaris-service -n polaris 8181:8181 8182:8182` - To create secure connections between a local machine and a pod within the cluster for both service and metrics endpoints.
- Currently supported metrics endpoints:
- localhost:8182/metrics
- localhost:8182/healthcheck
- `kubectl get pods -n polaris` - To check the status of the pods.
- `kubectl get deployment -n polaris` - To check the status of the deployment.
- `kubectl describe deployment polaris-deployment -n polaris` - To troubleshoot if things aren't working as expected.
Running regression tests
- `./regtests/run.sh` - To run regression tests in another terminal.
- `docker compose up --build --exit-code-from regtest` - To run regression tests in a Docker environment.
Building docs
- Docs are generated using [Hugo](https://gohugo.io/) using the [Docsy](https://www.docsy.dev/docs/) theme.
- To view the site locally, run
```bash
site/bin/run-hugo-in-docker.sh
```
- See [README in `site/`](site/README.md) for more information.
## License
Apache Polaris is under the Apache License Version 2.0. See the [LICENSE](LICENSE).
## ASF Incubator disclaimer
Apache Polaris™ is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Apache®, Apache Polaris™, Apache Iceberg™, Apache Spark™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.