https://github.com/apache/paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://github.com/apache/paimon

big-data data-ingestion flink paimon real-time-analytics spark streaming-datalake table-store

Last synced: 7 days ago
JSON representation

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

• Host: GitHub
• URL: https://github.com/apache/paimon
• Owner: apache
• License: apache-2.0
• Created: 2022-01-12T03:13:15.000Z (almost 3 years ago)
• Default Branch: master
• Last Pushed: 2024-10-29T08:25:53.000Z (2 months ago)
• Last Synced: 2024-10-29T09:47:21.262Z (2 months ago)
• Topics: big-data, data-ingestion, flink, paimon, real-time-analytics, spark, streaming-datalake, table-store
• Language: Java
• Homepage: https://paimon.apache.org/
• Size: 44.9 MB
• Stars: 2,389
• Watchers: 71
• Forks: 941
• Open Issues: 404
• Metadata Files:
  • Readme: README.md
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

• awesome-datalake - Apache Paimon - Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations. (Open Table Formats)
• awesome-datalake - Apache Paimon - Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations. (Open Table Formats)

README

        
![Paimon](https://github.com/apache/paimon/blob/master/docs/static/paimon-simple.png)

[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)

[![Get on Slack](https://img.shields.io/badge/slack-join-orange.svg)](https://the-asf.slack.com/archives/C053Q2NCW8G)

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark 

for both streaming and batch operations. Paimon innovatively combines lake format and LSM structure, bringing realtime 

streaming updates into the lake architecture.

Background and documentation are available at https://paimon.apache.org

`Paimon`'s former name was `Flink Table Store`, developed from the Flink community. The architecture refers to some 

design concepts of Iceberg. Thanks to Apache Flink and Apache Iceberg.

## Collaboration

Paimon tracks issues in GitHub and prefers to receive contributions as pull requests.

## Mailing Lists

  

    Name

    Subscribe

    Digest

    Unsubscribe

    Post

    Archive

  

  

    

      user@paimon.apache.org


      User support and questions mailing list

    

     Subscribe

     Subscribe

     Unsubscribe

     Post

    

      Archives

    

  

  

    

      dev@paimon.apache.org


      Development related discussions

    

     Subscribe

     Subscribe

     Unsubscribe

     Post

    

      Archives

    

  

Please make sure you are subscribed to the mailing list you are posting to! If you are not subscribed to the mailing list, your message will either be rejected (dev@ list) or you won't receive the response (user@ list).

## Slack

You can join the Paimon community on Slack. Paimon channel is in ASF Slack workspace.

- Anyone with an @apache.org email address can become a full member of the ASF Slack workspace.

  Search [Paimon channel](https://the-asf.slack.com/archives/C053Q2NCW8G) and join it.

- If you don't have an @apache.org email address, you can email to `[email protected]` to apply for an

  [ASF Slack invitation](https://infra.apache.org/slack.html). Then join [Paimon channel](https://the-asf.slack.com/archives/C053Q2NCW8G).

## Building

JDK 8/11 is required for building the project. Maven version >=3.3.1.

- Run the `mvn clean install -DskipTests` command to build the project.

- Run the `mvn spotless:apply` to format the project (both Java and Scala).

- IDE: Mark `paimon-common/target/generated-sources/antlr4` as Sources Root.

## How to Contribute

[Contribution Guide](https://paimon.apache.org/docs/master/project/contributing/).

## License

The code in this repository is licensed under the [Apache Software License 2](LICENSE).