{"id":13481815,"url":"https://github.com/hazelcast/hazelcast-jet","last_synced_at":"2025-10-17T15:30:40.147Z","repository":{"id":37256880,"uuid":"48046454","full_name":"hazelcast/hazelcast-jet","owner":"hazelcast","description":"Distributed Stream and Batch Processing","archived":false,"fork":false,"pushed_at":"2024-12-19T11:55:10.000Z","size":179009,"stargazers_count":1104,"open_issues_count":127,"forks_count":207,"subscribers_count":77,"default_branch":"master","last_synced_at":"2025-01-26T17:01:31.568Z","etag":null,"topics":["batch-processing","big-data","cdc","event-processing","hacktoberfest","java","kafka","low-latency","stream-processing"],"latest_commit_sha":null,"homepage":"https://jet-start.sh","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hazelcast.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-12-15T14:01:14.000Z","updated_at":"2025-01-11T06:25:30.000Z","dependencies_parsed_at":"2025-01-26T17:10:51.140Z","dependency_job_id":null,"html_url":"https://github.com/hazelcast/hazelcast-jet","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hazelcast%2Fhazelcast-jet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hazelcast%2Fhazelcast-jet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hazelcast%2Fhazelcast-jet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hazelcast%2Fhazelcast-jet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hazelcast","download_url":"https://codeload.github.com/hazelcast/hazelcast-jet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":236822949,"owners_count":19210383,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","big-data","cdc","event-processing","hacktoberfest","java","kafka","low-latency","stream-processing"],"created_at":"2024-07-31T17:00:56.134Z","updated_at":"2025-10-17T15:30:30.135Z","avatar_url":"https://github.com/hazelcast.png","language":"Java","funding_links":[],"categories":["Table of Contents","Java","大数据"],"sub_categories":["Streaming Engine"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cp\u003e\n\u003cimg src=\"images/hazelcast-jet.png\" width=\"150\" style=\"margin-bottom: 14px\"\u003e\n\u003c/p\u003e\n\u003cp\u003e\n\u003ca href=\"https://slack.hazelcast.com\"\u003e\u003cimg\nsrc=\"https://hz-community-slack.herokuapp.com/badge.svg\" alt=\"Join the\ncommunity on Slack\"\u003e\u003c/a\u003e\n\u003ca\nhref=\"https://lgtm.com/projects/g/hazelcast/hazelcast-jet/context:java\"\u003e\u003cimg\nsrc=\"https://img.shields.io/lgtm/grade/java/g/hazelcast/hazelcast.svg?logo=lgtm\u0026amp;logoWidth=18\"\nalt=\"Code Quality: Java\"\u003e\u003c/a\u003e\n\u003ca\nhref=\"https://img.shields.io/docker/pulls/hazelcast/hazelcast-jet\"\u003e\u003cimg\nsrc=\"https://img.shields.io/docker/pulls/hazelcast/hazelcast-jet\"\nalt=\"Docker pulls\"\u003e\u003c/a\u003e\n\u003ca\nhref=\"https://img.shields.io/github/downloads/hazelcast/hazelcast-jet/total\"\u003e\u003cimg\nsrc=\"https://img.shields.io/github/downloads/hazelcast/hazelcast-jet/total\"\nalt=\"Downloads\"\u003e\u003c/a\u003e\n\u003ca\nhref=\"https://img.shields.io/github/contributors/hazelcast/hazelcast-jet\"\u003e\u003cimg\nsrc=\"https://img.shields.io/github/contributors/hazelcast/hazelcast-jet\"\nalt=\"Contributors\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\n# Note on Hazelcast 5\n\nWith the release of Hazelcast 5.0, development of Jet has been moved to the \n[core Hazelcast Repository](https://github.com/hazelcast/hazelcast) - please \nfollow the repository for details on  how to use Hazelcast for building data pipelines.\n\nHazelcast 5 also comes with extensive documentation, replacing the existing Jet\ndocs: https://docs.hazelcast.com/hazelcast/latest/index.html\n\n# What is Jet\n\n[Jet](https://jet-start.sh/) is an open-source, in-memory, distributed\nbatch and stream processing engine. You can use it to process large\nvolumes of real-time events or huge batches of static datasets. To give\na sense of scale, a single node of Jet has been proven to [aggregate 10\nmillion events per\nsecond](https://jet-start.sh/blog/2020/08/05/gc-tuning-for-jet) with\nlatency under 10 milliseconds.\n\nIt provides a Java API to build stream and batch processing applications\nthrough the use of a [dataflow programming\nmodel](https://jet-start.sh/docs/concepts/dag). After you deploy your\napplication to a Jet cluster, Jet will automatically use all the\ncomputational resources on the cluster to run your application.\n\nIf you add more nodes to the cluster while your application is running,\nJet automatically scales up your application to run on the new nodes. If\nyou remove nodes from the cluster, it scales it down seamlessly without\nlosing the current computational state, providing [exactly-once\nprocessing](https://jet-start.sh/docs/architecture/fault-tolerance)\nguarantees.\n\nFor example, you can represent the classical word count problem that\nreads some local files and outputs the frequency of each word to console\nusing the following API:\n\n```java\nJetInstance jet = Jet.bootstrappedInstance();\n\nPipeline p = Pipeline.create();\np.readFrom(Sources.files(\"/path/to/text-files\"))\n .flatMap(line -\u003e traverseArray(line.toLowerCase().split(\"\\\\W+\")))\n .filter(word -\u003e !word.isEmpty())\n .groupingKey(word -\u003e word)\n .aggregate(counting())\n .writeTo(Sinks.logger());\n\njet.newJob(p).join();\n```\n\nand then deploy the application to the cluster:\n\n```bash\nbin/jet submit word-count.jar\n```\n\nAnother application which aggregates millions of sensor readings per\nsecond with 10-millisecond resolution from Kafka looks like the\nfollowing:\n\n```java\nPipeline p = Pipeline.create();\n\np.readFrom(KafkaSources.\u003cString, Reading\u003ekafka(kafkaProperties, \"sensors\"))\n .withTimestamps(event -\u003e event.getValue().timestamp(), 10) // use event timestamp, allowed lag in ms\n .groupingKey(reading -\u003e reading.sensorId())\n .window(sliding(1_000, 10)) // sliding window of 1s by 10ms\n .aggregate(averagingDouble(reading -\u003e reading.temperature()))\n .writeTo(Sinks.logger());\n\njet.newJob(p).join();\n```\n\nJet comes with out-of-the-box support for many kinds of [data sources\nand sinks](https://jet-start.sh/docs/api/sources-sinks), including:\n\n* Apache Kafka\n* Local Files (Text, Avro, JSON)\n* Apache Hadoop (Azure Data Lake, S3, GCS)\n* Apache Pulsar\n* Debezium\n* Elasticsearch\n* JDBC\n* JMS\n* InfluxDB\n* Hazelcast\n* Redis\n* MongoDB\n* Twitter\n\n## When Should You Use Jet\n\nJet is a good fit when you need to process large amounts of data in a\ndistributed fashion. You can use it to build a variety of\ndata-processing applications, such as:\n\n* Low-latency stateful stream processing. For example, detecting trends\n  in 100 Hz sensor data from 100,000 devices and sending corrective\n  feedback within 10 milliseconds.\n* High-throughput, large-state stream processing. For example,\n  tracking GPS locations of millions of users, inferring their velocity\n  vectors.\n* Batch processing of big data volumes, for example analyzing a\n  day's worth of stock trading data to update the risk exposure of a\n  given portfolio.\n\n## Key Features\n\n### Predictable Latency Under Load\n\nJet uses a unique execution model with [cooperative\nmultithreading](https://jet-start.sh/docs/architecture/execution-engine)\nand can achieve [extremely low\nlatencies](https://jet-start.sh/blog/2020/08/05/gc-tuning-for-jet) while\nprocessing millions of items per second on just a single node:\n\n\u003cimg src=\"images/latency.png\"/\u003e\n\nThe engine is able to run anywhere from tens to thousands of jobs\nconcurrently on a fixed number of threads.\n\n### Fault Tolerance With No Infrastructure\n\nJet stores computational state in a distributed, replicated [in-memory\nstore](https://jet-start.sh/docs/architecture/in-memory-storage) and\ndoes not require the presence of a distributed file system nor\ninfrastructure like Zookeeper to provide high-availability and\nfault-tolerance.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"images/storage.png\"/\u003e\n\u003c/p\u003e\n\nJet implements a version of the\n[Chandy-Lamport](https://jet-start.sh/docs/architecture/fault-tolerance)\nalgorithm to provide _exactly-once_ processing under the face of\nfailures. When interfacing with external transactional systems like\ndatabases, it can provide end-to-end processing guarantees using\n[two-phase\ncommit](https://jet-start.sh/blog/2020/02/20/transactional-processors).\n\n### Advanced Event Processing\n\nEvent data can often arrive [out of\norder](https://jet-start.sh/docs/concepts/event-time) and Jet has\nfirst-class support for dealing with this disorder. Jet implements a\ntechnique called [_distributed\nwatermarks_](https://jet-start.sh/docs/architecture/event-time-processing)\nto treat disordered events as if they were arriving in order.\n\n\u003cimg src=\"images/watermarks.svg\"\u003e\n\n## How Do I Get Started\n\nFollow the [Get Started](https://jet-start.sh/docs/get-started/intro)\nguide to start using Jet.\n\n## Download\n\nYou can download Jet from\n[https://jet-start.sh](https://jet-start.sh/download).\n\nAlternatively, you can use the latest [docker\nimage](https://jet-start.sh/docs/operations/docker):\n\n```java\ndocker run -p 5701:5701 hazelcast/hazelcast-jet\n```\n\nUse the following Maven coordinates to add Jet to your application:\n\n```xml\n\u003cgroupId\u003ecom.hazelcast.jet\u003c/groupId\u003e\n\u003cartifactId\u003ehazelcast-jet\u003c/artifactId\u003e\n\u003cversion\u003e4.2\u003c/version\u003e\n```\n\n## Tutorials\n\nSee the [tutorials](https://jet-start.sh/docs/tutorials/kafka) for\ntutorials on using Jet. Some examples:\n\n* [Process Events from Apache\n  Kafka](https://jet-start.sh/docs/tutorials/kafka)\n* [Windowed Aggregation](https://jet-start.sh/docs/tutorials/windowing)\n* [Apply a Python Function](https://jet-start.sh/docs/tutorials/python)\n\n## Reference\n\nJet supports a variety of transforms and operators. These include:\n\n* [Stateless\n  transforms](https://jet-start.sh/docs/api/stateless-transforms) such\n  as mapping and filtering.\n* [Stateful\n  transforms](https://jet-start.sh/docs/api/stateful-transforms) such as\n  aggregations and stateful mapping.\n\n## Community\n\nHazelcast Jet team actively answers questions on [Stack\nOverflow](https://stackoverflow.com/tags/hazelcast-jet) and\n[Hazelcast Community Slack](https://slack.hazelcast.com).\n\nYou are also encouraged to join the [hazelcast-jet mailing\nlist](http://groups.google.com/group/hazelcast-jet) if you are\ninterested in community discussions\n\n## How Can I Contribute\n\nThanks for your interest in contributing! The easiest way is to just\nsend a pull request. Have a look at the issues marked as [good first\nissue](https://github.com/hazelcast/hazelcast-jet/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\nfor some guidance.\n\n### Building From Source\n\nTo build, use:\n\n```bash\n./mvnw clean package -DskipTests\n```\n\n### Use Latest Snapshot Release\n\nYou can always use the latest snapshot release if you want to try the\nfeatures currently under development.\n\nMaven snippet:\n\n```xml\n\u003crepositories\u003e\n    \u003crepository\u003e\n        \u003cid\u003esnapshot-repository\u003c/id\u003e\n        \u003cname\u003eMaven2 Snapshot Repository\u003c/name\u003e\n        \u003curl\u003ehttps://oss.sonatype.org/content/repositories/snapshots\u003c/url\u003e\n        \u003csnapshots\u003e\n            \u003cenabled\u003etrue\u003c/enabled\u003e\n            \u003cupdatePolicy\u003edaily\u003c/updatePolicy\u003e\n        \u003c/snapshots\u003e\n    \u003c/repository\u003e\n\u003c/repositories\u003e\n\u003cdependencies\u003e\n    \u003cdependency\u003e\n        \u003cgroupId\u003ecom.hazelcast.jet\u003c/groupId\u003e\n        \u003cartifactId\u003ehazelcast-jet\u003c/artifactId\u003e\n        \u003cversion\u003e4.3-SNAPSHOT\u003c/version\u003e\n    \u003c/dependency\u003e\n\u003c/dependencies\u003e\n```\n\n## Trigger Phrases in the Pull Request Conversation\n\nWhen you create a pull request (PR), it must pass a build-and-test\nprocedure. Maintainers will be notified about your PR, and they can\ntrigger the build using special comments. These are the phrases you may\nsee used in the comments on your PR:\n\n* `verify` - run the default PR builder, equivalent to `mvn clean\n  install`\n* `run-nightly-tests` - use the settings for the nightly build (`mvn\n  clean install -Pnightly`). This includes slower tests in the run,\n  which we don't normally run on every PR\n* `run-windows` - run the tests on a Windows machine (HighFive is not\n  supported here)\n* `run-cdc-debezium-tests` - run all tests in the\n  `extensions/cdc-debezium` module\n* `run-cdc-mysql-tests` - run all tests in the `extensions/cdc-mysql`\n  module\n* `run-cdc-postgres-tests` - run all tests in the\n  `extensions/cdc-postgres` module\n\nWhere not indicated, the builds run on a Linux machine with Oracle JDK\n8.\n\n## License\n\nSource code in this repository is covered by one of two licenses:\n\n 1. [Apache License 2.0](licenses/apache-v2-license.txt)\n 2. [Hazelcast Community\n    License](licenses/hazelcast-community-license.txt)\n\nThe default license throughout the repository is Apache License 2.0\nunless the\nheader specifies another license. Please see the [Licensing\nsection](https://jet-start.sh/license) for more information.\n\n## Credits\n\nWe owe (the good parts of) our CLI tool's user experience to\n[picocli](https://picocli.info/).\n\n## Copyright\n\nCopyright (c) 2008-2021, Hazelcast, Inc. All Rights Reserved.\n\nVisit [www.hazelcast.com](http://www.hazelcast.com/) for more info.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhazelcast%2Fhazelcast-jet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhazelcast%2Fhazelcast-jet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhazelcast%2Fhazelcast-jet/lists"}