{"id":13425743,"url":"https://github.com/confluentinc/ksql","last_synced_at":"2025-05-14T14:07:29.655Z","repository":{"id":37251774,"uuid":"69400326","full_name":"confluentinc/ksql","owner":"confluentinc","description":"The database purpose-built for stream processing applications.","archived":false,"fork":false,"pushed_at":"2025-05-07T07:23:00.000Z","size":215969,"stargazers_count":194,"open_issues_count":1310,"forks_count":1043,"subscribers_count":198,"default_branch":"master","last_synced_at":"2025-05-07T07:41:10.801Z","etag":null,"topics":["event-streaming-database","interactive","kafka","kafka-connect","ksqldb","ksqldb-documentation","ksqldb-tutorials","materialized-views","real-time","sql","stream-processing","streaming-queries"],"latest_commit_sha":null,"homepage":"https://ksqldb.io","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/confluentinc.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-09-27T21:30:15.000Z","updated_at":"2025-05-06T08:26:58.000Z","dependencies_parsed_at":"2023-10-23T20:35:43.853Z","dependency_job_id":"43d9a7ef-e8db-411b-9b5a-e122d85bda42","html_url":"https://github.com/confluentinc/ksql","commit_stats":null,"previous_names":[],"tags_count":20829,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fksql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fksql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fksql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/confluentinc%2Fksql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/confluentinc","download_url":"https://codeload.github.com/confluentinc/ksql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254159194,"owners_count":22024558,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["event-streaming-database","interactive","kafka","kafka-connect","ksqldb","ksqldb-documentation","ksqldb-tutorials","materialized-views","real-time","sql","stream-processing","streaming-queries"],"created_at":"2024-07-31T00:01:17.852Z","updated_at":"2025-05-14T14:07:24.644Z","avatar_url":"https://github.com/confluentinc.png","language":"Java","readme":"# ![KSQL rocket](ksql-rocket.png) ksqlDB\n\n### The database purpose-built for stream processing applications\n\n# Overview\n\nksqlDB is a database for building stream processing applications on top of Apache Kafka. It is **distributed**, **scalable**, **reliable**, and **real-time**. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax. ksqlDB offers these core primitives:\n\n* **[Streams](https://docs.ksqldb.io/en/latest/concepts/collections/streams/) and [tables](https://docs.ksqldb.io/en/latest/concepts/collections/tables/)** - Create relations with schemas over your Apache Kafka topic data\n* **[Materialized views](https://docs.ksqldb.io/en/latest/concepts/materialized-views/)** - Define real-time, incrementally updated materialized views over streams using SQL\n* **[Push queries](https://docs.ksqldb.io/en/latest/concepts/queries/push/)** - Continuous queries that push incremental results to clients in real time\n* **[Pull queries](https://docs.ksqldb.io/en/latest/concepts/queries/pull/)** - Query materialized views on demand, much like with a traditional database\n* **[Connect](https://docs.ksqldb.io/en/latest/concepts/connectors)** - Integrate with any [Kafka Connect](https://docs.confluent.io/current/connect/index.html) data source or sink, entirely from within ksqlDB\n\nComposing these powerful primitives enables you to build a complete streaming app with just SQL statements, minimizing complexity and operational overhead. ksqlDB supports a wide range of operations including aggregations, joins, windowing, sessionization, and much more. You can find more ksqlDB tutorials and resources [here](https://developer.confluent.io/tutorials/use-cases.html).\n\n# Getting Started\n\n* Follow the [ksqlDB quickstart](https://ksqldb.io/quickstart.html) to get started in just a few minutes.\n* Read through the [ksqlDB documentation](https://docs.ksqldb.io).\n* Take a look at some [ksqlDB use case recipes](https://developer.confluent.io/tutorials/use-cases.html) for examples of common patterns.\n\n# Documentation\n\nSee the [ksqlDB documentation](https://docs.ksqldb.io/) for the latest stable release.\n\n# Use Cases and Examples\n\n## Materialized views\n\nksqlDB allows you to define materialized views over your streams and tables. Materialized views are defined by what is known as a \"persistent query\". These queries are known as persistent because they maintain their incrementally updated results using a table.\n\n```sql\nCREATE TABLE hourly_metrics AS\n  SELECT url, COUNT(*)\n  FROM page_views\n  WINDOW TUMBLING (SIZE 1 HOUR)\n  GROUP BY url EMIT CHANGES;\n\n```\n\nResults may be **\"pulled\"** from materialized views on demand via `SELECT` queries. The following query will return a single row:\n\n```sql\nSELECT * FROM hourly_metrics\n  WHERE url = 'http://myurl.com' AND WINDOWSTART = '2019-11-20T19:00';\n```\n\nResults may also be continuously **\"pushed\"** to clients via streaming `SELECT` queries. The following streaming query will push to the client all incremental changes made to the materialized view:\n\n```sql\nSELECT * FROM hourly_metrics EMIT CHANGES;\n```\n\nStreaming queries will run perpetually until they are explicitly terminated.\n\n## Streaming ETL\n\nApache Kafka is a popular choice for powering data pipelines. ksqlDB makes it simple to transform data within the pipeline, readying messages to cleanly land in another system.\n\n```sql\nCREATE STREAM vip_actions AS\n  SELECT userid, page, action\n  FROM clickstream c\n  LEFT JOIN users u ON c.userid = u.user_id\n  WHERE u.level = 'Platinum' EMIT CHANGES;\n```\n\n## Anomaly Detection\n\nksqlDB is a good fit for identifying patterns or anomalies on real-time data. By processing the stream as data arrives you can identify and properly surface out of the ordinary events with millisecond latency.\n\n```sql\nCREATE TABLE possible_fraud AS\n  SELECT card_number, count(*)\n  FROM authorization_attempts\n  WINDOW TUMBLING (SIZE 5 SECONDS)\n  GROUP BY card_number\n  HAVING count(*) \u003e 3 EMIT CHANGES;\n```\n\n## Monitoring\n\nKafka's ability to provide scalable ordered records with stream processing make it a common solution for log data monitoring and alerting. ksqlDB lends a familiar syntax for tracking, understanding, and managing alerts.\n\n```sql\nCREATE TABLE error_counts AS\n  SELECT error_code, count(*)\n  FROM monitoring_stream\n  WINDOW TUMBLING (SIZE 1 MINUTE)\n  WHERE  type = 'ERROR'\n  GROUP BY error_code EMIT CHANGES;\n```\n\n## Integration with External Data Sources and Sinks\n\nksqlDB includes native integration with [Kafka Connect](https://docs.ksqldb.io/en/latest/concepts/connectors) data sources and sinks, effectively providing a unified SQL interface over a [broad variety of external systems](https://www.confluent.io/hub).\n\nThe following query is a simple persistent streaming query that will produce all of its output into a topic named `clicks_transformed`:\n\n```sql\nCREATE STREAM clicks_transformed AS\n  SELECT userid, page, action\n  FROM clickstream c\n  LEFT JOIN users u ON c.userid = u.user_id EMIT CHANGES;\n```\n\nRather than simply send all continuous query output into a Kafka topic, it is often very useful to route the output into another datastore. ksqlDB's Kafka Connect integration makes this pattern very easy.\n\nThe following statement will create a Kafka Connect sink connector that continuously sends all output from the above streaming ETL query directly into Elasticsearch:\n\n```sql\n CREATE SINK CONNECTOR es_sink WITH (\n  'connector.class' = 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector',\n  'key.converter'   = 'org.apache.kafka.connect.storage.StringConverter',\n  'topics'          = 'clicks_transformed',\n  'key.ignore'      = 'true',\n  'schema.ignore'   = 'true',\n  'type.name'       = '',\n  'connection.url'  = 'http://elasticsearch:9200');\n```\n\n\u003ca name=\"community\"\u003e\u003c/a\u003e\n# Join the Community\n\nFor user help, questions or queries about ksqlDB please use our [user Google Group](https://groups.google.com/forum/#!forum/ksql-users)\nor our public Slack channel #ksqldb in [Confluent Community Slack](https://slackpass.io/confluentcommunity). Everyone is welcome!\n\nYou can get help, learn how to contribute to ksqlDB, and find the latest news by [connecting with the Confluent community](https://www.confluent.io/contact-us-thank-you/).\n\nFor more general questions about the Confluent Platform please post in the [Confluent Google group](https://groups.google.com/forum/#!forum/confluent-platform).\n\n\n# Contributing and building from source\n\nContributions to the code, examples, documentation, etc. are very much appreciated.\n\n- Report issues and bugs directly in [this GitHub project](https://github.com/confluentinc/ksql/issues).\n- Learn how to work with the ksqlDB source code, including building and testing ksqlDB as well as contributing code changes\n  to ksqlDB by reading our [Development and Contribution guidelines](CONTRIBUTING.md).\n- One good way to get started is by tackling a [newbie issue](https://github.com/confluentinc/ksql/labels/good%20first%20issue).\n\n\n# License\n\nThe project is licensed under the [Confluent Community License](LICENSE).\n\n*Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the [Apache Software Foundation](https://www.apache.org/).*\n","funding_links":[],"categories":["Java","Table of Contents","Java (504)","_Table of Contents_","数据库"],"sub_categories":["Streaming SQL","On-Prem"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconfluentinc%2Fksql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconfluentinc%2Fksql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconfluentinc%2Fksql/lists"}