https://github.com/decodableco/examples
🌟 Examples of use cases that utilize Decodable, as well as demos for related open-source projects such as Apache Flink, Debezium, and Postgres.
https://github.com/decodableco/examples
datamesh dataproducts etl flink flink-stream-processing kafka pyflink python stream-processing
Last synced: 2 months ago
JSON representation
🌟 Examples of use cases that utilize Decodable, as well as demos for related open-source projects such as Apache Flink, Debezium, and Postgres.
- Host: GitHub
- URL: https://github.com/decodableco/examples
- Owner: decodableco
- License: apache-2.0
- Created: 2022-04-29T16:18:39.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2025-06-20T11:16:02.000Z (12 months ago)
- Last Synced: 2025-06-20T12:29:07.828Z (12 months ago)
- Topics: datamesh, dataproducts, etl, flink, flink-stream-processing, kafka, pyflink, python, stream-processing
- Language: Java
- Homepage: https://decodable.co
- Size: 19.9 MB
- Stars: 76
- Watchers: 8
- Forks: 34
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Changelog: change-streams/.gitignore
- License: LICENSE
Awesome Lists containing this project
README
# Decodable Examples
## Introduction
This repository contains examples of use cases that utilize Decodable streaming solution as well as demos for related open-source projects such as Apache Flink, Debezium, and Postgres.
Examples are presented "as-is" and are maintained on a best effort basis. PRs for updating existing (or adding new) examples are welcome!
For help with any of the examples, or using Decodable in general, please [join our Slack group](https://join.slack.com/t/decodablecommunity/shared_invite/zt-uvow71bk-Uf914umgpoyIbOQSxriJkA).
## About Decodable
_Decodable radically simplifies real-time data, making it easier to access the freshest, high-quality data. Reduce infrastructure overhead, connect data sources, transform, and deliver data reliably to any destination._
_Learn more [here](https://decodable.co), and [sign up for a free trial](https://app.decodable.co/-/accounts/create) today!_
## Contents
### Stream Processing Techniques
| Example | Description |
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------|
| [Change Streams](change-streams) | Using change streams to build materialized views in Postgres |
| [XML Processing](xml) | Parse XML and transform to JSON |
| [Masking](masking) | Ways to mask data |
| [Array Aggregation (1)](array-agg-postgres) | Demonstrating how to aggregate the elements of the many side of 1:n join into an array with data from Postgres |
| [Array Aggregation (2)](array-agg) | Using the `array_agg()` UDF for denormalizing data in a pipeline from MySQL to OpenSearch |
### Data Pipelines
| Example | Description |
|---------------------------------------------------------|----------------------------------------------------------------------------------------|
| [Opinionated Data Pipelines](opinionated-pipelines) | Building data pipelines with schema on write streams. |
| [Postman](postman) | Building data pipelines with Postman. |
| [Postgres to Snowflake](postgres-to-snowflake-with-cdc) | Getting data from Postgres to Snowflake using Decodable. |
| [Flink CDC](flink-cdc) | Trying out Flink CDC and comparing it to Flink SQL. |
| [Flink SQL and Custom Pipelines](sql-cupi-hybrid) | Bridging Flink SQL and Custom Java Pipelines with the Decodable SDK. |
| [Testing Custom Pipelines on Decodable](testing-cupi) | How to write more modular Flink jobs with pluggable components to improve testability. |
### PyFlink
_Decodable provides a managed PyFlink service. Learn more [here](https://docs.decodable.co/pipelines/create-pipelines-using-your-own-apache-flink-jobs.html#_create_a_custom_pipeline_python)._
| Example | Description |
|-----------------------------------------------------------------|-----------------------------------------------------------------------------------------|
| [PyFlink](pyflink) | Running a basic PyFlink job on Kubernetes |
| [PyFlink on Decodable](pyflink-decodable) | Running a PyFlink job as a Custom Pipeline on Decodable |
| [PyFlink and MongoDB Vector Search](pyflink-vector-embeddings) | End-to-end example for PyFlink Vector Ingestion on Decodable with MongoDB Vector Search |
| [PyFlink Intro](pyflink-intro) | A Hands-On Introduction to PyFlink |
### Integrations
| Example | Description |
|----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| [Apache Druid](druid) | Sending COVID-19 data to Decodable using its REST API, cleaning it with SQL, and then sending it to Apache Druid |
| [Apache Kafka / Flink / Iceberg](kafka-iceberg/apache-flink) | Integrating Apache Kafka with Apache Iceberg through Apache Flink. _As presented at Kafka Summit London 2024_ |
| [Apache Kafka / Flink / Iceberg](kafka-iceberg/decodable) (with Decodable) | Streaming from Apache Kafka to Apache Iceberg with Decodable |
| [Apache Kafka Upsert connector](kafka-upsert/) | Explaining the difference between the Flink Kafka and Kafka Upsert connectors |
| [Apache Kafka mTLS](mtls) | Installing Apache Kafka on EC2 and configuring it with mTLS |
| [Apache Kafka with ngrok](kafka-ngrok) | Using Docker Compose for running Apache Kafka locally, accessible from the internet using ngrok |
| [Apache Kafka](kafka2s3) | Installing Apache Kafka on EC2 and writing to S3 with Decodable |
| [Apache Pinot](pinot) | Transforming osquery logs to Apache Pinot and Superset |
| [AsyncAPI](asyncapi) | Publishing Data Products with AsyncAPI |
| [Confluent](confluent) | Clickstream from Confluent Cloud joined with CDC user data from Postgres |
| [Delta Lake / Flink](flink-delta-lake) | Writing to Delta Lake with Apache Flink |
| [GitHub Webhooks](github-webhooks) | Processing GitHub Webhook events using the Decodable REST source connector |
| [OSQuery Routing](osquery) | Routing OSQuery logs with SQL |
| [Redpanda](redpanda) | Reading and writing data to Redpanda from Flink |
| [S3 Events in a Lambda Function](s3events/) | Configuring an S3 bucket with a Lambda notification to send data to Kinesis to be processed in Decodable |
| [Tinybird](tinybird) | Writing data to Tinybird from Decodable |
### Changed Data Capture (CDC)
| Example | Description |
|------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
| [MSSQL CDC](mssql_cdc/) | Enabling MSSQL in Docker with CDC, reading from it with Debezium, writing change events into AWS Kinesis |
| [Oracle CDC](oracle_cdc/) | Configuring Oracle AWS RDS with LogMiner, reading from it with Debezium, writing change events into AWS Kinesis |
| [DynamoDb CDC](dynamodb_cdc/) | Configure DynamoDB to send change data to Kinesis, reading changes into Decodable for transformation or replication. |
| [Logical Decoding Message Examples](postgres-logical-decoding) | How to retrieve logical decoding messages from the Postgres WAL |
| [Logical Replication on Postgres 16 Stand-By Servers](postgres-logical-replication-standby) | How to use logical replication on Postgres 16 stand-by servers |
| [Postgres 17 Fail-Over Slots](failover-slots) | How to use fail-over slots with Postgres 17 |
| [Transactional CDC Event Aggregation](tx-aware-cdc-buffering) | Aggregating Change Data Capture Events based on Transactional Boundaries |
### Flink SQL
| Example | Description |
|-------------------------------------------------------|-------------|
| [Flink SQL Troubleshooting](troubleshooting-flinksql) | A set of Docker Compose environments for demonstrating various Flink SQL troubleshooting scenarios (see [related blog](https://www.decodable.co/blog/flink-sql-misconfiguration-misunderstanding-and-mishaps?utm_medium=github&utm_source=examples_repo&utm_campaign=blog&utm_content=troubleshooting-flinksql))|
### Decodable tools
| Example | Description |
|-------------------------------------------------------|-------------|
| [Decodable CI/CD](declarative-cicd) | An example of using Decodable with GitHub Actions|
| [Decodable CLI Docker image](cli-docker) | An example Dockerfile for running the Decodable CLI under Docker.|
### Kubernetes
| Example | Description |
|-------------------------------------------------------|-------------|
| [Fink on Kubernetes](flink-on-kubernetes) | An example of running Flink via the Flink Kubernetes Operator|
## License
This code base is available under the Apache License, version 2.