{"id":18802160,"url":"https://github.com/oracle-quickstart/oke-flink","last_synced_at":"2026-01-05T01:30:17.898Z","repository":{"id":92409638,"uuid":"353142444","full_name":"oracle-quickstart/oke-flink","owner":"oracle-quickstart","description":"Apache Flink on OKE","archived":false,"fork":false,"pushed_at":"2023-10-30T17:24:27.000Z","size":85,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-29T20:15:49.013Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"upl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oracle-quickstart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-30T21:11:03.000Z","updated_at":"2024-05-16T15:25:37.000Z","dependencies_parsed_at":"2024-11-07T22:30:51.321Z","dependency_job_id":"4f9c47c8-e14a-45e7-9abd-ed511d247588","html_url":"https://github.com/oracle-quickstart/oke-flink","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foke-flink","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foke-flink/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foke-flink/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oracle-quickstart%2Foke-flink/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oracle-quickstart","download_url":"https://codeload.github.com/oracle-quickstart/oke-flink/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239735263,"owners_count":19688262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T22:26:48.499Z","updated_at":"2026-01-05T01:30:17.851Z","avatar_url":"https://github.com/oracle-quickstart.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"![tests](https://github.com/oracle-quickstart/oke-flink/actions/workflows/tests.yaml/badge.svg)\n\n# oke-flink\n\nDeploy a Kubernetes cluster on Oracle Cloud Infrastructure with multiple node pools and add-ons like Apache Flink.\n\n[![Deploy to Oracle Cloud][magic_button]][magic_oke_flink_stack]\n\n## Template Features\n\nThe Flink operator deployment is performed by the terraform helm provider, on a cluster templates that can also be used for other purposes.\n\nThe OKE cluster template features the following:\n\n- Up to 3 node pools. This allow for usage of different shapes within the same cluster (for example, CPU and GPU, or DenseIO shapes)\n- Cluster node-pool auto-scaler, from 0 nodes (shut down) and up, allowing to only use more expensive shapes when needed (i.e. GPU)\n- Option to use Secrets encryption.\n- Option to enable Image Validation and Pod Admission Controllers.\n- Option to install metrics server (required by cluster auto-scaler)\n- Option to install cert-manager (required by Flink Operator)\n- Option to install a monitoring stack based on Prometheus and Grafana\n\n## Getting started with Apache Flink Operator\n\nThe Operator is deployed in the cluster, and offers two modes of operation:\n\n### Application Mode\n\nIn this mode flink job are deployed independently, creating a Job Manager and Task Manager(s) for each Job. It is the preferred way of using the operator if the Job code is packaged in a Docker image. In this mode, since a Job Manager is deployed per job, the UI access is also per job and one needs to connect to the specfic Job Manager REST service.\n\n### Session Mode\n\nIn session mode, a FlinkDeployment creates a JobManager (or more if High Availability is configured), and then FlinkSessionJob can be created that uses that Job Manager. The advantage is that the UI is accessible in one place, however Job code needs to be staged on a remote location (like an Object Storage bucket). On OCI, the code can be stored on storage bucket and made accessible through a Pre-authenticated Request, or S3 compatibility file system needs to have been configured as a plugin to reference the jar (the difference between the two is in the way to control access)\n\nThe demo job referenced in the docs uses Application Mode.\n\nTo start a Flink Session Cluster, create a FlinkDeployment without a job field.\n\nFor example:\n\n```yaml\napiVersion: flink.apache.org/v1beta1\nkind: FlinkDeployment\nmetadata:\n  # the name of the session cluster, to be referenced in the FlinkSessionJob\n  name: flink-session\nspec:\n  image: \u003cflink base image, or your own image with the proper plugins enables, like flink-s3-fs-hadoop\u003e\n  flinkVersion: v1_16\n  flinkConfiguration:\n    taskmanager.numberOfTaskSlots: \"2\"\n    ## S3 compatible mode Object Storage access\n    # s3.endpoint: \u003ctenancy_namespace\u003e.compat.objectstorage.us-ashburn-1.oraclecloud.com\n    # s3.endpoint.region: us-ashburn-1\n    # s3.path.style.access: \"true\"\n    ## state backend for savepoints\n    # state.backend: rocksdb\n    # state.backend.incremental: \"true\"\n    kubernetes.jobmanager.replicas: \"3\" # 3 for High Availability\n    kubernetes.operator.periodic.savepoint.interval: 1h\n    kubernetes.operator.savepoint.history.max.age: 24h\n    kubernetes.operator.savepoint.history.max.count: \"25\"\n    ## State storage location\n    # state.checkpoints.dir: s3://\u003cstate_storage_bucket\u003e/\u003cpath\u003e\n    # state.savepoints.dir: s3://\u003cstate_storage_bucket\u003e/\u003cpath\u003e\n    # high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory\n    # high-availability.storageDir: s3://\u003cstate_storage_bucket\u003e/ha\n    rest.flamegraph.enabled: \"true\"\n    restart-strategy: exponential-delay\n    metrics.reporters: prom\n    metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory\n  serviceAccount: flink\n  podTemplate:\n    apiVersion: v1\n    kind: Pod\n    metadata:\n      name: flink-session\n    spec:\n      serviceAccount: flink\n    ## When using your own image, create a secret for OCI Docker Registry, and add it here\n    #   imagePullSecrets:\n    #   - name: oci-registry\n      containers:\n        # Do not change the main container name\n        - name: flink-main-container\n          imagePullPolicy: IfNotPresent\n        ## You can pass global env for jobs here\n        #   env:\n        #   - name: KAFKA_PASSWORD\n        #     valueFrom:\n        #       secretKeyRef:\n        #         name: flink-kafka\n        #         key: KAFKA_PASSWORD\n        #   - name: AWS_ACCESS_KEY_ID\n        #     valueFrom:\n        #       secretKeyRef:\n        #         name: aws-s3\n        #         key: AWS_ACCESS_KEY_ID\n        #   - name: AWS_SECRET_ACCESS_KEY\n        #     valueFrom:\n        #       secretKeyRef:\n        #         name: aws-s3\n        #         key: AWS_SECRET_ACCESS_KEY\n        #   envFrom:\n        #   - configMapRef:\n        #       name: flink-kafka\n  jobManager:\n    replicas: 3\n    resource:\n      memory: \"2048m\"\n      cpu: 1\n  taskManager:\n    resource:\n      memory: \"2048m\"\n      cpu: 1\n```\n\nThen a Session Job can be launched with:\n\n```yaml\napiVersion: flink.apache.org/v1beta1\nkind: FlinkSessionJob\nmetadata:\n  name: flink-journal\n  namespace: flink\nspec:\n  deploymentName: flink-session # \u003c-- the name of the Session deployment\n  job:\n    jarURI: # need to use s3:// scheme or https:// and put jar in Object Storage\n    parallelism: 2\n    ## optional entry class name (full qualified name)\n    # entryClass: com.example.MyEntryClass\n    ## Command line arguments to pass to the job\n    args: []\n    upgradeMode: stateless # Use savepoint if state management is configuered. `last-state` is not supported.\n```\n\n## Send Flink metrics to Prometheus\n\nTo send Flink metrics to Prometheus, some specific configuration is needed in the Flink deployment.\n\nMake sure to add the following to you Flink Session or Application deployment:\n\n```yaml\nspec:\n  flinkConfiguration:\n    metrics.reporters: prom\n    metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory\n```\n\n## Use the Terraform template\n\nTo use the Terraform template locally, configure the OCI Command Line Interface with a Private/Public key pair added to your user.\n\nCreate a `terraform.tvfars` from the `terraform.tvfars.template` file and fill in the values for the variables.\n\nRun:\n\n```bash\n# init the repo\nterraform init\n# check the plan\nterraform plan\n# deploy\nterraform apply\n```\n\n## References\n\n- [Apache Flink operator documentation](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/)\n\n[magic_button]: https://oci-resourcemanager-plugin.plugins.oci.oraclecloud.com/latest/deploy-to-oracle-cloud.svg\n[magic_oke_flink_stack]: https://cloud.oracle.com/resourcemanager/stacks/create?zipUrl=https://github.com/oracle-quickstart/oke-flink/releases/latest/download/oke-flink.zip\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle-quickstart%2Foke-flink","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foracle-quickstart%2Foke-flink","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foracle-quickstart%2Foke-flink/lists"}