{"id":21514900,"url":"https://github.com/getindata/flink-spring","last_synced_at":"2025-04-24T03:09:24.861Z","repository":{"id":158060536,"uuid":"633796849","full_name":"getindata/flink-spring","owner":"getindata","description":"A library that allows using Spring dependency injection framework in Flink Jobs","archived":false,"fork":false,"pushed_at":"2023-05-08T11:14:57.000Z","size":13,"stargazers_count":13,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-24T03:09:18.092Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/getindata.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-28T09:49:49.000Z","updated_at":"2025-02-28T06:54:11.000Z","dependencies_parsed_at":null,"dependency_job_id":"68d248fd-b31c-44b2-9b46-1b26c239a8e1","html_url":"https://github.com/getindata/flink-spring","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-spring","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-spring/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-spring/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/getindata%2Fflink-spring/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/getindata","download_url":"https://codeload.github.com/getindata/flink-spring/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250552074,"owners_count":21449165,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-23T23:53:27.283Z","updated_at":"2025-04-24T03:09:24.853Z","avatar_url":"https://github.com/getindata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# flink-spring\nWith this library you can build Flink jobs using Spring dependency injection framework.\nFlink jobs can be build and set up using well known Spring mechanisms for dependency injection\nmaking the implementation more clean, efficient and portable.\n\nThe goal of this library is **NOT** to run entire Flink job within Spring context.\nInstead, we provide you with a helper classes that can be used in your Flink job to create Spring context\nbased on your Spring configuration classes and use this context to set up your pipeline. This can be done both on\nJob Manager while processing `main` method (job submission phase) and also on Task Managers, for example in\n`RichFunction::open` method. The created Spring context is short-lived.\n\nThis library provides all core Spring dependencies like:\n- spring-context\n- spring-beans\n- spring-core\n- spring-expression\n- spring-aop\n\nIt is based on Spring version **5.3.27,** and it is compiled using Java **11**.\n\n# How it works\nThe `flink-spring` library apart from providing Spring dependencies also provides a utility/registry `ContextRegistry` class.\nThis class has an API that allows you to load Spring context.\n \nAPI Usage:\n```java\nDataStreamJob dataSteamJob = new DataStreamJob();\ndataSteamJob = new ContextRegistry().autowiredBean(dataSteamJob, \"org.example.config\");\n```\n\nBy calling `new ContextRegistry().autowiredBean(new DataStreamJob(), \"org.example.config\")` two things have happened:\n1. The Spring context was created based on Spring configuration classes from `org.example.config` package.\n2. All fields marked as `@Autowired` in `DataStreamJob` instance were injected by Spring. \n\nAdditionally, the created Spring context was added to `ContextRegistry` instance scope registry.\nthanks to this we can avoid recreating the context for every `.autowiredBean(...)` call.\n\n# Usage in your code\n1. Clone the repository and build it using\n   ```shell\n   mvn clean install\n   ```\n\n2. Copy created artifact `target/flink-spring-0.1.0-SNAPSHOT-jar-with-dependencies.jar` to `lib` folder\n   of your Flink's distribution. Restart the cluster.\n\n3. In your Flink job `pom.xml` add:\n   ```xml\n   \u003cdependency\u003e\n      \u003cgroupId\u003ecom.getindata\u003c/groupId\u003e\n      \u003cartifactId\u003eflink-spring\u003c/artifactId\u003e\n      \u003cversion\u003e0.1.0-SNAPSHOT\u003c/version\u003e\n      \u003cscope\u003eprovided\u003c/scope\u003e\n   \u003c/dependency\u003e\n   ```\n   Mind that the `scope` is set to `provided`. We don't want to include `flink-spring` lib in our job's uber jar.\n\n# Example\nIn this example:\n- sink is injected by Spring\n- Source is not injected by Spring (it could) but instead we are passing a `EventProducer\u003cOrder\u003e` object\nto its constructor.\n\nWhat we want to show here is that both, Flink components (Sources, Sinks etc.) as well as business code (`EventProducer\u003cOrder\u003e`)\ncan be injected by this library.\n\nFor now, more detailed example can be found [here](https://github.com/kristoffSC/flink-using-springDI).\n\n#### The main class\nThis class will have all its dependencies marked as `@Autowired` injected by Spring based on\nconfiguration classes located in `org.example.config` package.\n\n```java\npackage org.example;\n\nimport com.getindata.fink.spring.context.ContextRegistry;\nimport org.springframework.beans.factory.annotation.Autowired;\n/* other imports omitted for clarity. */\n\npublic class DataStreamJob {\n\n    // Will be injected by Spring based on Spring context configuration.\n\t@Autowired\n\tprivate EventProducer\u003cOrder\u003e eventProducer;\n\n    // Will be injected by Spring based on Spring context configuration.\n\t@Autowired\n\tprivate SinkFunction\u003cSessionizeOrder\u003e sink;\n\n\tpublic static void main(String[] args) throws Exception {\n        // Using flink-spring library to inject DataStreamJob.class dependencies that are marked as\n        // @Autowired. \n\t\tnew ContextRegistry()\n\t\t\t.autowiredBean(new DataStreamJob(), \"org.example.config\")\n\t\t\t.run(args);\n\t}\n\n\tprivate void run(String[] args) throws Exception {\n\t\tStreamExecutionEnvironment env = createStreamEnv();\n\t\tenv.addSource(new CheckpointCountingSource\u003c\u003e(5, 5, eventProducer))\n\t\t\t.setParallelism(1)\n\t\t\t.process(new FlinkBusinessLogic())\n\t\t\t.setParallelism(2)\n\t\t\t.addSink(sink) // sink will be injected by Spring\n\t\t\t.setParallelism(2);\n\n\t\tenv.execute(\"Flink Job  Powered By Spring DI.\");\n\t}\n\n\tprivate static StreamExecutionEnvironment createStreamEnv() {\n\t\tStreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();\n\t\tenv.getConfig().setRestartStrategy(RestartStrategies.noRestart());\n\t\tenv.setRuntimeMode(RuntimeExecutionMode.STREAMING);\n\t\tenv.enableCheckpointing(3000, CheckpointingMode.EXACTLY_ONCE);\n\t\treturn env;\n\t}\n}\n```\n\n#### Configuration classes\nThis is a Spring configuration class that can be used for loading Spring context by `flink-spring` library.\n\n```java\npackage org.example.config;\n\n// Spring libraries comes from flink-spring library\nimport org.springframework.context.annotation.Bean;\nimport org.springframework.context.annotation.Configuration;\n/* other imports omitted for clarity. */\n\n@Configuration\npublic class JobSpringConfig {\n\n    @Bean\n    public EventToStringConverter\u003cSessionizeOrder\u003e converter() {\n        return event -\u003e String.format(\"Order Details - %s\", event.toString());\n    }\n\n    @Bean\n    public SinkFunction\u003cSessionizeOrder\u003e sink(EventToStringConverter\u003cSessionizeOrder\u003e converter) {\n        return new ConsoleSink\u003c\u003e(converter);\n    }\n\n    @Bean\n    public EventProducer\u003cOrder\u003e eventProducer() {\n        return new OrderProducer();\n    }\n\n    @Bean\n    public SessionManager sessionManager() {\n        return new SimpleSessionManager();\n    }\n\n    @Bean\n    public OrderProcessor\u003cSessionizeOrder\u003e orderProcessor(SessionManager sessionManager) {\n        return new BusinessOrderProcessor(\n            List.of(new SideNameAnonymization()),\n            new OrderSessionize(sessionManager)\n        );\n    }\n}\n\n```\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fflink-spring","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgetindata%2Fflink-spring","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgetindata%2Fflink-spring/lists"}