{"id":19910095,"url":"https://github.com/ververica/lab-flink-latency","last_synced_at":"2025-05-03T03:30:29.722Z","repository":{"id":37479004,"uuid":"394334126","full_name":"ververica/lab-flink-latency","owner":"ververica","description":"Lab for testing different Flink job latency optimization techniques covered in a Flink Forward 2021 talk","archived":false,"fork":false,"pushed_at":"2021-10-25T20:38:54.000Z","size":192,"stargazers_count":27,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-07T10:35:55.087Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ververica.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-09T15:01:16.000Z","updated_at":"2025-01-12T01:42:53.000Z","dependencies_parsed_at":"2022-09-12T09:32:07.522Z","dependency_job_id":null,"html_url":"https://github.com/ververica/lab-flink-latency","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ververica%2Flab-flink-latency","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ververica%2Flab-flink-latency/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ververica%2Flab-flink-latency/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ververica%2Flab-flink-latency/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ververica","download_url":"https://codeload.github.com/ververica/lab-flink-latency/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252137491,"owners_count":21700221,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T21:17:35.998Z","updated_at":"2025-05-03T03:30:29.384Z","avatar_url":"https://github.com/ververica.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"lab-flink-latency\n=================\n\nLab to showcase different Flink job latency optimization techniques covered in our Flink Forward 2021 talk\n[\"Getting into Low-Latency Gears with Apache Flink\"](https://www.flink-forward.org/global-2021/conference-program#getting-into-low-latency-gears-with-apache-flink).\n\nThis lab consists of several jobs which are described as follows.\n\n## IngestingJob\n\nThis job is used to ingest randomly generated sensor measurements into a Kafka topic. Use `--kafka` to specify the\nKafka bootstrap servers. This defaults to `localhost:9092`. Use `--topic` to specify the name of the Kakfa topic to\ningest into. This default is `lablatency`. You can also use\n`--wait-micro \u003cnumber of micro seconds\u003e` to adjust the ingestion rate.\n\n## WindowingJob\n\nThis job calculates the number of measurements and the sum of the measurement values per minute (window size), and updates the\nresult every 10 seconds (slide size). The latency of this job can be optimized by using the following techniques.\n\n### Optimization 1\nIncrease the job parallelism, e.g., from 2 to 3. Best to have the number of the partitions of your Kafka topic\ndivisible by 2 and by 3 to avoid data skew.\n\n### Optimization 2\nUse the hashmap/filesystem state backend by changing the configuration from\n\n    state.backend: rocksdb\n    # 0.4 is Flink's default\n    taskmanager.memory.managed.fraction: '0.4'\n\nto\n\n    # use filesystem if Flink \u003c 1.13\n    state.backend: hashmap\n    taskmanager.memory.managed.fraction: '0.0'\n\n### Optimization 3\nReduce the watermark interval from the default `200 ms` to `100 ms`:\n\n    pipeline.auto-watermark-interval: 100 ms\n\n### Optimization 4\nReduce the network buffer timeout from the default `100 ms` to `10 ms`:\n\n    execution.buffer-timeout: 10 ms\n\n## WindowingJobNoAggregation\n\nSimilar to WindowingJob, except that there is no incremental aggregation during windowing in this job.\n\n## EnrichingJobSync\n\nThis job enriches measurements with the location information retrieved from a simulated external service which has a\nrandom latency in the range of 1-6 ms. When location information is retrieved, the job caches it for 1 second to serve further\nretrieving requests.\n\n## EnrichingJobAsync\n\nSimilar to `EnrichingJobSync`, except that this job uses\n[Flink's Async I/O](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/operators/asyncio/)\nto get better performance.\n\n## SortingJobPerEventTimer\n\nThis job sorts a stream of measurements keyed by sensor IDs, then calculates an\n[exponential moving average](https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average ) for each\nsensor. When sorting, it creates a timer per event.\n\n## SortingJobCoalescedTimer\n\nSimilar to `SortingJobPerEventTimer`, except that when sorting, it coalesces timers to the next 100ms (configurable\nvia `--round-timer-to`) or to the next watermark if `--round-timer-to` is set to `0`.\n\nThis job can be run with the follow options/configurations to manage the per-event overhead.\n\n### User Code\n\nCreate only one ObjectMapper per operator instance (default)\n\n    --use-one-mapper true\n\nCreate one ObjectMapper per event\n\n    --use-one-mapper false\n\n### Serialization\n\nUse the POJO serializer  (default)\n\n    --force-kryo false\n\nForce using the Kryo serializer\n\n    --force-kryo true\n\n### Object Reuse\n\nDisable object reuse with the following configuration (default)\n\n    pipeline.object-reuse: false\n\nEnable object reuse with the following configuration\n\n    pipeline.object-reuse: true\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fververica%2Flab-flink-latency","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fververica%2Flab-flink-latency","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fververica%2Flab-flink-latency/lists"}