Objectives

The objectives of this project are twofold: (i) investigate run-time scaling in Apache Flink by varying the level of parallelism of a running program during its execution. Instead of stopping the execution to modify a program’s parameters, the runtime scaling mechanisms should allow increasing and decreasing the parallelism and resources of a running program on stream without stopping it, nor dropping any state. (ii) develop a mechanism for automatic migration of operators to improve utilization of available resources.

Expected results

  • Study and analysis of a run-time stream processing engine internal architecture, like Apache Flink, so to investigate the possibilities of implementing elasticity mechanisms and migration of operators.
  • Design and implement elasticity mechanisms and migration of operators in Apache Flink taking into account high (PaaS level) information like resources available, traffic and load.
  • Design of a use case or application suitable to test the implemented improvements in Apache Flink.
  • Exhaustive testing under real time conditions of load and traffic

 

Location

TU Berlin is an internationally renowned research-intensive university dedicated to promoting science and technology for the benefit of broader society and committed to the principles of sustainability and research for peaceful purposes. The DIMA group is the original author of Apache Flink.

Planned secondments

14 months at U-Hopper (Trento, Italy); 7 months at Las Naves (Valencia, Spain).

Conditions

Please write your application in English, as it will be examined by multiple people in different European countries. Please do not apply to more than two positions within the FogGuru project.

Apply Online

Fields with (*) are compulsory.