HFSP: Size-based Scheduling for Hadoop

In this paper, we present a size-based scheduling protocol for Hadoop, that caters to both interactivity and efficiency requirements raised by current Hadoop clusters. Our scheduler addresses several challenges such as job size estimation, resource management and scheduling complex jobs with inter-related phases. Furthermore, by employing the technique of job aging, we avoid the problem of job starvation typical of well known size-based policies. Our experiments pinpoint at a significant decrease in average job sojourn times -- a metric that accounts for the total time a job spends in the system, including waiting and serving times -- for realistic workloads generated according to production traces available in literature.