In this paper, we present a size-based scheduling protocol for Hadoop, that caters to both interactivity and efficiency requirements raised by current Hadoop clusters. Our scheduler addresses several challenges such as job size estimation, resource management and scheduling complex jobs with inter-related phases. Furthermore, by employing the technique of job aging, we avoid the problem of job starvation typical of well known size-based policies. Our experiments pinpoint at a significant decrease in average job sojourn times -- a metric that accounts for the total time a job spends in the system, including waiting and serving times -- for realistic workloads generated according to production traces available in literature.

HFSP: Size-based Scheduling for Hadoop

CARRA, DAMIANO;
2013-01-01

Abstract

In this paper, we present a size-based scheduling protocol for Hadoop, that caters to both interactivity and efficiency requirements raised by current Hadoop clusters. Our scheduler addresses several challenges such as job size estimation, resource management and scheduling complex jobs with inter-related phases. Furthermore, by employing the technique of job aging, we avoid the problem of job starvation typical of well known size-based policies. Our experiments pinpoint at a significant decrease in average job sojourn times -- a metric that accounts for the total time a job spends in the system, including waiting and serving times -- for realistic workloads generated according to production traces available in literature.
2013
mapreduce; scheduling
File in questo prodotto:
File Dimensione Formato  
2013-c-BigData.pdf

non disponibili

Tipologia: Documento in Pre-print
Licenza: Dominio pubblico
Dimensione 253.55 kB
Formato Adobe PDF
253.55 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11562/618357
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 29
  • ???jsp.display-item.citation.isi??? 4
social impact