As more ICU EHR data becomes available, the interest in developing clinical prediction models to improve healthcare protocols increases. However, insufficient data quality still hinders clinical prediction using Machine Learning (ML). Many vital sign measurements, such as heart rate, contain sizeable missing segments, leaving gaps in the data that could negatively impact prediction performance. Previous works have introduced numerous time-series imputation techniques. Nevertheless, more comprehensive work is needed to compare a representative set of imputation methods for imputing ICU vital signs to determine the best practice. In reality, simple and ad-hoc imputation techniques that could decrease prediction accuracy, like zero imputation, are still used. In this work, we compare established and recently developed imputation techniques to guide researchers in improving clinical prediction model performance by choosing the most accurate imputation technique. We introduce an extensible, reusable benchmark with, currently, 15 imputation and 4 amputation methods created for benchmarking on major ICU datasets. We hope to provide a comparative basis and facilitate further clinical ML development to bring more models to practice.