Early detection of critical events is one of the mainstays of clinical time series prediction tasks. As data from electronic health records become larger in volume and availability increases, models that can predict critical events before they occur and inform clinical decision making have the potential to transform aspects of clinical care. There has been a recent surge in literature looking at early detection in the context of clinical time series. However, methods used to evaluate clinical time series models in which multiple predictions per time series are made often do not adequately measure the utility of the models in the clinical setting. Classical metrics such as the Area Under the Receiver Operating Characteristic (AUROC) and the Area Under the Precision Recall Curve (AUPRC) fail to fully capture the true, real-world performance of these models. In this work, we i)propose a method to evaluate early prediction models in a way that is consistent with their application in the clinical setting, and ii) provide a fast, open-source, and native cross-platform implementation.