Computer vision models applied in medical imaging domains are capable of diagnosing diseases beyond what human physicians are capable of unassisted. This is especially the case in cardiology, where echocardiograms, electrocardiograms, and other imaging methods have been shown to contain large amounts of information beyond that described by simple clinical observation. Using 67,762 echocardiograms and temporally associated laboratory hemoglobin test results, we trained a video-based deep learning algorithm to predict abnormal lab values. On held-out test data, the model achieved an area under the curve (AUC) of 0.80 in predicting abnormal hemoglobin. We applied smoothgrad to further understand the features used by the model, and compared its performance with a linear model based on demographics and features derived from the echocardiogram. These results suggest that advanced algorithms can obtain additional value from diagnostic imaging and identify phenotypic information beyond the ability of expert clinicians.