Sensory prediction is vital to organisms, and humans have engineered complex neural networks to predict, but it is difficult to benchmark how well real and artificial agents predict given that ground truth is often unknown. We utilize so-called epsilon-Machines, a special type of hidden Markov model, to calibrate how well real and artificial agents predict. First we show that large random epsilon-Machines produce output that artificial agents do not predict very well, though they come close to limits set by Fano's inequality. But then, we note that newly collected data shows that neurons in a dish and humans are resource-rational predictors, meaning that they predict as well as possible given their limited memory-- an outgrowth of rate-distortion theory. This allows us insight into artificial neural networks as well, and we find that LSTMs predict as well as possible given limited memory in challenging (undersampled) conditions. Altogether, we advance the idea that epsilon-Machines can be used to benchmark the performance of predictive agents and also the idea that these agents might be only boundedly optimal at prediction because they are subject to limitations on memory.