We propose a framework for comparing a set of image representations (artificial or biological) in terms of their sensitivities to local distortions. We quantify the local geometry of a representation using the Fisher information matrix (FIM), a standard statistical tool for characterizing the sensitivity to local distortions of a stimulus, and use this as a substrate for a metric on the local geometry of representations in the vicinity of a base image. This metric may then be used to optimally differentiate a set of models, by optimizing for a pair of distortions that maximize the variance of the models under this metric. We use the framework to compare a set of simple models of the early visual system, identifying a novel set of image distortions that allow immediate comparison of the models by visual inspection. In a second example, we show that the method can reveal distinctions between standard and adversarially trained object recognition networks.