Many advances in machine learning can be attributed to designing systems with inductive biases well-suited for particular tasks. However, it can be challenging to ascertain what inductive biases a learning system has, much less control them in the design process. We propose a framework to capture the inductive biases in a learning system by meta-learning hyperparameters of a Gaussian process from observations of the behavior of a machine learning system. We illustrate the potential of this framework across several case studies, including investigating the inductive biases of both untrained and trained neural networks, and assessing whether a given neural network family is well-suited for a task family.