In this paper we tackle the problem of Generalized Category Discovery (GCD). Given a dataset with labelled and unlabelled images, the task is to cluster all images in the unlabelled subset, whether or not they belong to the labelled categories. Our first contribution is to recognize that most existing GCD benchmarks only contain labels for a single clustering of the data, making it difficult to ascertain whether models are using the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. As such, we present a synthetic dataset, named 'Clevr-4', for category discovery. 'Clevr-4' contains four equally valid partitions of the data, i.e. based on object shape, texture, color or count. To solve the task, models are required to extrapolate the taxonomy specified by the labelled set, rather than simply latching onto a single natural grouping of the data. We use this dataset to demonstrate the limitations of unsupervised clustering in the GCD setting; showing that even very strong unsupervised models fail on 'Clevr-4', and further reveal that they each have characteristic biases from their pre-training. We also use 'Clevr-4' to examine the weaknesses of existing GCD algorithms, and propose a new method which addresses these shortcomings, outperforming state-of-the-art models on 'Clevr-4' and the challenging Semantic Shift Benchmark.