\textit{Generalized Category Discovery} seeks to cluster unknown categories while simultaneously discerning known ones. Existing approaches mostly rely on contrastive learning to produce distinctive embeddings for both labeled and unlabeled data. Yet, these methods often suffer from dispersed clusters for unknown categories due to a high rate of false negatives. To alleviate this problem, we introduce label smoothing as a hyperparameter that permits ‘forgivable mistakes’ for visually similar samples. We introduce a self-supervised cluster hierarchy, which allows us to control the strength of label smoothing to apply. By assigning pseudo-labels to emerging cluster candidates and using these as ‘soft supervision’ for contrastive learning, we effectively combine the benefits of clustering-based learning and contrastive learning. We demonstrate state-of-the-art generalized category discovery performance on various fine-grained datasets.