In lossy image compression, models face the challenge of either hallucinating details or generating out-of-distribution samples due to the information bottleneck.This implies that at times, introducing hallucinations is necessary to generate in-distribution samples.The optimal level of hallucination varies depending on image content, as humans are sensitive to small changes that alter the semantic meaning.We propose a novel compression method that dynamically balances the degree of hallucination based on content.We collect data and train a model to predict user preferences on hallucinations.By using this prediction to adjust the perceptual weight in the reconstruction loss, we develop a \textbf{Con}ditionally \textbf{Ha}llucinating compression model (\textbf{ConHa}) that outperforms state-of-the-art image compression methods.