We demonstrate how representational similarity can be leveraged to improve accuracy in medical image decision-making. In a series of experiments conducted on novices and experts, we aggregate responses made by a single individual on similar images to improve overall accuracy on the task. The similarity between the two images was calculated as the euclidean distance between representations obtained from artificial neural networks. Across our experiments, we observed that this algorithm can make significant improvements for novices but not for experts, suggesting that both of them have different decision-making mechanisms. We observe that experts make similar decisions on similar images, unlike novices which indicate that experts are more biased in their errors whereas novices make errors more randomly.