In this study, we investigate the similarity of representations between biological and artificial visual systems that are optimized for object recognition. We propose that this similarity could be a result of constraints on the representation of task-optimized systems, which necessitate the development of an abstraction from the input stimuli. To measure this, we constructed a two-dimensional coordination system in which we measured the distance of each neural representation from the pixel space and the class space. Our results show that proximity in this space predicts the similarity of neural representations between different visual systems. We observe that the trajectories of representations in any given task-optimized visual neural network start close to the pixel space and gradually move towards higher abstract representations such as categories. This suggests that the similarity between different task-optimized systems is due to constraints on representational trajectories, as revealed by the abstraction space. We present abstraction space as a simple yet effective analysis tool to draw inferences on the representations of neural network and to uncover the constraints that lead to similar representations in different visual systems.