Abstract: One-shot image segmentation (OSS) methods enable semantic labeling of image pixels without supervised training with an extensive dataset. They require just one example (image, mask) pair per target class. Most neural-network-based methods train on a large subset of dataset classes and are evaluated on a disjoint subset of classes. We posit that the data used for training induces negative biases and affects the accuracy of these methods. Specifically, we present evidence for a \textit{Class Negative Bias} (CNB) arising from treating non-target objects as background during training, and \textit{Salience Bias} (SB), affecting the segmentation accuracy for non-salient target class pixels. We also demonstrate that by eliminating CNB and SB, significant gains can be made over the existing state-of-the-art. Next, we argue that there is a significant disparity between real-world expectations from an OSS method and its accuracy reported on existing benchmarks. To this end, we propose a new evaluation dataset - Tiered One-shot Segmentation (TOSS) - based on the PASCAL $5^i$ and FSS-1000 datasets, and associated metrics for each tier. The dataset enforces uniformity in the measurement of accuracy for existing methods and affords fine-grained insights into the applicability of a method to real applications. The paper includes extensive experiments with the TOSS dataset on several existing OSS methods. The intended impact of this work is to point to biases in training and introduce nuances and uniformity in reporting results for the OSS problem. The evaluation splits of the TOSS dataset and instructions for use are available at \url{https://github.com/fewshotseg/toss}.