Zero-shot classification (ZSC) is the task of learning predictors for classes not seen during training. The different proposals in the literature are commonly evaluated using standard category splits but little attention has been paid to the impact on performance under different class partitions. In this work we show experimentally that ZSC perform with strong variability with respect to the class partitions. We propose an ensemble learning method to attempt to mitigate it.