Despite great academic attention towards gender bias in computer representations of language, most of this research conceptualises gender as binary and as inherently tied to biological sex (Devinney et al., 2022). This does not align with more critical conceptualisations of gender, such as gender performativity theory (Butler, 1990). In response, this paper conducts a critical audit of the representation of gender in the language model BERT, investigating the extent to which BERT can adapt to evolving social norms and distinguishes between gender and sex as separate categories. It finds that while a singular BERT model can not fulfil the gender performativity requirement of adapting to changing social norms, finetuning might be a viable strategy. It also finds that the base version of BERT encodes a binary understanding of gender and sex, with narrower encodings for words typically associated with maleness. These findings underscore current limitations of language models in representing gender beyond a binary and highlight a need for more critical engagement with gender theory in literature on technological gender bias.