Deep learning and language models are increasingly used to model DNA and protein sequences. While many models and tasks are inspired and borrowed from the field of natural language processing, biological sequences have specificities that deserve attention. In this talk I will discuss two such specificities: 1) the inherent symmetry in double-stranded DNA sequences due to reverse-complement pairing, that calls for equivariant architectures, and 2) the fact that sequence alignment is a natural way to compare evolutionary related sequences.