Directed evolution is a powerful but resource-intensive method for optimizing protein functionalities by screening a vast range of mutations. Recent advances in machine learning aim to streamline this process by using surrogate sequence-function models. We propose Latent-based Directed Evolution (LDE), an evolutionary algorithm that efficiently explores high-fitness mutants in the latent space. Built on a regularized variational autoencoder (VAE) and the state-of-the-art Protein Language Model ESM-2, LDE creates a meaningful latent representation of sequences and combines gradient-based methods with directed evolution for effective fitness landscape traversal. Our experimental results on eight protein design tasks show that LDE outperforms existing baseline algorithms.