An appealing method for Neural Architecture Search (NAS) is based on growing networks via small local changes to the network’s architecture called network morphisms. These methods start with a small seed network and progressively grow the network by adding new neurons in an automated way. However, efficiently determining the best way to grow the network remains a challenge. Here we propose a NAS method for growing a network which uses a Gauss-Newton approximation of the loss function to efficiently learn and evaluate candidate network morphisms. We then optimize this approximate loss function to efficiently learn morphism parameters. We compare our method with similar NAS methods for CIFAR-10 and CIFAR-100 classification tasks, and conclude our method learns similar quality or better architectures at a smaller computational cost.