To target challenges in differentiable optimization we analyze and propose strate-gies for derivatives of the Matérn kernel with respect to the smoothness parameter.This problem poses a challenge in Gaussian processes modelling due to the lack ofrobust derivatives of the modified Bessel function of second kind. In the currentwork we scrutinize the mathematical and numerical hurdles posed by the differ-entiation of special functions and provide a set of options. Special focus is givento a newly derived series expansion for the modified Bessel function of secondkind which yields highly accurate results using the complex step method and ispromising for classical AD implementations.