The ability of deep neural networks to generalize over a large diversity of function modeling tasks, remains a key mystery of the field. While the workings of the network are fully known, it remains unclear which specific properties are necessary and/or sufficient for the observed generalization.In this paper, we approach the characterization of this generalization by studying the ability to learn the evolution of discrete dynamical systems. Our findings reveal a strong correlation between the number of examples needed for generalization and the sensitivity of the dynamical systems to perturbations of the initial state.