Monolithic deep learning models are typically not interpretable, and not easily transferable. They also require large amounts of data for training the millions of parameters. Alternatively, modular neural networks (MNN) have been demonstrated to solve these very issues of monolithic neural networks. However, to date, research in MNN architectures has concentrated on their performance and not on their interpretability. In this paper we would like to address this gap in research in modular neural architectures, specifically in the gated modular neural architecture (GMNN). Intuitively, GMNN should inherently be more interpretable since the gate can learn insightful problem decomposition, individual modules can learn simpler functions appropriate to the decomposition and errors can be attributed either to gating or to individual modules thereby providing either a gate level or module level diagnosis. Wouldn't that be nice? But is this really the case? In this paper we empirically analyze what each module and gate in a GMNN learns and show that GMNNs can indeed be interpretable, but current GMNN architectures and training methods do not necessarily guarantee an interpretable and transferable task decomposition.