Research on neural networks for time series has mostly focused on developing models that learn patterns about the target signal without the use of additional auxiliary or exogenous information. In applications such as selling products on a marketplace, the target signal is influenced by these variables, and leveraging exogenous variables is important. In particular, knowing that a product would go into promotion would mostly likely generate a spike in its demand; and ignoring this information would degrade the forecasting ability of the models. In such applications, the exogenous information comes as a mixture of categorical and real variables on different scales. In this paper we develop a decoder method that leverages the time structure of exogenous information through structured state-space model layers and learns relationships between the variables through MLPs. We show that this decoder method can be applied to a wide variety of models such as NBEATS, NHITS, PatchTST, and S4, yielding notable performance improvements across a different datasets.