Abstract: Learn-to-Defer is a paradigm that enables learning algorithms to work not in isolation, but as a team with human experts. In this paradigm, we permit the system to defer a subset of its tasks to the expert. Although there are currently systems that follow this paradigm and are designed to optimize the accuracy of the final human-AI team, the general methodology for designing such systems under a set of constraints (e.g., algorithmic fairness, expert intervention budget, defer of anomaly, etc.) remains largely unexplored. In this paper, using a $d$-dimensional generalization to the fundamental lemma of Neyman and Pearson ($d$-GNP), we obtain the Bayes optimal solution for learn-to-defer systems under a variety of constraints. Furthermore, we design a generalizable algorithm to estimate that solution, and apply this algorithm to COMPAS and ACSIncome dataset. Our algorithm shows improvements in terms of constraint violation over a set of baselines.