As reinforcement learning is increasingly being considered in the healthcare space, it is important to consider how best to incorporate practitioner expertise. One notable case is in improving tuberculosis drug adherence, where a health worker must simultaneously monitor and provide services to many patients. We find that without considering domain expertise, the state of the art algorithms allocates all resources to a small number of patients, neglecting most of the population. To avoid this undesirable behavior, we propose a human-in-the-loop model, where constraints are imposed by domain experts to improve the equitability of resource allocations. Our framework enforces these constraints on the distribution of actions without significant loss of utility on simulations derived from real-world data. This research opens a new line of research inquiry on human-machine interactions in restless multi-armed bandits.