In this work, we introduce backtracking counterfactual fairness, a novel definition of counterfactual fairness that uses backtracking rather than interventional counterfactuals. This definition captures the following intuition: would changing your predicted outcome place an undue burden on you? Our definition is compatible with different normative choices about what constitutes an undue burden. Backtracking counterfactuals, unlike interventional counterfactuals, consider counterfactual worlds in which the causal mechanisms remain unchanged. This allows backtracking counterfactual fairness to avoid one of the key sociological and normative tensions running through other counterfactual-based fairness notions: modularity. We demonstrate how our proposal relates to other notions of fairness and fair recourse on both real and simulated data, suggesting a novel way to make use of causal information for more equitable decision making and a possible path to considering counterfactual-based fairness notions even in the presence of non-modular variables.