Causal modeling is often valued for its interpretability in attributing cause and defining counterfactuals. However, these framings are fundamentally ideological, and may not align with political or sociological understandings of structural inequality or actions of resistance by marginalized people. We outline high-level conceptual conflicts in statistically modeling causal effects of race and sociologically understanding causal effects of racism. By drawing upon Disability Studies, we trace the logic of counterfactuals in social movements and theories to demonstrate how complicating notions of constructed social groups give rise to to differing definitions of fairness. These different counterfactual perspectives create systematic differences in calculated causal quantities, leading to common-sense fairness shortfalls in cases of assimilation, e.g., racism driving forced proximity to whiteness. We advocate for creating a formalized approach to defining these alternative constructions of fairness, articulating political-sociological limitations in counterfactual interpretations, establishing evaluation criteria for conflict with social change movements, and exploring possible interfaces between causal models and other fairness definitions.