The field of political philosophy has spent centuries of collective effort examining questions around what it means to be human, to act morally, and to coordinate groups for making collective decisions. In this work, we give a brief summary of some high-level ideas about agency from the political science and philosophy literature, and explore what consequences these theories may suggest for the pursuit of moral or ethical AI. In particular, while there may be fundamental disagreement about whether AI satisfies the definition of an "agent" (and therefore also about the corresponding moral implications), "ethical AI" is undeniably a political process. When understood as such, ideas about collective action, majoritarianism, and legitimate governance can be useful frameworks for how we ought to reason about AI as value-laden technology. (This is very much an early work in progress --- we plan to develop these ideas further in the coming months, and think feedback from workshop participants would be invaluable! In future iterations of this work, we hope to conclude with concrete suggestions for technical and interdisciplinary work on ethical AI.)