Skip to yearly menu bar Skip to main content


Invited Talk
in
Workshop: I Can’t Believe It’s Not Better (ICBINB): Failure Modes in the Age of Foundation Models

Measurement in the Age of LLMs: An Application to Political Ideology Scaling

Aaron Schein


Abstract:

Much of social science is centered around terms like “ideology” or “power”, which generally elude precise definition, and whose contextual meanings are trapped in surrounding language. This talk explores the use of large language models (LLMs) to flexibly navigate the conceptual clutter inherent to social scientific measurement tasks. We rely on LLMs’ remarkable linguistic fluency to elicit ideological scales of both legislators and text, which accord closely to established methods and our own judgement. A key aspect of our approach is that we elicit such scores directly, instructing the LLM to furnish numeric scores itself. This approach is methodologically "dumb" and shouldn't "work" according to classical principles of measurement. We nevertheless find surprisingly compelling results, which we showcase through a variety of different case studies.

Chat is not available.