Poster
in
Workshop: Socially Responsible Language Modelling Research (SoLaR)
Position: Governments Need to Increase and Interconnect Post-Deployment Monitoring of AI
Merlin Stein · Jamie Bernardi · Connor Dunlop
Keywords: [ sociotechnical ] [ evaluations ] [ reporting ] [ Post-deployment monitoring ] [ information ] [ incident reporting ]
Language-based AI systems are diffusing into society, bringing positive and negative impacts. Mitigating negative impacts depends on accurate impact assessments, drawn from an empirical evidence base that makes causal connections between AI usage and impacts. Interconnected post-deployment monitoring combines information about model integration and use, application use, and incidents and impacts. For example, inference time monitoring of chain-of-thought reasoning can be combined with long-term monitoring of sectoral AI diffusion, impacts and incidents. Drawing on information sharing mechanisms in other industries, we highlight example data sources and specific data points that governments could collect to inform AI risk management.