Abstract: Joint observations in electromagnetic and gravitational waves shed light on the physics of objects and surrounding environments with extreme gravity that are otherwise unreachable via siloed observations in each messenger. However, such detections remain challenging due to the rapid and faint nature of counterparts. Protocols for discovery and inference still rely on human experts manually inspecting survey alert streams and intuiting optimal usage of limited follow-up resources. Strategizing an optimal follow-up program requires adaptive sequential decision-making given evolving light curve data that maximizes a global objective despite incomplete information and is robust to stochasticity introduced by detectors/observing conditions. We design a novel reinforcement learning agent that executes such a design for the goal of maximizing follow-up photometry for the true kilonova among several contaminant transient light curves from the Zwicky Transient Facility. It achieves 3$\times$ higher accuracy compared to a random strategy and comes close to human-level performance. We suggest that more complex agents (e.g. using deep Q networks or policy gradient algorithms) could perform at par or surpass human experts. Agents like these could pave the way for machine-directed software infrastructure to efficiently respond to next generation detectors, for conducting science inference and optimally planning expensive follow-up observations, scalably and with demonstrable performance guarantees.