Humans and animals need to adapt to dynamically changing environments given only few experiences. Augmenting data has been shown to substantially improve few-shot learning of artificial agents and recently been suggested to play a role in mental modeling and memory consolidation, crucial strategies for biological agents to improve behavioral flexibility. However, how such augmentation should be coordinated online during open-ended interactions with the world is unclear. We take inspiration from the brain in addressing this issue. Unexpected changes in the environment (formalized as high state prediction errors, SPEs) are associated with the phasic release of serotonin, a neurotransmitter known to mediate cognitive flexibility in humans. We hypothesize that serotonin triggers augmentations and that this facilitates adaptation to novel environments. In our simulations, learning from augmentations improves zero-shot state-prediction given new contexts in a minimal circular environment and in gridworlds. During online context-switching, we find that augmentations timed to high SPEs are particularly effective. These results suggest a potential role for serotonergic brain modulation in open-ended adaptation through self-initiated augmentation of experiences and supports a proposed computational mechanism by which augmentations may be endogenously regulated by natural and artificial systems.