Large language models (LMs) acquire substantial knowledge during pretraining but often need adaptation to new contexts, tasks, or domains, typically achieved through fine-tuning or prompting. However, fine-tuning incurs significant training costs, while prompting increases inference overhead. Inspired by fast weight memory, we introduce GenerativeAdapter, an effective and efficient adaptation method that encode test-time context into LM's parameters with a single forward pass.GenerativeAdapter augments a frozen pretrained LM with a lightweight adapter generator, trained via self-supervised learning, to produce parameter-efficient adapters.Notably, our generator is general-purpose, i.e., one generator can adapt the corresponding base model for all langauge processing scenarios.We apply GenerativeAdapter to two pretrained LMs (Mistral-7B and Llama2-7B) and evaluate the adapted models across knowledge acquisition from documents, learning from demonstrations, and personalization for users.Overall, GenerativeAdapter provides a viable solution for adapting large LMs to evolving information and providing tailored user experience, while reducing training and inference costs relative to traditional fine-tuning and prompting techniques.