As the quality of image generators continues to improve, deepfakes become a topic of considerable societal debate. Image watermarking allows responsible model owners to detect and label their AI-generated content, which can mitigate the harm. Yet, current state-of-the-art methods in image watermarking remain vulnerable to forgery and removal attacks. In this work, we first demonstrate that the initial noise used in the diffusion process can itself be a distortion-free watermarking method for images. However, detecting the watermark requires comparing the latent noise of an image to all previously used initial noises. Additionally, the initial noise may still be susceptible to some removal attacks.To mitigate these issues, we propose a two-stage watermarking framework for efficient detection. During generation, we augment the initial noise with generated Fourier patterns to embed information about the group of initial noises we used. For detection, we (i) retrieve the relevant group of noises, and (ii) search within the given group for an initial noise that might match our image. This watermarking approach achieves state-of-the-art robustness to forgery and removal against a large battery of attacks. The project code is anonymously available at https://github.com/anonymousiclr2025submission/Hidden-in-the-Noise.