Reflective photoplethysmography (PPG) has become the standard sensing technique in wearable devices to monitor cardiac activity via a person's heart rate (HR). However, PPG-based HR estimates can be substantially impacted by factors such as the wearer's activities, resulting motion artifacts, sensor placement, and environmental characteristics such as temperature, all of which can decrease prediction reliability. In this paper, we show that state-of-the-art HR estimation methods struggle with representative data from everyday activities in outdoor environments, likely due to their reliance on existing datasets that captured controlled conditions. We introduce a novel dataset and benchmark results for continuous PPG recordings from 16 participants over 13.5 hours, captured from wearable sensors on four different locations on the body, totaling 216 hours. The recordings include accelerometer, temperature, and altitude data, as well as a synchronized Lead I-based electrocardiogram for ground-truth HR references. Participants completed a round trip to a tall mountain in Europe over the course of one day. This included outdoor and indoor activities such as walking, hiking, stair climbing, eating, drinking, and resting at various altitude levels and temperatures as well as taking trains, cable cars, lifts, and cars for transport---all of which impacted participants' physiological dynamics. We also introduce a HR estimation method, compare it with existing baselines, and show that it can produce more robust values for the real-world scenarios captured by our dataset. The dataset and HR estimation method are available at https://siplab.ethz.ch/datasets/wildppg-neurips-2024/