The COVID-19 outbreak demonstrates the need for measurement of hand hygiene behaviors such as handwashing and face touching to prevent the spread of infectious diseases. Wearable technologies and machine-learning-based algorithms can be used to automatically detect these behaviors. In this work, we demonstrate a recurrent neural network with a set of local-extrema-based features for detecting hand hygiene behaviors (handwashing and face touching activities simultaneously) using data from inertial sensors (i.e., accelerometer, magnetometer, and gyroscope) on the wrist(s). The training and validation dataset were gathered from ten individuals; each person provided 60 min of data (sampled at 100 Hz) while performing 12 steps of handwashing, 8 variations of face touching, and 7 variations of other face-to-head gestures across six sessions. With 10 min of person-specific training data, the real-time algorithm achieved its best performance (F1-score of 0.88 for handwashing steps and 0.80 for face touching) using leave-one-session-out validation. We also describe a pilot evaluation on six-hour, free-living waking-day datasets of two participants annotated via front-facing video.