Poster
Mechanic: A Learning Rate Tuner
Ashok Cutkosky · Aaron Defazio · Harsh Mehta
Great Hall & Hall B1+B2 (level 1) #1219
Abstract:
We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call Mechanic. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate Mechanic on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, Mechanic either comes very close to, matches or even improves upon manual tuning of learning rates.
Chat is not available.