We investigate the problem of learning parameters (i.e., objective functions or constraints) of a multi-objective optimization problem, based on a set of sequentially arrived solutions. In particular, these solutions might not be exact and possibly carry measurement noise or are generated with the bounded rationality of decision makers. In this paper, we propose a general online learning framework to deal with this learning problem using inverse multi-objective optimization, and prove that this framework converges at a rate of under certain regularity conditions. More precisely, we develop two online learning algorithms with implicit update rules which can handle noisy data. Numerical results with both synthetic and real world datasets show that both algorithms can learn the parameters of a multi-objective program with great accuracy and are robust to noise.