The current machine learning applications in the hiring process are prone to bias, especially due to poor quality and small quantity of data. The bias in hiring imposes potential societal and legal risks. Thus, it is important to evaluate ML applications' bias in the hiring context. To investigate the algorithmic bias, we use real-world employment data to train models for predicting job candidates' performance and retention. The result shows that ML algorithms make biased decisions toward a certain group of job candidates. This analysis motivates us to resort to an alternative method---AI-assisted hiring decision making. We plan to conduct an experiment with human subjects to evaluate the effectiveness of human-AI collaboration for algorithmic bias mitigation. In our designed study, we will systematically explore the role of human-AI teaming in enhancing the fairness of hiring in practice.