Early detection of banana diseases is necessary to develop the effective control plans and minimize quality and financial losses. Fusarium Wilt Race 1 and Black Sigatoka diseases are among the most harmful banana diseases globally. In this study, we propose a model based on the Mask R-CNN architecture to effectively segment the damage of these two banana diseases. We also include a CNN model for classifying these diseases. We used an image dataset of 6000 banana leaves and stalks collected in the field. In our experiment, Mask R-CNN achieved a mean Average Precision of 0.04529, while CNN model achieved an accuracy of 33%. The Mask R-CNN model was able to accurately segment areas where the banana leaves and stalk were affected by Black Sigatoka and Fusarium Wilt Race 1 diseases in the image dataset. This model can assist farmers to take required measures for early controlling and minimizing the harmful effects of these diseases and rescue their yields.