Deep Learning (DL), a type of Machine Learning, has gained significant interest in many fields, includingagriculture. This paper aims to shed light on deep learning techniques used in agriculture forabiotic and biotic stresses detection in fruits and vegetables, their benefits, and the challenges facedby users. Scientific papers were collected from web of science, Scopus, Google scholar, Springer, andDirectory of Open Access Journal (DOAJ) using combinations of specific keywords such as: ’DeepLearning’ OR ’Artificial Intelligence’ in combination with ’fruit disease’, ’vegetable disease’, ’fruitstress’, OR ’vegetable stress’ following PRISMA guidelines. From the initial 818 papers identifiedusing the keywords, 132 were reviewed after excluding books, reviews, and the irrelevant. Therecovered scientific papers were from 2003 to 2022; ninety-three percent of them addressed bioticstress on fruits and vegetables. The most common biotic stresses on species are fungal diseases (greyspots, brown spots, black spots, downy mildew, powdery mildew, and anthracnose). Few studieswere interested in abiotic stresses (nutrient deficiency, water stress, light intensity, and heavy metalcontamination). Deep Learning and Convolutional Neural Network were the most used keywords,with GoogleNet (18.28%), ResNet50 (16.67%), and VGG16 (16.67%), the most used architectures.Fifty-two percent of the data used to compile these models come from the fields, followed by dataobtained online. We provided the gaps and some perspectives from the reviewed papers. Precisionproblems due to unbalanced classes and the small size of some databases were analyzed. The results suggest that further work should be done to improve the performance of the models.