We propose KNN-Kmeans MT, a sample efficient algorithm that improves retrieval based augmentation performance in low resource settings by adding an additional K-means filtering layer after the KNN step. KNN-Kmeans MT like its predecessor retrieval augmented machine translation approaches doesn't require any additional training and outperforms the existing methods in low resource settings. The additional K-means step makes the model more robust to noise. We benchmark our proposed approach on EMEA and JTRC-Acquis dataset and see 0.2 points improvement in BLEU score on an average in low resource settings. More importantly, the trend of improvement from high to low resource setting is consistently obvious across both the datasets. We conjecture that the observed improvement is a consequence of eliminating bad neighbors as their retrieval databases are small and retrieving a fixed number of neighbors leads to adding noise to the model. The simplicity of the approach makes it a promising direction in opening up the use of retrieval augmentation in low resource setting.