Usually, parameters of a machine learning (ML) model are used to fine-tune a new ML model using a new dataset. Since a ML model can generate other useful information, such as features, we propose a new method that extracts locally diverse features sets by using different ML models, then applies feature selection (FS) methods to identify the best globally diverse hybrid features, and finally uses them to build an accurate hybrid classifier. These ML models may be pretrained and/or non-pretrained. Simulation results using the medical image dataset DermaMNIST (from MedMNIST2D) indicate that the new hybrid classifiers using the hybrid features extracted by a fine-tuned pretrained ResNet model and the Vision Transformer (ViT) can outperform both the fine-tuned pretrained ResNet model and the ViT, and also perform more accurately than the five commonly used image classifiers (ResNet18, ResNet50, auto-sklearn, AutoKeras, and Google AutoML Vision). New optimization methods will be developed to extract highly informative feature sets from more fine-tuned pretrained ML models and other non-pretrained ML models, select best features, and build a highly accurate, fast, energy-efficient, and memory-efficient classifier for image recognition.