Reliable and precise diagnoses are essential to mitigate severe outcomes of COVID-19. Here we develop a deep learning workflow for robust detection of opacities and sub-classification of COVID-19 anomalies additionally supervised by segmentation of opacity locations. For the classification, we propose ensemble of convolutional neural networks with auxiliary branches that learn to segment the opacity regions for enhanced features. To detect opacities, we used ensemble of detectors trained in a semi-supervised manner. Our workflow was evaluated in the SIIM-FISABIO-RSNA COVID-19 challenge (https://www.kaggle.com/c/siim-covid19-detection). Our method was ranked 5th on the public (mAP 64.8%) and 7th (top 1%) on the private (mAP 62.6%) test sets out of a total of 1305 competing teams. Notably, we did not use any external datasets. Interestingly, geometrical augmentations significantly boosted our model performance, and CheXpert pretraining (much smaller than ImageNet) achieves comparable results to that of ImageNet pre-trained models. In summary, incorporating opacity segmentation branches directly into the classification model architecture appears to be a powerful strategy.