Autism Spectrum Disorders (ASDs) are neurodevelopmental disorders which inhibits linguistic, cognitive, communication and social skills of affected individuals. Currently, ASD is diagnosed by means of time-consuming and expensive screening tests. Hence, Machine Learning (ML) techniques have been applied to construct predictive models able to diagnose autism at early stages. However, the binary setting (ASD vs not-ASD) and the not-exciting performance reached by such models highlight the need for further de-identified datasets and interdisciplinary work linking computer scientists and Subject Domain Experts (SDEs). In this work, we propose a novel dataset in which labels refer to the severity level of autism as required by the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) standard reference. Then we analyze the quality of resulting ML models (i.e. Random Forest, XGBoost, Neural Network) based not only on their performance metrics (i.e. precision, recall, F1) but also on the most important features they consider for classification and their similarity with the ones suggested by the SDE.