Recent advances in 3D sensing technologies make it possible to easily record color and depth images which together can improve object recognition. Most current methods rely on very well-designed features for this new 3D modality. We introduce a novel model based on sparse and recursive autoencoders (RAE) for learning both features and object categories from raw 3D point clouds as well as standard images. The model differs from previous RAE models in that it fixes the tree structures and includes short-circuit connections from all tree nodes to the final classifier. This allows the model to take into consideration both low-level features as well as global features of the object. Using our fully learned architecture, we achieve state of the art performance on a standard RGB-D object recognition dataset, rivaling random forest classifiers on hand-designed features such as SIFT and spin images. Our method is very fast and can classify 71 images in 1 second on a standard desktop machine in Matlab. This is possible because the method only requires 16 matrix multiplications to classify each image into one of 51 household objects.