Modeling of protein side-chain conformations is a long-standing subproblem in protein structure prediction. It helps to refine experimental structures with poor resolution, and is used for sampling side chains in computational protein design. Related studies date back to the 1980s starting from statistically analyzing side-chain conformations, developing energy functions, and implementing algorithms for decomposing the side-chain interaction graph as subgraphs such as in SCWRL4. Here, we employ a geometric deep-learning method Relation-Shape Convolution (RSConv), originally applied to point clouds, to the side-chain problem. With features consisting of the backbone atom Cartesian coordinates (in a local frame), backbone dihedral angles, and residues types of neighbors, we achieve a favorable testing set accuracy of the chi1 dihedral angles of 89% (within 40° of the native structure) and chi2 accuracy of 83% given correct chi1 angles. Our prediction accuracy strongly correlates with the experimental atomic displacement B-factors of the side chains. The chi1 dihedrals with B-factor less than 30° representing about 53% of all side chains have prediction accuracy of 93%. The 93% rate is comparable to the chi1 accuracy in AlphaFold2 when it achieves high backbone structure recovery (100 IDDT Cα).