The fast assessment of the binding strength between adsorbates and catalyst surfaces is crucial for catalyst design, where global minimum adsorption energy (GMAE) is one of the most representative descriptors. However, catalyst surfaces typically have multiple adsorption sites and numerous possible adsorption configurations, which makes it prohibitively expensive to calculate the GMAE using Density Functional Theory (DFT). Additionally, most machine learning methods can only predict local minimum adsorption energies and rely on information about adsorption configurations. To overcome these challenges, we designed a graph transformer (AdsGT) that can predict the GMAE based on surface graphs and adsorbate feature vectors without any binding structure information. To evaluate the performance of AdsGT, three new datasets on GMAE were constructed from OC20-Dense, Catalysis Hub, and FG-dataset. For a wide range of combinations of catalyst surfaces and adsorbates, AdsGT achieves test mean absolute errors of 0.10 and 0.14 eV on the two GMAE datasets respectively, demonstrating its good reliability and generalizability.