Surface energy is one of the most important physical and chemical properties for crystals, which has a significant impact on surface catalysis, surface adsorption, epitaxial growth, nucleation, and dendrite growth. Rapid calculation and prediction of crystal surface energies can favor accelerating the design and optimization of catalysis materials, battery materials, and alloys. In this paper, a data-driven machine learning algorithm was proposed with a crystal graph convolutional neural network framework for the prediction of metal surface energy from the crystal structure. Using a physics-based surface representation that couples the surface dimensions to the atomic and bonding features of the crystal, we obtained an MAE value of less than 0.002 eV/?2, which surpasses other math-based surface models. Compared with the first-principles calculation, the computation time is reduced by approxiamtely 5 orders of magnitude. In addition, we discussed the main challenges and solutions towards the surface energy prediction of more complicated systems such as Silicates. It is expected that this work could be a paradigm for the surface energy prediction with machine learning.