The traditional Crossmedia Relevance Model(CMRM) is based on the relevance between visual information and annotation words, while ignoring the inter-word semantic relevance. Therefore, a new CMRM image semantic annotation model based on a graph learning was proposed. Firstly, the ontology of a sport field was established to label the images of the sport field according the annotation words in an image training set. Then, the traditional CMRM was adopted in the training images to complete the basic image annotations and obtain the image annotation result based on a probability model. Finally, the graph learning was used to refine the basic image annotations based on ontology concept similarity, and the top N keywords in the probability table for each image were chosen as the final annotation results. Experimental results show that the recall and precision of the proposed model are improved as compared with those of the traditional CMRMs.