A method for recognizing human-human interaction is proposed based on the spatial-temporal semantic information. Different from traditional methods to model the interactions, this framework achieves recognizing activities based on the transformation of the spatial semantic meaning. First, with detection and tracking results, the spatial semantic meaning (front, back, face to face, back to back, and left or right) between the persons are extracted based on motion directions and spatial relationships (including topological and directional relations). Then, stochastic context-free grammar is used to recognize interactions that the rules are learned based on the transformation of spatial semantics. Extensive experiments have been executed to validate the effectiveness of the proposed approach, and the method can recognize the interactions without additional training.