The high-speed and high-accuracy geometric calibration is the key step for building projection display system, which is especially important for the panoramic display system that composed of a large number of projectors. In response to this, a geometric calibration method based on fisheye camera is proposed and its theoretical derivation is given. The proposed method firstly calibrates the fisheye camera, and then builds the sub-pixel level mapping relationship between projector and camera through capturing the structured light feature fringes on the display wall. Finally, the geometric calibration data for each projection channel is calculated. Experimental results show that the proposed method can provide a high-speed and high-accuracy solution for large-scale panoramic display system, and it has a good application value.