Pose estimation between LIDAR and imaging system is the prerequisite for the data fusion. Among current mainstream off-line calibration methods, common checkerboard is generally effective for 64-line and above LIDAR, but not for 16-line LIDAR due to its sparse data and will lead to large error. Furthermore, when involving calibration of infrared imaging system, specially-made checkerboard is needed to produce difference of emissivity. Aiming at the problem of less information provided by sparse LIDARs, we propose a new calibration method that can jointly calibrate LIDAR and visible/infrared imaging systems. A novel diamond-shaped nine-hole calibration board is designed, and a geometric constraint loss function is proposed to optimize the coordinates of feature points. Finally, the infrared and visible light imaging systems are used respectively, to calibrate with 16-line LIDAR. Good results are achieved and show that, all the average reprojection error is within 3 pixels. The proposed method can also be used in calibration of multi-band imaging systems that include sparse LIDAR, visible imaging system and infrared imaging system.