The calibration of the reconstruction matrix for polarization detection affects the accuracy of the polarization state restoration. In this paper, a detailed theoretical analysis of the reconstruction matrix calibration problem of a polarization detection system based on metalens is carried out. The selection of the calibration reference state is discussed, and two criteria of condition number and equal weighted variance are used to select the calibration reference state. Through the comparison of simulation and polarization detection reduction experiments, it is proved that selecting a regular octahedron reference state for calibration can obtain a more accurate reconstruction matrix and have better anti-noise ability.