To solve the problems that the averaging time of Allan variance can only reach half of total data length，and confidence at long averaging time is lower, according to the random errors properties of the inertial sensors at time and frequency domains, a new method for analyzing the random error properties of the inertial sensor based on high confidence and hybrid Allan variance (TheoH variance) is proposed, which can better adapt to the characteristics of optical gyroscope by changing the location of the transition point. The simulated random error of optical gyroscope and measured data of FOG start-up signal are analyzed with the TheoH variance and other variance analysis method. The results show that the estimation values of the TheoH variance are consistent with power-law noise，It can efficiently improve the confidence in the case of great averaging time, and its estimation accuracy is higher than the Allan variance, especially in long term τ-values.