A key dwell time algorithm in bonnet polishing was investigated to get the high-precision optical surface. Firstly, the removal function of bonnet polishing was acquired from the process experiments. Based on the matrix iteration, an appropriate initial dwell time function was given to calculate the dwell time with the liminal removal method . Then, the variance of the residual error was considered to optimize the result of dwell time function and to acquire the dwell time function quickly.The method is suitable for polishing the optical elements such as aspheric surface, and free-form surface. A simulation experiment was performed on a residual surface error through MATLAB, and the results show that the residual error PV value has reached up to 0.1 μm. Finally, an optical glass surface was processed on the bonnet polishing machine developed by ourselves, and results show that its surface roughness Ra is reduced from 0.159 μm to 0.024 μm and the PV value from 0.756 μm to 0.158 μm. Moreover, the rationality of the dwell time algorithm was verified. It demonstrates that the matrix-based algorithm satisfies the requirement of optical precision machining very well and provides theoretical basis for bonnet polishing of complex optical elements.