In order to verify the feasibility of the mathematical model of two-circle coaxial optical fiber sensors for measuring the surface topography of objects, the parameters of the novel structure optical fiber sensor were optimized to improve the linear measurement range and sensitivity of the optical fiber sensors. The mirror paper, which does not affect the front slope data of the modulation characteristic curve, was used as the reflector for experimental verification. The theoretical analysis and mathematical modeling of the new structure of optical fiber sensor were also carried out. Then, the tilt angle of the novel structure was optimized by using particle swarm optimization. The results show that the experimental curve obtained by using mirror paper as reflector coincides with the simulation modulation characteristic curve. The feasibility of the mathematical model of the two-circle coaxial optical fiber sensor used to measure the surface topography of the object is proved. The slope angle optimized by particle swarm optimization is 2.33°. At this time, the linear measurement range and sensitivity of the front slope are 1.14 times and 1.89 times respectively when the inclination angle is zero. The novel structure of two-circle coaxial optical fiber sensor shows good measurement performance and application prospects.