A focus error detection system in sub-micrometer grade is designed based on critical angle method with high resolution and low light losses. The principle of defocus detection by critical angle method is introduced and the expression of defocus is deduced with Gauss optical formula based on Fresnel formula and a reasonable hypothesis. The detection system consists of He-Ne laser, critical angle prism, four-element photodetector, signal sampling circuit and data acquisition card, etc.,the error of defocus is detected by single light path critical-angle method. Moreover,a digital filter and the normalization technology are used to extract the Focus Error Signal(FES) to obtain the defocus curve. Experimental results indicate that the focus resolution is less than 15 nm and static linearity range is ±4 μm,which satisfies the focus error detection system requirements in sub-micrometer grade and provides references for engineering application of focus error detection system with critical-angle method.