The calculation of light intensity distribution for an imaging system based on special-shaped aperture diaphragm has a significant theoretical importance for developing imaging techniques such as the automatic focus and depth-of-field expansion of microscopic systems. The traditional Fresnel diffractive light intensity distribution calculation formula applies only to the problems of an axisymmetric aperture diaphragm and light intensity distribution in the focal plane. We derived the mathematical relationship between the optical field intensity distribution of the image space suitable for the arbitrary aperture shape and arbitrary out-of-focus amount using scalar diffraction theory along with a combination of spectral transformation and orthogonal separation. Furthermore, by using the principle of discrete Fourier transform, we obtained the numerical calculation expression of optical intensity distribution for the imaging space of an optical imaging system with a particular aperture diaphragm. Additionally, we compared the calculation and analysis of the obtained numerical calculation expression with that of the traditional calculation formula, which prove the accuracy of the derived mathematical model calculation at a circular aperture diaphragm and focal plane. For a semicircular aperture diaphragm, under the same system parameters, at the three image positions of 0, 4, and 8 μm, respectively, the results calculated using the obtained mathematical model theory agree with the experimental test results, thereby indicating that the derived mathematical model applies to any shape of the aperture diaphragm.