Most mucosal lesions originate from superficial epithelial tissues. In view of this, it is of great significance to study the distribution characteristics of spatially resolved diffuse reflection spectra for the early identification of mucosal lesions in the sub-diffusive regime. Under the circumstance, the second-order scattering orientation factor γ also greatly influences the diffuse reflectance of tissues, besides the absorption coefficient μ_a and reduced scattering coefficient μ'_s. For the quantitative and simultaneous determination of the three optical parameters, this paper firstly establishes a forward Monte Carlo numerical model based on optical fiber geometry and Gegenbauer kernel (GK) phase function for practical applications. Then, the 3D look-up table (3D-LUT) method and back-propagation neural network (BPNN) are used to determine the optical parameters and verified by numerical simulation and phantom experiments. The numerical simulation results show that small errors are produced by both methods, and BPNN has higher accuracy, faster calculation speed, and better robustness to the random noise than the 3D-LUT method. The phantom experiments demonstrate that the two methods can effectively determine the three optical parameters in the presence of the experimental system noise and the phantom calibration error, which proves the feasibility and effectiveness of the proposed method.