In order to describe photon migration at small source detector separation and in media with low reduced albedo where diffuse approximation and P3 approximation have limitations, the solution of radiance transfer equation in the P5 approximation for the case of an isotropic continuouswave point source in an infinite medium was derived. The P3 and P5 were compared with Monte Carlo simulation to verify the correctness of the derived solution. Under the condition of high reduced albedo (0.97), the maximum relative error between P3 and Monte Carlo simulation is about 41.57% while that between P5 and Monte Carlo simulation is just about 13.17%. In the case of low reduced albedo (0.69), the maximum relative error is about 27.78%, 286.70%, respectively. In the situations of other optical parameters, the maximum relative errors between P5 and Monte Carlo simulation are also no more than that between P3 and Monte Carlo simulation. By abandoning the expressions related to the maximum eigenvalue which has little impact to the result of P5 calculation in the validity condition of P5, a simpler analytical expression can be obtained to accelerate the calculation of P5 approximation. Experiments on a solid phantom with a continuouswave radiance measurement system are conducted and the results show that the radiance calcualted from P5 approximation meets well with that from the experimental measuring at small source detector separation and in media with high or low albedo.