Approximate nonlinear equation about modes is derived by W. K. B. method for waveguide with perfectly matched layer (PML). Asymptotic solutions for the leaky modes and Berenger modes are derived and the effect of PML is analyzed. Theoretical analysis shows that for leaky modes of transverse electric (TE) case, asymptotic solutions of waveguide are the same as that of open waveguide, but for transverse magnetic (TM) case, asymptotic solutions of waveguide are different from that of open waveguide. For both TE and TM cases of three-layer slab waveguide, if refractive index of top cladding is different form that of bottom cladding, there will be two traces of Berenger modes, otherwise there is only one trace of Berenger modes. Numerical simulations show that these asymptotic solutions of leaky modes and Berenger modes are very close to exact solutions and as the norms of modes become larger, the error will become smaller.