Enhancement of nonlinear optical susceptibility can occur in layered composite materials. The equivalent dielectric constant for TM modes in an optical waveguide of layered composite materials is analyzed. We find that under experimentally realizable conditions the nonlinear second enhancement of such a structure can exist because of the waveguide boundary effects.