The magnetization-induced nonlinear optical and nonlinear magneto-optical properties in a magnetic metal-insulator composite are studied based on a tensor effective medium approximation with shape factor and Taylor-expansion method. There is a weakly nonlinear relation between electric displacement D and electric field E in the composite. The results of our studies on the effective dielectric tensor and the nonlinear susceptibility tensor in a magnetic nanocomposite are surveyed. It is shown that such a metal-insulator composite exhibits the enhancements of optical and magneto-optical nonlinearity. The frequencies at which the enhancements occur, and the amplitude of the enhancement factors depend on the concentration and shape of the magnetic grains.