The variable rotation transformation (VRT) method is proposed to control nonlinear chaos system. Control principle is introduced,and acousto-optical bistable (AOB) system is taken as a typical example to demonstrate the effectiveness of this method. The dynamical behaviors of controlled AOB system are given by computer numerical simulation. The results show that the correlation coefficients cosθ, sinθ are system control parameters, the Lyapunov exponent λ of the system can be changed from the positive numerical value in the chaotic state into the negative numerical value in the periodic state by appropriately selecting correlation coefficients ,and the method not only can control the unstable periodic 2np orbits embedded in initial system to stable periodic orbits, but can control chaos system to other periodic 3mp and 2n×3mp (n,m are integers) orbits.