As the complex refractive index of a rough-surface object is difficult to be inverted precisely, the current Vimal-Milo method for inversion was improved in this paper. The characteristics of the polarimatic bidirectional reflection from the rough surface were analyzed, and the defects of current Vimal-Milo method with a mode of "polarization ratio - angle correlation" were pointed out. Then, a improved Vimal-Milo inversion method was proposed based on the model of " relative polarization component - angle correlation". The inversion formula for the complex index was derived and a global searching was designed. According to the measurement data of polarization characteristics from green paint and aluminum, the inversion algorithm for the complex index was verified and the polarimatic bidirectional reflection characteristics of materials were also verified by the forward deducing and inverting. The inversion data with proposed method show that the inversion precision of the complex index of rough-surface aluminum has more improved than that of the Vimal-Milo method and the parameter errors for real and imaginary parts are within ±0.01 with the assessment index less-than 0.07. The inversion data by the forward deducing and inverting show that the error of radiance intensity of green paint has decreased 5 times that of Vimal-Milo method, and the difference from measuring data is within 0.01. These results all demonstrate that proposed inverting method can be used in the inversion for complex indexes of rough-surface objects and flat-surface objects.