On the basis of the principle of a VISNAV sensor, this paper researches how to determine the vision-based relative state between two satellites with disturbances. By taking the VISNAV sensor as a relative state sensor, the problem mentioned above is solved by a modified Kalman filter algorithm. In consideration of the relative orbit dynamics equation with disturbance and the relative attitude kinematic equation, it gives modified discrete equation and a calculation method for the error covariance matrix. By which the method avoids the complicated real-time operations of large dimensional matrices in every loop and reduces computation amounts. Finally, the corresponding modified Unscented Kalman Filter (UKF)method followed by a numerical simulation experiment is performed under STK and MATLAB. Results demonstrate that the method is feasible, effective and has good convergence. Under disturbance condition, its determination accuracy in 200 m is an angle second level for the relative attitude and a millimeter level for the relative position, which meets the requirements of determining the relative state between two satellites in a shorter distance.