Space operations such as on-orbit assembly and maintenance hinges on the use of large-scale and high-precision on-orbit measurement methods. Vision measurement holds the greatest potential in this regard. However, insufficient assistant artificial targets necessitate the deployment of on-orbit targets to reliably and accurately calibrate the multi-camera vision measurement system. To address this issue, this study proposes an exterior parameter calibration method using fixed stars and scale rulers. First, we propose an imaging model of fixed stars and scale rulers based on relative exterior parameters to solve the problem of multi-camera localization and orientation when there are insufficient artificial target points. Then, we propose a weighted joint bundle adjustment algorithm based on prior error estimation, which fuses three different kinds of observation data to achieve high-precision exterior parameter calibration. Real data experiments demonstrate that this calibration method yields standard deviations of image errors of the fixed stars and the scale ruler endpoints of 0.48 μm (1/7 pixel) and 0.21 μm (1/16 pixel), respectively. In addition, with this calibration method, the standard deviations of spatial coordinate measurement errors along the X, Y, and Z axes are 0.15 mm, 0.04 mm, and 0.05 mm, respectively, within the measurement range of 2.5 m×1.4 m. This study provides a method and reference data for calibrating vision system parameters in on-orbit applications.