In high-resolution astronomical spectral detection using multi-mode optical fibers for solar surface magnetic field and visual velocity measurement, there is a significant phenomenon of uneven energy distribution and speckle in the obtained spectral images. Analyzing the reasons, it can be concluded that during fiber optic spectroscopic imaging, the imaging system receives the image of the output fiber end at different wavelengths and spreads along the dispersion direction. The high dispersion rate of the spectrometer results in a very narrow spectral range, which has excellent coherence, corresponding to each pixel in the imaging system. There are multiple transmission modes in multimode optical fibers, and the energy center of the speckle pattern formed by interference between different modes at each wavelength will deviate from the fiber’s geometric center, reducing the accuracy of spectral measurement. To address this issue, this paper proposes a multi-dimensional mechanical perturbation system to suppress speckle effects and improve the accuracy of fiber optic spectral measurements. The multi-dimensional mechanical disturbance device consists of three mechanical structures with reciprocating motion in different directions and frequencies. By adjusting three mechanical devices at different frequencies, the conduction mode in the optical fiber undergoes a nearly random phase drift, resulting in a random change in the speckle pattern at the output end of the optical fiber. After long-term exposure (corresponding to superimposing and averaging multiple speckle patterns), the influence of speckles can be eliminated. To test the mode disturbance effect of the system, the energy distribution of the outgoing speckle field of a 650 nm laser was studied using a fiber with a core diameter of 35 μm. The standard deviation of the energy center position was proposed as the evaluation function for the dispersion of the fiber mode speckle energy and its impact on the accuracy of astronomical fiber spectral measurement. Firstly, a comparative analysis was conducted on the average stacking effect of 1 000 speckle patterns under non-disturbance mode, manual disturbance mode, one-dimensional disturbance mode, and multi-dimensional disturbance mode. The results showed that manual disturbance mode and multi-dimensional mechanical disturbance mode had better effects, and the energy distribution of the average speckle pattern stacking was relatively uniform. Then, the multi-dimensional perturbation effect of the number of speckle pattern superpositions(equivalent to different exposure times) was compared. The experimental results show that after stacking and averaging 100 speckle patterns with multi-dimensional mechanical disturbance, the standard deviation is only one-thirteenth of that of a single speckle pattern. Finally, the influence of the frequency and amplitude of the scrambler device on the scrambler effect is tested. The results show that higher frequency and amplitude are beneficial to speckle suppression. For the optical fiber in this experiment, the frequency of 1.2 Hz and the amplitude of 6 cm are the most appropriate. Under these conditions, the mean centroid deviation distance of the speckle is the smallest, which is 0.21 pixels.