Brillouin spectroscopy is an emerging tool for microscopic optical imaging as it allows for noninvasive and direct assessment of the viscoelastic properties of materials. Recent advances of background-free confocal Brillouin spectrometer allows investigators to acquire the Brillouin spectra for turbid samples as well as transparent ones. However, due to strong signal loss induced by the imperfect optical setup, the Brillouin photons are usually immersed in background noise. In this report, we proposed and experimentally demonstrated multiple approaches to enhance the signal collection efficiency. A signal enhancement by >4 times can be observed, enabling observation of ultra-weak signals.