Electronic structures and optical properties of transition metal (TM, TM refers to Ru, Rh and Pd, respectively) doped GaSb were studied by first-principles calculations. The results show that the three TMs are more likely to substitute for Ga to form TM@Ga defect and enhance the response of GaSb semiconductor to infrared photons, making the optical absorption edge of the system red shift. The doped TM@Ga introduce impurity levels distributed around the Fermi level in the band gap, which greatly strengthens the dielectric properties of the system. The enhancement of the photoelectric field intensity can promote the generation and migration of electron-hole pairs, thus improving the photoelectric conversion efficiency of GaSb semiconductor. The optical properties of TM@Ga doped GaSb are all superior to that of undoped system, but Ru is the best choice among the three dopants. Moreover, the molar concentration and location of Ru also have an impact on optical properties of GaSb semiconductor. The absorption amplitude of Ru-GaSb is the biggest when the concentration is 6.25% (atomic fraction), which can improve the photoelectric conversion efficiency and photocatalytic performance effectively.