Animal-derived food is one of the most essential parts of human nutrient ingestion. Veterinary drugs are vital for farming and are widely used for livestock breeding and disease prevention. However, excessive veterinary drug residue has severely impacted consumers’ health, which also hinders the development of animal-derived food. In such a concern, developing a rapid and effective detecting method is important to avoid adverse effects on consumers’ health. As a trace-level detection method, surface-enhanced Raman spectroscopy (SERS) demonstrates great potential in fulfilling the rapid, effective, and sensitive demands for veterinary drug residue in animal-derived food. This work reviewed the development of the SERS-based detection method for veterinary drug residue in animal-derived food, including meat (i.e., pork, chicken, duck, and fish), dairy products, and honey products. First, this review introduced the development of SERS technology in detecting the primary veterinary drug in meat products. The veterinary drug analysis includes several aspects, for example, tetracycline, sulfonamides, enrofloxacin, hormones in poultry products, β-agonists, chloramphenicol, and levamisole in pork, dye, sulfonamides, chloramphenicol in fish products. Second, the SERS-based detection of tetracycline, aminoglycosides, penicillin, and amide alcohols in dairy products is discussed. Third, this review briefly introduced the use of SERS for chloramphenicol and tetracycline detection in honey products. Finally, the conclusion and the perspectives of the SERS detection technology in animal-derived food are provided. The SERS demonstrates broad interest in the trace-level analysis of complicated chemical components in the food industry, especially suitable for prohibited and restricted chemical substances that may be hazardous to human health, making this technology highly perspective. However, opportunities exist with challenges. Breaking through the key technical bottlenecks, establishing rapid detection strategies for veterinary drug residue in animal-derived food, and developing on-site and real-time detection protocols will be significant in food safety supervision.