This paper proposes a sorting method for non-ferrous metal fragments based on machine vision to overcome the problems of complicated source, difficult sorting, and low recognition accuracy rate of waste non-ferrous metal fragments. Using color moments and Tamura texture characteristics, the optimized support vector machine (SVM) sorting algorithm based on principal component analysis (PCA) is established. Thus, a new concept of high-precision automatic sorting from the machine vision perspective is proposed herein. Results show that the proposed SVM algorithm based on color and texture features can effectively detect and classify metal fragments with an accuracy of 93.89% and improve the recognition speed. The proposed algorithm meets the requirements of large-scale and efficient separation of scrap metals.