Lensless imaging systems use masks instead of lenses, reducing costs and making equipment lighter. However, before object recognition, reconstructing an image is necessary. This reconstruction involves parameter tuning and time-consuming calculations. Hence, a reconstruction-free object recognition scheme, which directly trains networks to recognize objects on encoded images captured via lensless cameras, that saves computing resources and protects privacy, is proposed herein. Using lensless cameras with a phase mask and an amplitude mask, the real MNIST dataset is collected and the simulated MNIST and Fashion MNIST datasets are generated. Subsequently, the ResNet-50 and Swin_T networks are trained on these datasets for object recognition. The results show that with respect to the simulated MNIST, Fashion MNIST, and real MNIST datasets, the highest recognition accuracy achieved by the proposed scheme is 99.51%, 92.31%, and 98.06%, respectively. These accuracies are comparable to those achieved by the reconstructed object recognition scheme, proving that the proposed scheme is an efficient end-to-end scheme that provides privacy protection. Moreover, the proposed scheme is verified using two types of masks and two types of conventional backbone classification networks.