Accurate detection of foreign objects on ballastless railway track beds is crucial for ensuring train safety. Although unsupervised anomaly detection algorithms based on deep learning address the effect of insufficient abnormal data on detection, the "generalization" ability of the encoder can proficiently reconstruct anomalous instances, thereby affecting detection accuracy. To solve this problem, this study proposes an anomaly detection framework for ballastless track beds utilizing image inpainting. First, inpainting was employed to obscure and subsequently restore the image using training on nonanomalous and incomplete image data, aiming to improve the model's contextual semantic understanding and enhance its reconstruction ability. Second, the maximum value obtained from the average anomaly map of the test and reconstructed images, which was analyzed across multiple scales, was utilized as the reconstruction error to calculate the anomaly score. This step aimed to widen the reconstruction error boundary between the abnormal and normal images. Finally, experimental results show a notable advantage of the proposed algorithm over alternative methods on public datasets, such as MNIST, CIFAR-10, and the ballastless track bed dataset.