To address the memory and computational resource constraints of IR small-target detection under an embedded hardware platform, high-frame-rate detection demand, and higher target-level detection performance requirements, a detection network called CAMNet is proposed. The network combines the advantages of self-attentive global modeling with the lightweight and fast processing characteristics of convolution and adopts a four-stage stacked encoder and decoder architecture, which effectively reduces the algorithmic resource requirements and improves the detection frame rate. A center-of-mass loss function is proposed in terms of the loss function, which effectively improves the target-level detection performance of the algorithm. Experimental results on the public SIRST dataset show that CAMNet achieves a detection frame rate of 107 FPS on common embedded platforms. Compared with other state-of-the-art networks, such as ISTDU-Net and UIU-Net, CAMNet improves the probability of detection by at least 0.76% and reduces the false alarm rate by at least 87.30%. These findings indicate that the proposed detection network offers both fast detection speed and superior target-level detection performance.