To tackle the challenges posed by the cumbersome computation and intricate decoding structure of codec semantic segmentation networks, we present a novel decoder-free binary semantic segmentation model DFNet. By discarding the complex decoding structure and jump connections that are ubiquitous in conventional segmentation networks, our model adopts a convolutional remolding upsampling method to directly reshape feature coding and obtain precise segmentation results, significantly streamlining the network architecture. Moreover, our encoder integrates a lightweight dual attention mechanism EC&SA to facilitate the effective communication of channel and spatial information, bolstering the network’s coding capability. To further enhance the model’?s segmentation accuracy, we replace the traditional segmentation loss with PolyCE loss, a powerful tool that resolves the issue of positive and negative sample imbalance. Experimental results on binary segmentation datasets such as DeepGlobe road segmentation and Crack Forest defect detection show that the segmentation accuracy F1 mean and IoU mean of this model reach 84.69% and 73.95%, respectively, and the segmentation speed is as high as 94 FPS, which far exceeds the mainstream semantic segmentation model and greatly improves the efficiency of the segmentation task.