Super-resolution structured illumination microscopy (SR-SIM) has become a widely used nanoscopy technique for rapid, long-term, and multi-color imaging of live cells. Precise but troublesome determination of the illumination pattern parameters is a prerequisite for Wiener-deconvolution-based SR-SIM image reconstruction. Here, we present a direct reconstruction SIM algorithm (direct-SIM) with an initial spatial-domain reconstruction followed by frequency-domain spectrum optimization. Without any prior knowledge of illumination patterns and bypassing the artifact-sensitive Wiener deconvolution procedures, resolution-doubled SR images could be reconstructed by direct-SIM free of common artifacts, even for the raw images with large pattern variance in the field of view (FOV). Direct-SIM can be applied to previously difficult scenarios such as very sparse samples, periodic samples, very small FOV imaging, and stitched large FOV imaging.