Phase-shifting interferometry is a widely used technique in precision optical metrology. However, conventional PSI methods typically require three or more phase-shifted interferograms, which limits their applicability in dynamic measurements and vibration-sensitive environments. To address this issue, this paper proposes a deep learning-based framework named PSI-IPENet. The framework adopts a Two-to-One structure, where two phase-shifted interferograms are used as dual-channel input, corresponding to a single phase map as the supervisory signal, and a dedicated dataset is constructed for training. PSI-IPENet leverages the feature extraction capability of IPENet and incorporates the physical characteristics of interferometric imaging, thereby enhancing the robustness and noise resistance of phase retrieval. Experimental results demonstrate that the proposed method maintains high-precision phase recovery performance even with a low number of input frames. Compared with the traditional four-step phase-shifting method, it shows significant advantages in terms of signal-to-noise ratio and phase error metrics.