An unconstrained frequency-domain equalization (FDE) based on multi-input multi-output algorithm is proposed for demultiplexing of the 6×6 mode-division multiplexing (MDM) system. It is used for eliminating the effects of mode coupling and differential mode group delay (DMGD) on signal. To verify the effectiveness of unconstrained FDE for demultiplexing, the unconstrained frequency-domain least mean square (FD-LMS) algorithm and the unconstrained frequency-domain constant-modulus algorithm (FD-CMA) are used for the MDM system. Besides, the equalization performance of the unconstrained FD-LMS algorithm and the unconstrained FD-CMA are compared with the constrained FD-LMS algorithm and the constrained FD-CMA. Simulation results show that the demultiplexing performance of the unconstrained FDE is comparable with that of the constrained FDE. However, the unconstrained FDE has great advantage over the constrained FDE in computation complexity.