High-speed single-carrier transmission can be yielded by increasing the modulation format cardinality for higher spectral efficiency. However, ultra-high-order QAM signals usually are more susceptible to various impairments. Hence, we propose a temporal feature-based memory (TFM) neural network (NN) equalizer to effectively mitigate signals’ impairments in ultra-high-order QAM. The temporal convolutional network is utilized as feature extraction layer to significantly improve performance of the bi-directional long short-term memory network. The TFM-NN equalizer was experimentally validated in a probabilistically shaped polarization-division multiplexed (PDM) 1024/4096-QAM coherent optical transmission systems and raw spectral efficiencies of 16.190 and 21.188bit/s/Hz have been achieved at NGMI thresholds.