Linear optical processors based on the cascaded topology of Mach-Zehnder Interferometer (MZI) have been demonstrated to be an important way of implementing Optical Neural Networks (ONN), but several practical challenges still need resolution. Concerning issues arising from chip manufacturing and testing processes that could lead to phase errors and insertion losses, we conducted experiments and theoretical simulations for various reconfigurable optical processors. We found that the weights of any arbitrary unitary matrix can be realized through some single N×N Clements units, that can substantially reduce the optical depth and enhance robustness against insertion losses. This approach allows for the construction of fully complex optical neural networks. Additionally, In multi-layer ONN, due to the limited degrees of freedom in constructing this arbitrary matrix, we introduced a phase-shift layer before each layer of the Clements unit. This design aids in mapping classification data to higher-dimensional spaces, facilitating faster neural network convergence.