Automatic segmentation of retinal vessels facilitates the diagnosis and treatment of retina-related diseases. The task is challenging because of the complex structure of retinal vessels and interferences in fundus images, such as low contrast, uneven illumination, and pathological exudates. To address the lack of global semantic dependency modeling and the semantic gap between encoders and decoders in the U-Net for this task, an Intra- and inter-scale augmented U-Net (I2A-Net) is proposed. I2A-Net is designed based on two perspectives: for the intra-scale encoding-decoding layer, a Spatial Enhanced Self-attention mechanism is integrated in each coding layer to enhance the global spatial aggregation capability, and further developed into the decoder to alleviate the information loss caused by up-sampling operations; For inter-scale encoding-decoding layer, a novel Cross-scale Fusion module is introduced to boost the semantic interaction with other layer by dynamically leveraging the rich feature information of the deepest layer, thus further alleviating the semantic gap between encoder and decoder. The experiments on three retinal standard datasets, DRIVE, CHASE_DB1 and STARE, demonstrate that I2A-Net can effectively segment the retinal vessel structure, and compared with Baseline, I2A-Net can yield superior performance in all evaluation metrics.