Recent advancements in artificial intelligence (AI) have shown promising potential for the automated screening and grading of cataracts. However, the different types of visual impairment caused by cataracts exhibit similar phenotypes, posing significant challenges for accurately assessing the severity of visual impairment. To address this issue, we pro-pose a dense convolution combined with attention mechanism and multi-level classifier (DAMC_Net) for visual im-pairment grading. First, the double-attention mechanism is utilized to enable the DAMC_Net to focus on le-sions-related regions. Then, a hierarchical multi-level classifier is constructed to enhance the recognition ability in dis-tinguishing the severities of visual impairment, while maintaining a better screening rate for normal samples. In addi-tion, a cost-sensitive method is applied to address the problem of higher false-negative rate caused by the imbalanced dataset. Experimental results demonstrated that the DAMC_Net outperformed ResNet50 and dense convolutional network 121 (DenseNet121) models, with sensitivity improvements of 6.0% and 3.4% on the category of mild visual impairment caused by cataracts (MVICC), and 2.1% and 4.3% on the category of moderate to severe visual impair-ment caused by cataracts (MSVICC), respectively. The comparable performance on two external test datasets was achieved, further verifying the effectiveness and generalizability of the DAMC_Net.