Facial expression recognition (FER) has various applications in human-computer interaction scenarios. However, existing FER methods are not that effective for blurred and occluded expression. To cope with facial expression blur and occlusion, this study proposes a novel network based on local manifold attention (SPD-Attention), which uses manifold learning to obtain the second-order statistical information with a stronger descriptive ability for strengthening the learning of facial expression details and suppressing the influence of irrelevant features in the occlusion area on the network. At the same time, in view of the disappearance and explosion of gradient caused by logarithmic calculation, this study proposes corresponding regular constraints to accelerate network convergence. The effect of the algorithm is tested on public expression recognition data sets, which is significantly improved compared with those of classic methods such as VGG. The accuracy is 57.10%, 99.01%, 69.51%, 87.90%, 86.63%, and 49.18% on AffectNet, CK+, FER2013, FER2013plus, RAF-DB, and SFEW, respectively. In addition, compared with state-of-the-art methods such as Covariance Pooling, the proposed method has an accuracy improved by 1.85% on a special blurred and occluded expression data set.