The deep learning-based algorithms of action recognition are often difficult to achieve fast performance and high accuracy due to the complexity of neural networks. In view of this, we modularize the existing temporal shift and split attention module as an end-to-end trainable block which can be easily plugged into the classical two-stream action recognition pipeline. In the RGB and optical flow branch network, we adopt a random sampling strategy with sparse temporal grouping to realize long-term modeling. Furthermore, we use the Temporal Shift module to replace some channels in the time dimension so as to enhance the sequential characterization ability with information of adjacent frames. In addition, the Split Attention module integrating multi-paths and feature map attention mechanism improves the recognition performance of the network. Experiments show that our method achieves appealing performance on two public benchmark datasets including UCF101 (recognition accuracy of 95.00%) and HMDB51 (recognition accuracy of 72.55%), demonstrating its effectiveness.