Terrain classification is a crucial research direction in remote sensing imagery. The technology of joint hyperspectral images and LiDAR data classification has drawn much attention in recent years. The classification performance of existing deep learning models significantly depends on the richness and quality of labeled samples, which often poses a major challenge in practical applications. In addition, many models fail to effectively utilize the information complementarity between hyperspectral images and LiDAR data. To solve the above problems, this study proposes a semi-supervised double-branch classification network with cross-modal channel weight adjustment. Through the attention mechanism, the similarity between two data channels is analyzed deeply, and the weight of each channel is adaptively adjusted accordingly. At the same time, the semi-supervised method of consistency regularization and pseudo-labeling is combined to effectively utilize the information of unlabeled samples. In the experiment of joint classification of hyperspectral images and LiDAR data on the two iconic joint datasets of Houston and MUUFL, the proposed method shows significant advantages over existing classification models, effectively improving classification accuracy and efficiency.