Hyperspectral images (HSIs) contain rich spectral information and spatial information. How to apply these two information types and fully combine the correlation between them remains a challenge worthy of further research and discussion. In this study, a multi-branch-multi-scale residual fusion network (MB-MS-RFN) for HSI classification is proposed. First, a 3D multi-branch-multi-scale convolution residual network, which can acquire image features of different scale in the training process and consider the correlation between spectral information and spatial information, is developed. Instead of deepening the network, the multi-branch structure widens the network horizontally to obtain more accurate classification. Finally, the different levels of HSI features are fused to obtain better classification results. Several experiments have been carried out to verify the proposed framework, and the results have demonstrated that the proposed MB-MS-RFN framework can improve the classification performance of HSIs. The performance of the MB-MS-RFN was evaluated using the Indian Pines, Pavia University, and Kennedy Space Center datasets; the performances’ overall accuracies were 99.66%, 99.92%, and 99.97%, respectively. The results from a series of experiments confirm that the proposed method offers several advantages in classification accuracy compared with five other methods.