Traditional fire warning methods have low detection accuracy and cannot give early warnings in time before the fire starts. Therefore, this study proposes an early fire warning algorithm based on deep learning. Firstly, an infrared thermal imager is used to collect infrared images in a specific scenario for dataset construction. Secondly, the improved YOLOv4 algorithm is applied for training, and the network weights are obtained. The convolutional attention module is introduced after the three output feature layers of the backbone network to improve the ability of the network to extract key information. Convolutional layers are added to the backbone network and path aggregation network to promote feature extraction capability. Finally, the proposed intelligent fire detection (IFD) algorithm is employed to process the predicted image and evaluate the fire hazard according to the score. The experimental results reveal that the mAP of the improved YOLOv4 algorithm on the dataset reaches 98.31%, which is 2.7% higher than that of the original YOLOv4 algorithm, and the FPS is 37.1 f/s; the accuracy of the IFD algorithm is 93%, and its false detection rate is 3.2%. The proposed early fire warning algorithm has the advantages of high detection accuracy and timely warnings when there is no fire.