Considering that fire warning in oilfield operation sites depends on manual inspection and cannot be realized in time, this study proposes an improved YOLOv4 fire detection algorithm. Specifically, due to the long distance between the camera and the fire target, fires are too small to be identified. Given this problem, the network feature fusion is improved and the pyramid convolution (PyConv) is added to enhance the detail extraction ability and increase the local receptive field. In response to the complex background interference in the oilfield operation sites, the attention mechanism is adopted to strengthen the network’s ability in weight calculation of important features, reducing non-critical data calculation. Finally, the anchor boxes of target samples are optimized through a clustering algorithm, and the self-built fire dataset is used for experiments. The experimental results prove that the improved algorithm model has quite good performance, has a mean average precision (mAP) of more than 90%, and can maintain a high recognition rate for small pyrotechnic targets in complex background. It shows that the improved algorithm has high practical value in pyrotechnic recognition in oil field operation site.