Different from ordinary object detection tasks, the difficulty of detecting tile surface defects lies in the detection of unconventional size objects, such as small-sized objects and objects with large aspect ratios. To solve these two problems, this study proposes a new type of tile surface defect detection algorithm based on improved Cascade R-CNN. To improve the detection ability for small defects, the model in this study uses the lateral connection structure to fuse the semantic information of the upper and lower layers and applies the dilated convolution with switchable dilation rates to increase the receptive field of the model. To improve the detection ability for defects with large aspect ratios, the proposed model introduces an offset field on the standard convolution to better extract the object feature information. In addition, the model adjusts the size and length of the pre-selected anchor box in the Cascade R-CNN framework. The experimental results show that on the dataset collected from the tile factory, the mean average precision (mAP) of the proposed algorithm reaches 73.5%, which is 9.7% higher than that of the Cascade R-CNN model before improvement. The experimental code of this study is available at: https://github.com/mashibin/Ceramic-tile-defect-detection.