Aiming at the problem that the single feature extraction algorithm has more problems in the detection of the athletes, this paper proposes a sports video player detection model with middle feature block classification. Firstly, the mid-level feature block is used to describe the characteristics of the athlete. Then, the SLIC algorithm is used to divide the pixel, and the 5-D feature of the pixel is constructed by using the CIELAB color space and the XY spatial coordinates. Finally, the Gaussian component is used to establish the foreground Background pixel description model to improve the detection accuracy. The simulation results show that the improved model proposed in this paper compares the HOG algorithm and SVM algorithm, and the detection result shows the athlete area more accurately.