Automated sorting of agricultural products requires image-classification methods that combine reliable predictive performance with computational efficiency for deployment on resource-constrained devices. This study develops a surrogate model-based image-processing framework to classify chili pepper (Capsicum annuum L.) products and seeds and evaluates its predictive accuracy, surrogate fidelity, and deployment efficiency. An initial dataset of 1,240 images representing green chili, fresh red chili, dried chili, and seeds was collected; after excluding 17 low-quality images, 1,223 images were analyzed. Image preprocessing and feature extraction generated a 14-dimensional representation incorporating color, texture, and morphological characteristics. MobileNetV3-Small served as the high-fidelity reference classifier, while Kriging, radial basis function (RBF), and polynomial response surface methodology (RSM) were evaluated as computationally efficient surrogates. The reference convolutional neural network achieved 97.1 ± 0.6% accuracy, compared with 95.8 ± 0.9%, 93.6 ± 1.0%, and 91.3 ± 1.2% for Kriging, RBF, and RSM, respectively. Kriging achieved the highest surrogate fidelity (R² = 0.94) while reducing inference latency from 46.2 to 5.4 ms/sample, an 8.6-fold improvement. Multi-objective optimization identified non-dominated accuracy–latency configurations for different computational constraints. The results demonstrate that surrogate selection depends on the required balance between predictive performance and computational resources. The proposed framework provides a practical basis for efficient, interpretable, and edge-deployable image classification in smart agricultural sorting systems.