Predicting the growth trajectory and yield of greenhouse strawberries based on knowledge-guided computer vision

Monitoring and modeling the growth of strawberries at the individual fruit level can open up new opportunities for yield prediction, fruit grading and supply chain optimization. However, existing strawberry growth models mainly focus on plot or plant level and can not simulate the growth of individual fruits, and existing computer vision (CV)-based studies primarily focus on instant tasks but lack the reasoning capabilities required for dynamic growth simulations. In this study, we developed a novel knowledge-guided CV framework, named KGCV-strawberry, to simulate the growth of strawberry fruits on an individual basis. We conducted two consecutive years of greenhouse experiments with intensive measurements to develop and test the framework. The KGCV-strawberry framework consists of two components: first, the fruit trait detector (acting as the “eye”) that interprets bounding boxes and biophysical traits of individual fruits from raw images, and second, the fruit growth simulator (acting as the “brain”) that uses the estimated traits to predict fruit growth. We employed a hybrid training approach for KGCV-strawberry, where the fruit trait detector was trained by ground observations and the fruit growth simulator was trained by synthetic data generated by the S-shape fruit growth curves. The KGCV-strawberry is designed to be able to dynamically assimilate observations (e.g., image sequences) such that the fruit growth simulator infers growth curve parameters from fruit size sequences. We tested the KGCV-strawberry by ground fruit trait measurements, with a case study showing the RMSE of diameter estimation decreased by 74 % as the sequence of observations expanded from 1 to 6. For the yield prediction task, we observed a reduction in the RMSE from 3.58 to 2.01 g and an increase in R2 from 0.25 to 0.73 as more images were assimilated into the framework. Additionally, the RMSE for predicting the remaining growing degree days (GDD) until maturity saw a significant reduction from 71.87 °C·day to 39.30 °C·day, accompanied by an increase in R2 from 0.11 to 0.61. Although the best prediction is achieved near maturity, the prediction accuracy is acceptable two weeks before fruit maturity. Additionally, we conducted a comparison between KGCV-Strawberry and a process-based model for predicting plant-level yields. KGCV-Strawberry exhibited superior performance in capturing yield dynamics for each harvest. These findings highlight the potential of applying this framework for precise management optimization of individual fruits in intelligent strawberry farming.