1. Introduction

According to statistics, China's citrus production in 2023 was about 45.2713 million tons, an increase of 9.4% year-on-year [1]. This growth trend shows the continuous development of the Chinese citrus industry and the increase in market demand. Under the huge output, how to use image processing technology to accurately distinguish the mature citrus is one of the current challenges. Therefore, Dairath [2] et al. developed a robotic picking and grading prototype system that integrates robotic picking and grading process based on fruit quality recognition. Rajasekar [3] et al. used deep learning techniques and OpenCV library to realize real-time stream detection of fruits. Diego [4] et al. used the Open CV library in Python to extract the pixel area of the fruit image by Otsu technology to identify the ripening state of passion fruit. Based on this, this work aims to study how to accurately identify citrus in complex situations.

2. Establish Citrus Recognition Model

2.1. Introduction to the dataset

This data set is all from the real orchard beat, and by observing the citrus data set collected by the direct orchard beat, it can be found that there are two kinds of orange yellow and cyan. Part of the data is shown in Figure 1 below:

Figure 1. Partial data presentation

2.2. Citrus recognition model based on improved HSV and morphological method

It is worth noting that most branches and leaves in the image are also green, which has a great impact on the count of immature citrus, so the interference of leaves needs to be eliminated in the future. In addition, due to illumination and other reasons, the local part of some oranges is in highlight, which also affects the count of oranges. Saini [5] et al. proposed a classifier based on deep Convolutional neural Network (DCNN) to make correct disease prediction at an early stage. Based on this, it can be seen that OpenCV has great universality in the field of image recognition of fruit picking robots. Therefore, this paper establishes the citrus counting model in complex situations based on OpenCV and solves and analyzes it. Firstly, this paper establishes a model that only considers yellow citrus, that is, when only yellow citrus is identified. On this basis, cyan and yellow citrus are considered at the same time by modifying the HSV range and morphological method. The above methods divide the research of this paper into three parts, and the recognition accuracy of the three method stages is constantly rising.

2.2.1. Consider only yellow oranges

First, the image is loaded as a multi-dimensional array in BGR (Blue, Green, Red) format through python. Each element of this array represents a pixel, and each pixel contains the intensity values of the three-color channels. Then, the image is converted from RGB to HSV color space. Since the transformation formula of H is determined by the value range of S and V, only S and V are shown in this paper, and the transformation formula is as follows.

(1)

HSV (Hue, Saturation, lightness) space is more suitable for dealing with color-related problems, it separates color information (hue and saturation) from brightness information. In HSV space, the color of yellow objects, such as citrus, is easier to recognize and isolate. The specific conversion process involves converting BGR values to HSV values through a series of transformation formulas. Next, we define the range of yellow in HSV space and create a mask that only keeps colors within the defined range (i.e., yellow), setting yellow areas to white (value 255) and other areas to black (value 0).

(2)

(3)

where

denotes the value of the pixel value at position on the ith channel,

and

are the lower and upper bound values of the ith channel, respectively.

In this paper, morphological operations, namely dilation and erosion, are applied to the mask to improve the accuracy of recognition. The dilation operation increases the size of the white area in the image by using a specific structuring element. The formula is as follows:

(4)

The structuring element, kernel, defines the size and shape of the dilation operation, in this case a 5x5 matrix. The centers of the structuring elements correspond to the pixels to be considered in the original image. For each pixel position

, the pixel value of the output image is calculated by aligning the structuring element with the neighborhood of that point and then taking the maximum value of the pixels at the positions corresponding to the structuring element in the neighborhood (in binary images, 0 means black and 255 means white). If at least one of the structuring elements at positions

in any src image is white (or the maximum value, typically 255), then the pixel at position

is set to white (255) in the output image. If everything is black, we keep it black (0). As a result, the white areas in the image will expand and the black areas will shrink.

After dilation, adjacent yellow regions may fuse together. To reduce this effect, we use the following formula for erosion:

(5)

If the structuring element at pixel of position in the fully covered src image is all white (or at the maximum value, typically 255), then the pixel at position in the output image will be set to white (255). If any of them is black, then that point in the output image will be set to black (0). The result is that white areas in the image will shrink and black areas will expand.

Erosion reduces the size of the white area, thus separating objects that are stuck together by the expansion. After obtaining the processed mask, the image is defined as in this paper, where represents white points and 0 represents black points. Then the contour can be regarded as such a point set. On this basis, the following formula is used to find the contour of all white regions in the mask:

(6)

This step identifies and returns the contours of all the individual objects in the image, where each contour corresponds roughly to a citrus.

Finally, by counting the number of contours, the amount of citrus in the image can be estimated. This count is based on the assumption that each independent yellow region represents a citrus. See Figure 2 below:

Figure 2. Quantitative distribution of citrus

Figure 2 shows how OpenCV can identify and count specific objects through color recognition and image processing techniques in the image recognition task of citrus by using color space transformation, color threshold setting, morphological operations, and contour detection. Together, these steps constitute a complete image recognition flow, from loading and preprocessing of images to identification and counting of objects. It is worth noting that citrus is not detected in the figure, which may be due to the occlusion of branches and leaves, light reflection, and the similar color of immature citrus and branches. In order to solve the above problems, this paper continues to improve the algorithm.

2.2.2. Consider both yellow and cyan oranges

This section focuses on increasing the ability to recognize different colors of citrus (yellow and cyan) and reducing the impact of light intensity. Let's examine the improvement in detail:

Step1. Add color range recognition

The original model only recognizes yellow citrus. In this paper, the model is improved to add the recognition of cyan (green) citrus, and the color range condition is appropriately relaxed. The model does this by defining two different HSV color ranges: one for yellow oranges and one for cyan oranges. This improvement allows the model to handle more diverse image data, such as recognizing both yellow and cyan oranges in the same image.

Step2. Mask merging

After creating two separate masks, the model can be improved using the following formula:

(7)

This step merges these two masks into one for image segmentation of the citrus target. In this way, both yellow and cyan citrus regions are represented as white in the final mask, while other regions remain black.

Step3. Morphological operation

Similar to the original model, dilation and erosion operations are performed to improve the quality of the mask. This helps to reduce noise and more clearly separate close citrus.

Step4. contour detection

Contours in the merged mask are detected by cv2.findContours. As in the original model, each contour roughly corresponds to a citrus in the image. Here, this paper chooses circle and oval to approximate the shape of citrus.

Through the above improvements, the model has a significant improvement in the recognition of the number of oranges, and by drawing the histogram of the number of oranges, it is helpful to visualize the distribution of the number of oranges in different images. As shown below:

Figure 3. Quantitative distribution of citrus

From Fig. 3, it is clear that the distribution of citrus is significantly different from Figure 2. The improved model can better identify the number of citrus in each image, reduce the misjudgment of illumination, branch occlusion and other conditions, and is closer to the actual situation. This improvement significantly enhances its functionality and applicability in image processing by adding the ability to recognize citrus with different colors. By incorporating masks of different colors and using morphological operations, image data can be processed and analyzed more efficiently.

2.3. Eliminate the interference of leaves

To address issues with distinguishing citrus from green branches and leaves, the following improvements were made to the model:

Step 1: Color Range Adjustment:

Relaxed the HSV threshold to better capture variations of yellow and cyan citrus, enhancing flexibility and adaptability.

Step 2: Morphological Operation Adjustment:

Reduced the kernel size to 3x3 and the erosion iteration count to 1, refining the operations to preserve citrus shapes and minimize false positives due to excessive dilation or erosion.

Step 3: Contour Analysis Improvement:

Used cv2.fitEllipse to fit ellipses to contours and extracted features including major and minor axis lengths and eccentricity.

Introduced thresholds for area and eccentricity; a contour is considered citrus if its area exceeds a threshold and its eccentricity is below 0.85.

This filtering step helps exclude non-citrus objects, such as leaves, improving recognition accuracy.Part of the processed image looks like this:

Figure 4. Display of image segmentation data

Fig. 4 displays the original and processed images, where the target citrus is highlighted in white against a black background. This simple segmentation technique effectively enhances contrast and supports object detection tasks with distinct color differences. While the method performs well, it may have limitations for objects with varied colors or those closely matching the background, introducing some noise. For more complex color scenarios, stricter constraints or advanced techniques may be needed to refine the segmentation.

3. Citrus recognition results

In this paper, part of the target detection images are randomly selected for display. It can be seen from the fig. 3 that the model can basically identify the naked or most of the exposed mature citrus. However, for the citrus with only a small part of the contour exposed due to leaf occlusion, the model may miss the detection phenomenon, which is because the target is too small and the contour shape is difficult to recognize. It can be seen from the figure that the model can not only identify mature citrus, but also reflect a good recognition effect for immature citrus.

Figure 5. Display of recognition results

Figure 6. Quantitative distribution of citrus

According to the above fig. 6, the citrus counting model based on color threshold can identify the true citrus more accurately and reduce misjudgment through more refined morphological operations and contour feature analysis. Relaxing the color range means that we can handle a wider variety of citrus color variants, increasing its applicability on different image datasets. By setting the area and eccentricity thresholds, the model was able to better distinguish between citrus and other shaped objects, especially in complex backgrounds.

4. Model Evaluation

4.1. Box plots and cumulative distribution functions

Fig. 7 presents a boxplot summarizing the distribution of orange counts in images. The median is around 10, with an interquartile range from 7.5 to 12.5, indicating that most counts (50%) fall within this interval. No outliers are evident, and the distribution is relatively symmetric, suggesting a uniform spread without significant skew. The minimum and maximum counts are near 0 and 20, respectively. This visualization effectively captures the statistical distribution of orange counts.

Figure 7. Box diagram of citrus quantity Figure 8. Cumulative distribution function of citrus quantity

Fig. 8 depicts the cumulative distribution function (CDF) of the number of oranges in the dataset. The CDF shows a gradual increase, resembling a smooth ladder shape, indicating a continuous and even distribution of orange counts. Smaller counts are more frequent, with the CDF approaching 1 as the number of oranges increases from 0 to 20, reflecting the absence of significant breaks or jumps in the data. This suggests a lack of extreme skewness or concentration within a specific range. The CDF plot facilitates the estimation of the probability of any orange count, offering valuable insights into the dataset's variability and uniformity.

4.2. Performance evaluation

Fig. 9 demonstrates the trade-off between precision and recall in the model. Precision remains high over most of the recall range, with a notable decline only at very high recall levels, indicating a good balance between true positives and false positives. Fig. 10 shows that model accuracy increases with prediction confidence until it plateaus. This suggests that highly confident predictions are more accurate, which is beneficial for reducing errors by selecting high-confidence predictions.

.

Figure 9. Precision-Recall Curve Figure 10. Precision-Confidence Curve

The model demonstrates high accuracy and reliability across various conditions. It maintains high precision over a wide recall range, indicating effective identification of positive examples with few false positives. The model's performance is robust under different confidence and recall thresholds, allowing for flexible adjustment to meet specific application needs.

5. Conclusion

In this paper, the task of citrus identification in complex orchards is divided into three steps: only considering mature citrus, considering both mature and immature citrus, and eliminating leaf interference. By continuously improving HSV threshold and introducing morphological and contour features, the model is improved to minimize the interference in leaf occlusion and citrus overlap, so that the model can identify more citrus in complex scenes. Of course, there are still some areas that need to be improved in this paper. The threshold correction of HSV may change constantly due to illumination factors, so we can consider introducing an optimization model to determine the optimal HSV range.