1. Introduction

New technologies are springing up and subtly changing people’s lives, especially the development and the use of Internet technology. The way people consume has changed, and a series of integrated services such as online shopping and online payment have been realized and ushered in the development of the opportunity. According to China Internet Network Information Center [1], as of December 2023, the scale of China's online shopping users reached 915 million, an increase of 69.67 million from December 2022, accounting for 83.8% of the overall number of Internet users; and, the national online retail sales amounted to 15.4 trillion yuan in 2023, an increase of 8.4% from the previous year, with the growth rate accelerated by 2.2% from the previous year. Therefore, the huge economic impact of online shopping on individuals, society, and even the whole country can no longer be ignored, and personalized recommendation systems, which play a crucial role in online shopping, are also very much up for discussion.

However, the sheer size and the plethora of numbers can sometimes make it difficult for consumers to inch their way through them. In traditional economic theory, "the more, the better" is an idea that has long been considered reasonable. However, research has shown that it may no longer be true when the set of choices becomes too large, referred to as choice overload [2, 3]. In a situation of choice overload, people tend to be confused, hesitant, or even ultimately abandon their choices in response to the plethora of choices offered, and that’s certainly not the result merchants want. Therefore, in order to mitigate the effects of choice overload, we should first know exactly what causes it. When there are no strong prior preferences or dominant options in the item set, a larger choice set may only lead to decreased satisfaction and increased choice delay [4]. Selection from an item set may also become more difficult when items do not vary much in attractiveness [5]. Thus, Scheibehenne et al. [4] conclude that the paucity of measures to control for similarities between programs and the difficulty of trade-offs may be one reason why studies of selection overload show such varied results. To alleviate this overload, a recommendation system is believed as a helpful tool [6].

One of the advantages of a personalized recommendation system is that it can be optimized by data collection and algorithms to provide better choices compared to the choice set without effective control. Recommendation systems use the user’s offline data information to tap into the user’s personal preferences and behavioral habits so as to provide accurate services to the user. Typically, offline data is derived from the following sources: the user's personal information and the user's explicit and implicit preference values. Explicit, which is the direct scoring of the user, is easier to obtain and can be utilized in addition to the influence of the user's personal subjective factors. The implicit preference value is generally calculated by the system and combined with the user's personal behavior records, such as collection, click or add to cart, purchase, and a series of behaviors, implicit preference without the influence of subjective assessment factors, and the effect is better. The recommendation engine, after obtaining the corresponding data information, can be calculated by the recommendation algorithm and quickly filter the recommended results.

In this paper, we will analyze the data generated by the three questionnaires we have designed: one is the control group, one shows the significant effect caused by choice overload in the field of online shopping, and one explores whether personalized recommendation systems actually contribute to the choice overload effect. The inter-comparison of these three sets of experimental data can highlight the existence of choice overload behavior in online shopping, and it helps us to summarize the improvements that personalized recommendation systems can make and the benefits to merchants when consumers are subject to choice overload.

The paper will be followed by a literature review examining past information on choice overload and personalized recommendation systems, and then will describe our experiments and analyze the data and the results. Finally, we will draw conclusions.

2. Literature review

There has been a long history of research on choice overload, and Iyengar & Lepper [2] are the pioneers whose experiment laid a firm foundation for the research on this phenomenon. In their study, participants are less likely to make a purchase when presented with a large number of jam flavors (24 choices) compared to those with more limited choices (6 choices). The study suggests that while people are initially attracted to more options, they are eventually overwhelmed by them. This confirms that when consumers are faced with more choices, they are less satisfied with their chosen option and, thus, more likely to fail in the final choice. The phenomenon is known as “choice overload”, or the “paradox of choice”, and is first mentioned by psychologist Barry Schwartz in his book The Paradox of Choice: Why More Is Less [3]. This seminal work argues that excessive choice leads to anxiety, regret, and a decline in the overall well-being of the person making the choice. Supporting his conclusion is Herbert Simon’s [7] notion of “bounded rationality”, meaning that human beings have difficulty dealing with a relatively large number of alternatives given their limited cognitive resources. Exploring the various factors that influence choice overload is a topic that subsequent studies have focused on. For example, the complexity of the options, the individual’s decision-making style, and the perception of the importance of the decision all play a crucial role. Reutskaja and Hogarth [8] found that when choices are complex, the probability of choice overload increases, leading to delayed decisions or default choices. In addition, Schwartz et al. [9] highlight the role of individual differences, noting that “maximizers” (those who pursue the best outcome) are more prone to choice overload than “satisfiers” (those who are satisfied with a good enough alternative).

Similarly, with the development of science and technology, studies on personalized recommendation systems have emerged one after another. There are some early models are used in the system, such as collaborative filtering, rely on the assumption that users with similar past behaviors will have similar preferences in the future [10]. Content-based filtering is another foundational approach, which recommends items similar to those the user has liked in the past based on attributes such as keywords, categories, or features [11]. Hybrid systems, which combine collaborative and content-based methods, have been developed to overcome the limitations of each approach, such as the cold-start problem [12]. The development of big data and computer technology has injected strong momentum into the improvement of personalized recommendation systems. For example, the matrix factorization technique, popularized by the Netflix Prize competition [13], has been a breakthrough in collaborative filtering techniques by reducing the dimensionality of the user-item interaction matrix to achieve more accurate predictions. Moreover, this technique is enhanced by integrating implicit feedback, temporal dynamics, and contextual information [14]. Another example is the rise of deep learning techniques. Neural networks, especially deep neural networks, have been used to capture complex user-item interactions and model non-linear relationships [15]. Another significant advancement is the incorporation of reinforcement learning, where recommendation systems learn optimal strategies by receiving feedback from user interactions in real time [16]. This dynamic learning approach is particularly useful in environments where user preferences evolve rapidly.

While choice overload and recommendation systems are widely discussed, the discussion focus on their relationship is rather limited. Bollen et al. [17] have studied it, but only at the level of association, and the economic impacts that it can cause have been ignored.

3. The experiment and data collected

3.1. Survey design

This comprehensive survey is comprised of three distinct questionnaires, each meticulously crafted to align with the specific needs of the control group, treatment group 1, and treatment group 2, respectively. The first section of each questionnaire delves into personal demographic information, seeking to gather insights through a series of questions that explore various aspects of the participants' backgrounds. These questions touch upon fundamental data points such as age, gender, and frequency of engagement with recommendation systems in their daily consumer activities.

To illustrate the details and effects of these questionnaires, let's consider an example. In the personal questions section, a participant might be asked to indicate their age range (e.g., 18-24, 25-34, etc.), gender (male, female, or non-binary), and how often they use recommendation systems when shopping online (e.g., daily, weekly, rarely). These responses provide a foundational understanding of the participant's demographics and their level of familiarity with recommendation systems.

Moving forward, the second part of the questionnaire constructs a simulated shopping scenario, inviting participants to envision themselves browsing for a new T-shirt. In the first questionnaire, the participant is presented with a limited selection of five T-shirt designs, each with unique features such as color, pattern, and style. This limitation is intentionally imposed to evoke a sense of focused decision-making, as participants must weigh their preferences within a constrained set of options.

As for the second questionnaire, the expansion of options to twenty-five T-shirts creates a scenario of decision overload. The participants may find themselves overwhelmed by the sheer number of alternatives, struggling to compare and contrast each option effectively. This can lead to increased cognitive load and potentially prolong the decision-making process.

Finally, the third questionnaire introduces a comprehensive data-driven recommendation system tailored to the participant's preferences based on their initial responses and interactions with the first four T-shirt options. This system suggests a personalized selection of T-shirts from the larger pool of twenty-five options, aiming to streamline the decision-making process by presenting relevant and appealing choices. The effectiveness of this system can be evaluated by measuring the participant's satisfaction with the recommended options, their willingness to engage with the recommendation system in the future, and the overall improvement in the shopping experience.

By incorporating these details and effects into the description of the survey, we provide a more nuanced and comprehensive understanding of the questionnaires and their potential impact on consumer decision-making.

3.2. Data and variables

Table 1 below shows the basic data collected from the three questionnaires. We suggest that “gender” = 1 when the answer is male, “gender” = 0 when the answer is female, the “age” = 1 if larger than 25, otherwise “age” = 0 ;if people think Taobao’s recommendation systems is effective, “Whether Aware of Taobao's Recommendation”=1, otherwise equals to 0; if people choose the first four clothes in the questionnaire, “Which Clothes” = 1, otherwise equals to 0; if people choose level 1 or 2 when considering how difficult to choose the clothes, “Which Clothes” = 1, otherwise equals to 0.

The mean, standard deviation, minimum and maximum values of factors such as gender, age, understanding of Taobao recommendations, and clothing selection exhibit distinct variations. Particularly in the "Wide selection" section, the newly added data on "Wide selection and recommendation" further emphasizes its uniqueness, potentially indicating a significant impact of the recommendation system on user choices. The distribution and disparities within the data may reflect behavioral distinctions among different user groups in the Taobao recommender system, thereby holding substantial implications for comprehending and optimizing this recommender system.

3.3. Discuss difference of the control group and treatment group

3.3.1. The difference in gender between control group and treatment group 1

From Table 2, we can obtain:

One-tailed test: Despite the p-value being close to 0.05, it slightly exceeds this threshold, indicating that we cannot reject the null hypothesis with 95% confidence, suggesting no significant difference in means between the two groups.

Two-sided tests: All p-values significantly surpass 0.05, further supporting the notion that there is no statistically significant distinction between the two groups.

Although the mean of the treatment group appears higher than that of the control group, this disparity fails to reach statistical significance according to t-test results.

3.3.2. The difference in age between treatment group 1 and treatment group 2

As shown in Table 3, the average for treatment group 1 is 0.741, while for treatment group 2, it is 0.694. Based on the average values, treatment group 1 exhibits a higher value compared to treatment group 2. Standard Deviation for treatment group 1 is calculated as 0.1989 and for treatment group 2 it is slightly larger at 0.2133, indicating that the data points in group 2 are more dispersed around the mean value. The one-tailed p-value (P(T<=t)) obtained is found to be equal to 0.2218, which exceeds the common significance level of α=0.05; thus we fail to reject the null hypothesis suggesting insufficient evidence that treatment group 1 has a significantly higher mean than treatment group two under this test condition."

Summary: "Although there was a slight difference observed in the mean values between both groups, these differences were not statistically significant according to our t-test results analysis; hence implying that any potential variation between both groups might not be attributed solely to age or other analyzed variables."

4. Main results

Based on the data collected from the three questionnaires, we mainly focused on the scenario setting part in the questionnaire survey and made further statistical analysis .

In the scenario setting part, we set two questions: one is for participants to choose their preferred clothing options for purchase; another is about how difficult it was to make this choice.

Therefore, we produced two statistical analysis output charts separately for product selection and selection difficulty.

4.1. Focus on the data of product selection

In this regression statistics table, the following data information is presented.

Notes: all the above values have 5 significant figures

Table 4 shows the coefficients and P values of the dependent variable "product selection" under different choice scenarios.

As shown in table 4, the intercept term, indicating the dependent variable value when treat1 and treat2 are both 0. The coefficient of "Intercept" is 0.84659 and the P-value is 3.7189, which means that when all other variables are zero, the probability that people would choose to recommend a T-shirt is 0.84659. Since the P-value is very small, much less than 0.05, this indicates that this coefficient is statistically significant, makes people tend to choose the recommended T-shirt.

The coefficient of "treat1" was -0.53813, and the P value was 1.6693, indicating a significant negative correlation between not using big data recommendations (treat1) and product selection which means the treatment related to "treat1" also significantly reduced the probability of people choosing to recommend T-shirts.

The coefficient of "treat2" is -0.44368, and the P value is 1.0664, indicating a significant negative correlation between using big data recommendations (treat2) and product selection which means the treatment related to "treat2" will reduce the probability that people will choose to recommend T-shirts. Again, this effect is statistically significant due to the very small P-value.

Since the coefficient of treat2 (-0.44368) is smaller than the coefficient of treat1 (-0.53813), we can conclude that the negative impact of using big data recommendation (treat2) on product selection is smaller than that of not using big data recommendation (treat1).

To summarize briefly, in control group, there are 84.66% people buy the first 4 T-shirts . In treatment group 1 (choice overload without recommendation), there is a reduction by 53.8% which means there are 30.85% people buy the first 4 T-shirts now. In treatment group 2 (choice overload with recommendation), there is a reduction by 44.37% which means there are 40.29% people buy the first 4 T-shirts now.

In conclusion, recommendation reduces the negative behavior experienced during choice overload,so the merchants can benefit from letting people choose the recommended products by big data recommendation systems.

4.2. Focus on the data of selection difficulty

In this regression statistics table, the following data information is presented.

Notes: all the above values have 5 significant figures

Table 5 shows the coefficients and P values of the dependent variable "selection difficulty" under different choice scenarios.

As shown in table 5, the coefficient of "Intercept" is 0.63068 and the P-value is 1.8944, which means that when all other variables are zero, the probability that one would think that choosing clothes is very easy is 0.63068. Since the P-value is very small, much less than 0.05, this indicates that this coefficient is statistically significant.

The coefficient of "treat1" was -0.17794 and the P value was 0.0005, also indicate a significant negative correlation between using big data recommendations (treat1) and selection difficulty. This suggests that when choice overload, treatment without a recommendation system also significantly reduced the probability that people would think that choosing clothes was very easy.

The coefficient of "treat2" is -0.16951and the P value is 0.0009 that shows a significant negative correlation between using big data recommendations (treat2) and selection difficulty, which indicates that when choice overload, the treatment with a recommendation system still reduces the probability that people think that choosing clothes is very easy. Since the P-value is very small, this effect is statistically significant.

The coefficient for treat2 (-0.16951) is slightly smaller than that for treat1 (-0.17794), indicating that statistically, using big data recommendation system (treat2) has a slightly smaller impact on selection difficulty than that of no recommendation (treat1).

To summarize briefly, in control group , there are 63.07% people think choosing T-shirts is easy. In treatment group 1 (choice overload without recommendation), there is a reduction by 17.80% which means there are 45.27% people think choosing T-shirts is easy now.In treatment group 2 (choice overload with recommendation), there is a reduction by 16.92% which means there are 46.15% people think choosing T-shirts is easy now.

In conclusion, recommendations cannot greatly reduce the difficulty of making choices during choice overload, but it does exist improvement, so the consumers can’t actually benefit from big data recommendation systems.

5. Discussion & implication

5.1. Discussion

From the survey we mentioned, many consumers prefer to buy the product with a recommendation. We suspect that this is because the products recommended by personalized recommendation systems are often those in line with consumers’ consumption tendencies and interests. By analyzing the behavior and preferences of consumers on various online retail platforms, including their historical purchase data concerning item types, styles, and pricing, a sophisticated big data-driven recommendation system is capable of identifying potential duplicate purchases and suggesting those items to consumers [18]. The technologies involved facilitate highly precise and efficient content delivery, which can help consumer to make choice easily. Accurate recommendations can reduce the choice overload of consumers.

Although we found this phenomenon, our experiment also has significant limitations. Based on the survey data acquired, a majority of participants opted for the initial choice. With a quintet of options in the control group, the initial selection was predominant. Similarly, within both treatment groups, the initial options remained the preferred choice among participants. Subsequent to the integration of the recommendation system, there was a noted upsurge in the number of individuals selecting the initial four options; however, it was challenging to ascertain whether this outcome was attributable to the recommendation system or if it stemmed from the positioning of the options.

Fortunately, we were assigned treatment 1 of the same style of garments as treatment 2, ensuring that treatment 1 maintains a consistent sequence. Both sets of recommendations can be observed; the group that provided recommendations tended to choose the first four items 10% more frequently than the group that did not provide recommendations.

Consequently, the establishment of preferred options within the initial quartet of selections, despite acknowledging the potential for associated issues, is justified by the capability to replicate authentic shopping scenarios. Generally, items suggested by sophisticated big data recommendation systems are showcased on the homepage of e-commerce platforms, thereby increasing the likelihood of customer purchases.

The second limitation pertains to the temporal aspect of the questionnaire response. We utilize the duration taken to complete the questionnaire as a criterion to assess the complexity of choice. For instance, an extended completion time for the entire questionnaire implies greater difficulty in subsequent clothing selection. It is postulated that the initial half of the questionnaire contains identical content.

Nonetheless, it is imperative that we do not limit our analysis to the mere recording of the time elapsed during individuals' engagement with the clothing selection process. Consequently, quantifying the complexity of a decision merely based on temporal metrics may result in certain inaccuracies.

5.2. Implication

Based on the advantage of recommendation systems, some producers use this function to let their own product on the first page when consumers click on the online shopping platform. The big data recommendation can reflect some good feedback on this product; it also stimulates the consumers to purchase. The product with recommendations becomes easier for consumers to buy. The sales volume of this product and online stores will continue to rise, thus maximizing the profit of producers. That would have been good, but sometimes some products recommended by big data will have some false marketing behaviors; for example, merchants may hire some people to create some false praise and good feedback to create some very high sales, but in fact, this information is not true [19]. But they also use this information to motivate consumers to make purchases. So that the goods they receive are not so high quality and do not meet their expectations. Therefore, it will lead to a very poor shopping experience for some consumers after receiving the goods. They think they have been cheated, leading to a breakdown of trust between businesses and consumers. We believe that this distrust of personalized recommendation systems is one of the reasons why the results of our experiment are not as significant as we expected.

In this regard, the state administration should exert oversight over the online retail sector. In instances where a vendor engages in widespread dissemination of fraudulent endorsements, the government, along with pertinent regulatory bodies, will initiate legislative actions to mitigate the plight of defrauded consumers [19]. Nonetheless, the complete eradication of fabricated reviews remains a challenging and intricate endeavor.

6. Conclusion

The in-depth analysis of the survey data has clearly brought to light the substantial impact that the big data recommendation system has on the everyday shopping experiences of consumers. This system has proven to be highly effective in reducing the amount of time and choice overload that on the process of selecting products, thereby streamlining their shopping experience. Furthermore, it has significantly alleviated the burden that consumers often face due to the overwhelming number of choices available in modern markets.

In this contemporary era, which is characterized by the swift pace of technological advancement and the widespread dissemination of big data, we find ourselves equipped with the capability to provide individuals with convenient and efficient services. This is achieved through the prudent and strategic application of these technological advancements. By harnessing the power of big data, businesses, and service providers can offer personalized recommendations that cater to the unique preferences and needs of each consumer. This not only enhances the shopping experience but also fosters a sense of satisfaction and loyalty among customers.

Acknowledgement

Zeyuan Lyu, Jiaying Li, Chunjia Tian, and Muxuan Wang contributed equally to this work and should be considered co-first authors.