Older adults over the age of 65 years need a system that supports distinguishing rotten food items because older adults are exposed to the danger of eating rotten food and suffer from health problems [1]. Compared to younger age groups, older adults have a lower ability to recognize whether food items are rotten [2], and as they get older, their cognitive function decreases [3], which causes a proportionate overall decrease in visual, olfactory, and gustatory functions [4].

Prior studies have found that decreased olfactory function increases the chances of eating rotten foods. According to a study by the University of Pennsylvania Medical School, olfactory disorders affect the quality of life, appetite, and weight [5]. Similarly, the results of the National Health and Nutrition Examination Survey in the United States found that people over the age of 70 years have difficulty recognizing dangerous odors, such as those of smoke and gas [6]. As such, older adults are in danger of being exposed to food poisoning if they ingest rotten food items due to the difficulty detecting whether the items have gone bad. Food poisoning is caused by the ingestion of toxic substances, as well as bacteria contained in rotten food items [7]. To address this danger, researchers have developed and evaluated tools to detect rotten food items [8,9] for unspecified users or specific users, such as people with olfactory impairment. Prior studies have also used chemical sensors and kits and, in some cases, cameras to detect rotten food items and measure freshness [10].

Prior studies have proposed and evaluated tools to measure food freshness for unspecified users [8-21]. Researchers have used chemical sensors or kits to detect the spoilage of specific food items such as pork and chicken [11-14] or unspecified food items [10,15,16], such as ham [11], fruits (eg, banana, grape) [12], and pork [13]. Specifically, to detect spoilage of ham produced in Spain, Choi et al [11] proposed a tool that determines whether ham is rotten using a sensor that responds to sulfur-containing compounds in the gas generated during ham decay. In addition, Caya et al [12] proposed a tool with an electronic nose to detect gas generated during the decay of bananas, carrots, and grapes by using k-nearest neighbors and principal component analysis of the collected data. Similarly, Tian et al [13] proposed a tool that detects spoilage of pork by determining the number of aerobic bacteria in pork through a sensor and principal component analysis. Meanwhile, Mikš‐Krajnik et al [14] proposed a tool detect 27 kinds of volatile organic compounds generated during the decay of chickens by selecting 3 types of indicators highly correlated with spoilage in order to detect spoilage of chickens. Researchers have also proposed tools with an electronic nose to detect decay using gas generated from meat, fruits, and vegetables [10,16]. Meanwhile, Janagama et al [15] used an organism detection kit called a dipstick to detect wine spoilage.

Researchers have also proposed various methods (eg, convolutional neural network [CNN], Fourier transform infrared spectroscopy [FTIR]) to detect the spoilage of specific food items using a camera [8,9,17-21]. Perez-Daniel et al [17] used a camera to detect the spoilage of unspecified food items. They proposed a tool that collects images of both normal and rotten food items through a neural network using RetinaNet and compared them to detect the food spoilage. Other researchers have proposed tools to detect the spoilage of certain food items, such as fruits [8,18,19], vegetables [9], beef [20], and rice [21]. For example, Karakaya et al [8] obtained a co-occurrence matrix from a grayscale histogram of an image to detect decay in apples, bananas, and oranges and extracted features from the obtained matrix using the bag-of-feature method. The extracted features were classified into normal food items and rotten food items using a CNN and a support vector machine (SVM). In addition, researchers have proposed tools to detect decay in citrus fruits [19] and other fruits [18] using visual features. Jagtap et al [9] proposed a tool to take pictures of potatoes moving through a conveyor belt and detect decay in them using a neural network and features extracted from the pictures. Meanwhile, Ellis et al [20] used FTIR and machine learning to detect food spoilage through chemical changes in beef. Additionally, Batugal et al [21] proposed a tool to detect decay in Philippine rice using both a chemical sensor and a camera; this tool uses machine learning to classify data collected into spoiled rice and normal rice [21].

Prior studies have proposed and evaluated tools to measure food freshness for specific users, such as people who manage the freshness of meats [22-26]. These tools detect food spoilage, even in unspecified food items [22], using chemical sensors or kits [23-25]. Specifically, researchers have proposed and evaluated tools to determine the spoilage of tomato dishes [24], fish and beef [24], and milk [25] using a gas sensor. Researchers who proposed and evaluated a tool to detect spoilage of tomato dishes in the Philippines using an electronic nose for people with an olfactory impairment equipped with a methane/CH₄ quality (MQ) gas sensor and a temperature and humidity sensor to reduce the spoilage of tomato dishes [23]. Similarly, a tool for identifying the spoilage of fish and beef using an electronic nose, an artificial neural network, an SVM, and k-nearest neighbors was proposed for food freshness inspection by butchers [24]. Another study proposed a tool for identifying spoiled milk using an electronic nose and fuzzy c-means clustering to prevent older adults suffering from olfactory disorders and dementia from drinking spoiled milk [25]. Musa et al [22] proposed a film for packaging material developers that changes color when it encounters rotten food items by sensing the pH with corn starch-glycerol and anthocyanin.

A previous study proposed a tool for certain users to use a chemical sensor and a camera to determine food spoilage. Kodogiannis et al [26] proposed a tool that detects microorganisms in meat using FTIR and an advanced clustering-based neuro-fuzzy identification model, determining the degree of meat spoilage.

Nevertheless, little is known about studies that have developed and evaluated apps to help determine the freshness of fruits for the basic diet of physically healthy older adults. Therefore, we developed and evaluated a smartphone app for healthy older adults that determines whether apples, bananas, and oranges are rotten. Apples, bananas, and oranges are among the most consumed traditional fruits in South Korea and the United States [27]. In addition, we could easily collect photos of rotten apples, bananas, and oranges, so we selected them as the target fruits of our app [28]. Textbox 1 shows the research questions of our study.

By answering our research questions, we will contribute to older adults and the community of researchers. First, we proposed a smartphone app that supports older adults in avoiding consuming rotten fruits. Second, we revealed their perceptions about the app and showed them how to evaluate the app, which uses an artificial intelligence (AI) model. Third, we contributed to the related research community by revealing which of the 7 pretrained backbone networks we used to classify rotten fruits showed the best performance. To answer the research questions, we reviewed the relevant literature and developed and evaluated the performance of the app. To evaluate the app, we used an after-scenario questionnaire [29] and conducted semistructured interviews [30] with older adults.

This study aimed to develop and evaluate a smartphone app that enables older adults to take pictures of food items with a smartphone camera and to determine whether the items are rotten. We first reviewed prior studies that have developed and evaluated similar spoilage detection tools. After reviewing the relevant literature, we trained a model that classifies the freshness of 3 fruits (ie, apples, bananas, and oranges) and evaluated the model’s performance. Next, we developed a smartphone app that uses the trained model to take images of fruits and classify them into images of fresh and rotten fruits. We recruited 23 older adult participants to evaluate the usability of the app and to determine their perceptions about the app.

The purpose of the literature review was to discover the limitations of prior studies that have developed and evaluated tools that classify food items into rotten and fresh items. To collect prior studies that have reported any developed apps determining food freshness by taking pictures of food items, we used a combination of keywords (eg, “rotten,” “food,” “collect,” and “app”) in Scopus, IEEE, the ACM Digital Library, PubMed, and EBSCO. We also used similar meanings for each keyword (eg, spoiled, corrupt, damaged, ruined, and expired for similar meanings of rotten) and linked them with “OR” and “AND” as query sentences (see Textbox 2). We exported each paper’s metadata, such as the title, year of publication, author, DOI, publisher, and keywords, and stored it to Google Spreadsheets. We used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram to manage duplicate papers between databases and to screen review papers (see Figure 1) [31]. We excluded papers not meeting the eligibility criteria and found 19 eligible papers. After reviewing each paper to examine the research questions and objectives of prior studies, we found that no prior studies have reported how to support physically healthy older adults in determining the freshness of their diet using readily available equipment, such as a smartphone.

The objective of this study was to develop an app that determines whether an apple, a banana, or an orange is rotten just by taking a picture of the fruit using a smartphone camera. App development and evaluation procedures consisted of 4 steps: (1) collecting images, (2) training classification models, (3) developing an app, and (4) evaluating the models. First, we collected pictures required for training the food classification model. We collected pictures of fresh and rotten apples, bananas, and oranges from Kaggle, a platform that provides preprocessed data sets to data scientists: 2088 pictures of fresh apples, 1962 pictures of fresh bananas, 1854 pictures of fresh oranges, 2943 pictures of rotten apples, 2754 pictures of rotten bananas, and 1998 pictures of rotten oranges (see Figure 2). We uploaded and stored the collected pictures on the server for classification model training.

Second, we trained food picture classification models. To use only the features of the fruit in the picture for training, we segmented the pixels of the fruit from the background using Otsu’s [32] method. We stored the segmented pictures with the same size as the original pictures. When training the models, we used several pretrained residual networks (ResNets; eg, ResNet-50, ResNet-50V2, ResNet-101, ResNet-101V2, ResNet-152, ResNet-152V2, Inception-ResNet-V2) for feature extraction, reducing the time for model training and improving model performance [33]. Among the parameters used in the single fully connected layer, we set the activation function among sigmoid, rectified linear unit (ReLU), and tanh. We set the dropout rate to 0, 0.1, 0.2, 0.3, 0.4, or 0.5 and the number of neurons to 64, 128, 256, 512, or 1025. We randomly selected 100 combinations of the activation functions, the dropout rate, and the number of neurons. Among the combinations, the layer combination that showed the best performance consisted of a dense (neurons=128, activation=sigmoid) layer, a dropout (drop rate=0.2) layer, a dense (neurons=128, activation = sigmoid) layer, and a dropout (drop rate=0) layer.

Third, we developed a smartphone app that uses a classification model that classifies fruits in pictures into fresh or rotten fruit. Android Studio [34] is the official integrated development environment for Android apps; the proportion of smartphone users in their sixties and older is steadily increasing, and the older they are, the more they use Android than iOS. Level 29 of the app is the minimum application programming interface (API) level that supports app download from Google Play, and the API level that can be operated on most smartphones was selected. We developed 3 functions: a function to take a picture using a smartphone camera, a function to input pictures into the classification function, and a function to show the classification result on the app screen. We created a low-fidelity prototype [35] to determine user tasks and 3 features of the app (see Figure 3). We added a camera function and set permissions (eg, camera, internal storage) for the function that would enable users to take pictures. The pictures taken are resized to 256 × 256 pixels to fit the input size of the trained classification model. The resized pictures are input to the converted classification model with the .h5 extension. Third, the app takes the output array from the classification model that consists of the probability value and displays one of the following texts: “The apple is rotten,“ ”The apple is fresh,“ ”The banana is rotten,“ ”The banana is fresh,“ ”The orange is rotten,“ and ”The orange is fresh“ (see Figures 4 and 5). To perform these functions, we added several buttons to allow users to interact with the app.

Lastly, we evaluated the performance of our trained models (eg, area under the curve [AUC], error rate). The model evaluation aimed to determine the best-performing model among the models developed using multiple backbone networks in the model training stage. To fix the metric to be used for model evaluation, we found the metric used to evaluate the classification model in each of the papers in the related area. A prior study used the F₁-score, balanced accuracy, the error rate, and the AUC to evaluate the classification model [23]. After extracting the above metrics through 10-fold cross-validation, we found a significant difference between each model’s performance by using the Kruskal-Wallis test [36], a method that can be used when n≤30 for ≥3 classes. We performed this test to check whether using different backbone networks makes significant differences in model performance.

We evaluated the app’s performance to answer research questions 4 and 5. We performed a performance evaluation of the classification model used in the app, a quantitative evaluation using an after-scenario questionnaire [29], and a qualitative evaluation through semistructured interviews with the participants.

We quantitatively and qualitatively analyzed the participants’ responses. We removed every participant’s identification information and assigned them a new one (eg, P1, P2). Next, we calculated the questionnaire responses’ mean (SD) scores for quantitative analysis. For qualitative analysis, we used open-coding methods that highlighted meaningful remarks within scripts. We highlighted statements that included important information (eg, participants’ experiences, preferences) in each script [39]. For each highlighted remark, we assigned a label to summarize the remark. We printed all the labels and inductively grouped them to extract important themes that answered our research questions (see Figure 6) [40]. For qualitative research, affinity diagramming, a technique using sticky notes, is a prevalent practice due to its inherent flexibility and tangibility, which facilitates the organization of thoughts and findings in a spatially representative way. The ability to physically manipulate and rearrange data points is a key advantage of affinity diagramming, fostering deeper thematic analysis and the identification of emergent relationships and patterns. This technique also promotes collaborative discussions among researchers by providing a clear visual representation of the data analysis process.

All experimental procedures were approved by the Institutional Review Board (IRB) of the University of Seoul (IRB ID: 2021-06-001-001). Prior to the experiment, each participant provided informed consent by signing a consent form about participating in the study and recording the interview, which also included a clause allowing the use of collected data for secondary analyses without additional consent. To protect the privacy of collected data, all participant data were anonymized and stored on a secure drive, which only authorized personnel have access to. After completing the interviews, all participants received a cash compensation of South Korean won 10,000 (approximately US $8) for their participation.

Table 1 shows the details of the 26 participants enrolled in this study.

The results of our study were classified into (1) performance evaluation results of the trained model during app development, (2) quantitative analysis results of responses collected through the survey, and (3) qualitative analysis results of responses collected through interviews.

The results of this study showed the behaviors and perceptions of older adult participants when using our app. The survey responses revealed their positive perceptions about the app’s ease of use and their satisfaction with the time taken to complete tasks. However, the survey also showed a low willingness to pay for the service among older adults. Interviews provided further insights, revealing participants’ appreciation for the app’s usability, rapid result generation, and straightforward interface. Conversely, some participants expressed a desire for more features, while others raised concerns regarding trust in the app. In the following sections, we discuss findings based on the results, the limitations of our study, and future work.

We developed an app that detects rotten fruit for the health management of older adults and investigated their experiences from various aspects. However, future work is still needed to gain deeper insight into the topics discussed. First, we developed a smartphone app that determines whether a fruit is fresh when older adults take a picture of it, but the app does not include a function to distinguish the freshness of food items other than the 3 fruits (apples, bananas, and oranges). According to the results, 8 participants wanted to know the freshness of food items other than the 3 fruits. It might be important to determine the freshness of a wider variety of food items, such as vegetables, herbs, and meat, to contribute to preventing older adults from experiencing health problems due to eating rotten food items. However, to determine the freshness of food items other than the 3 fruits, a large number of images are required, which requires a lot of time and resources. According to a previous study that proposed a visual-based method for detecting rotten food items, rotten food items have common visual characteristics [45], using which might enable the determination of the freshness of more diverse food items without collecting corresponding food image data.

Second, we used 7 ResNets as backbone networks to create a model that can distinguish the freshness of 3 fruits. We selected ResNet-101 with the highest classification performance among the 7 networks. However, we did not use several other backbone networks. Using more diverse backbone networks, including the recently proposed backbone network, might improve the performance of the model. Recently proposed models in a classification problem using ImageNet showed more than 10% higher accuracy than the backbone network used in our study [46,47]. The performance of classifying rotten food images might be greatly improved using the recently proposed network.

Third, we analyzed the collected interview scripts using open coding, but 2 or more researchers did not participate in this step. Participation of more than 2 open coders is required to enrich the quality of the results and reduce the bias that may occur when a single coder is performing the analysis.

Regarding future work, first, to help older adults use the app for determining the freshness of various food items and not eat rotten food, researchers need to add a function that distinguishes the freshness of food items other than the 3 fruits. Second, to enable older adults using the app to more accurately check the freshness of food items, researchers may adopt recently developed networks that show high performance in image classification as backbone networks. Researchers may use a pretrained image processing transformer [48] by fine-tuning it with the image data we collected. We may also extend the possible food categories by using generalized zero-shot learning [49]. Third, for app improvement, data collected from a questionnaire and interviews should be analyzed by 2 or more coders to guarantee the reliability of the findings of this study.

The goal of this study was to help older adults avoid eating any rotten food items and not suffer from health problems. We established research questions and goals based on the limitations of prior studies where researchers created and evaluated tools for detecting rotten food items. To achieve the goals, we found the highest-performing classification model among various backbone networks by training a model to determine whether the targeted fruit was rotten. The trained model was used for developing a smartphone app. The findings of this study with older adult participants revealed that the usability of the app and the older adults’ perceptions about the app are positive, but they tend to feel reluctant to use the app if they need to pay for it. We also found how to design and evaluate an app using AI models targeting older adults by uncovering their perceptions about the app. Our study contributes to the research community by revealing which of the 7 pretrained backbone networks shows the highest performance on the ImageNet classification problem for determining whether the targeted fruit is rotten. We hope our proposed app enables older adults to identify rotten food items efficiently for maintaining their health.