Web-based rating tools are prominent across industries, ranging from travel and hospitality to education and health. Their prevalence indicates the importance of reputation management systems in consumer decision-making. In health care, patients use web-based evaluation tools to assess their physicians, health care teams, and overall medical experience. Web-based patient reviews (WPRs) on websites such as RateMDs [1] continue to increase in popularity [2-4] and consist of a rich source of information on patient experience [4,5]. The terms “web-based physician ratings” and “physician rating websites” are also used in the literature to refer to patient-provided ratings and reviews of physicians. Although WPRs may offer information and perspectives valued by other patients, concern has been expressed regarding the representativeness and bias of these perspectives [6,7], and little is known about their potential utility as a source of feedback that might be used for improving care quality.

Web-based health information seeking has a substantial effect on reducing patient uncertainties [8]. WPRs can be considered a valid measure of patient experience, although not necessarily of care quality [9]. Although WPRs are growing in popularity [2], research demonstrates that web-based patient ratings may not be the best indicator of quality health care because of the low correlations between WPR scores and validated survey instruments [10]. Physician demeanor is a dominant factor in both positive and negative reviews [11]. It is challenging to use WPRs to observe changes in physician behavior or performance over time [12]. Patients respond most to certain quality signals, primarily physicians’ interpersonal and clinical skills [13]. Expressions of emotion and disease severity moderate WPR helpfulness levels, and expressions of anger in particular have a negative impact on the perceived helpfulness of physicians, specifically in terms of severe diseases [14]. The average number of physician ratings has almost doubled between 2010 and 2019 [15]. WPRs have a meaningful impact on the selection of one’s physician [16]. Overall, web-based health information seeking has an impact on patients’ evaluation of physicians, as well as on patients’ psychology and behavior [17].

This study draws on the CanMEDS Framework to assess the presence of standardized physician competency in WPRs. The CanMEDS Framework “identifies and describes the abilities physicians require to effectively meet the health care needs of the people they serve” [18]. The main purpose of the Framework is “to define the necessary competencies for all areas of medical practice and provide a comprehensive foundation for medical education and practice in Canada” [19]. The physician abilities described in the CanMEDS Framework are split into 7 roles: medical experts, communicators, collaborators, leaders, health advocates, scholars, and professionals. According to the Royal College, “a competent physician seamlessly integrates the competencies of all seven CanMEDS Roles” [18]. Each role is defined by key concepts, key competencies, and enabling competencies.

The specialty of medical oncology is characterized by rapid and consequential decision-making regarding prognosis and the prescription of highly toxic medications, both of which occur during a period of heightened stress for the patient. Medical oncology requires a combination of skills related to collaboration, communication, and professionalism, ultimately delivering technical and clinical knowledge in practice. A patient’s experience with their medical oncologist is a crucial element of care that often marks the beginning of their cancer journey, as the physician’s knowledge and interpersonal skills have a substantial impact on the patient’s experience [20,21]. Standard assessment tools (eg, written examinations and objective structured clinical examinations) may not be the most effective in evaluating competencies beyond technical skills and knowledge base. Moreover, these standardized assessments typically take place only at the start of the physician’s career, rather than providing periodic feedback as clinicians grow and adapt and as standards and patient expectations change over time. As unstructured and unsolicited assessments, WPRs provided at any time throughout the physician’s career may potentially represent a more meaningful source of ongoing physician evaluation at the experiential level. Altogether, the prevalence of WPRs, the importance of physician demeanor in patients’ perception of care, and the challenges of cancer treatment lead to the following 2 research questions: To what extent do the standardized competencies set out by the CanMEDS Framework appear in the WPRs of medical oncologists? and How do patients, when producing WPRs, perceive and assess the quality of their experience with their medical oncologist?

The purpose of this study was to evaluate the contributions and potential utility of WPRs in the context of medical oncology patient-provider relationships. Given the difficulty of interpreting the extent to which WPRs are valid or reliable measures of physician performance [6], scoring comments according to a valid and established framework provides an opportunity to measure their potential value. The CanMEDS Framework identifies and describes the physician competencies required for the effective delivery of health care. We assessed the presence of CanMEDS competencies in WPRs using a reproducible structure for ranking web-based comments and matching them with specific CanMEDS competencies.

This 2-stage mixed methods analysis of the WPRs of medical oncologists aimed to quantify the focus on specific CanMEDS competencies and to qualitatively describe the contents of such patient-provided assessments of medical competencies. The goal was to understand which CanMEDS competencies are the most frequently commented on in the WPRs of medical oncologists. The outcomes of this study include knowledge of the extent to which CanMEDS competencies are present in WPRs as well as an understanding of patients’ perception of important physician qualities in the context of quality cancer care.

The WPRs of university-affiliated medical oncologists in 4 midsized urban areas with medical schools in the province of Ontario (Canada) were collected for the purpose of this study. Physicians were identified by searching the College of Physicians and Surgeons of Ontario (CPSO) by city, place of practice, and specialty (medical oncology). The CPSO is an organization that “regulates the practice of medicine in Ontario” [22]. As such, its website provides a searchable directory of every physician who is licensed to practice medicine in the province of Ontario, with the goal of helping the population make decisions when seeking a health care provider. The CPSO does not provide information on physicians’ birth dates. The CPSO provides the following information for each physician: name, gender, date of registration and class, languages spoken, education, location of practice, hospital privileges, specialties, terms and conditions, postgraduate training, registration history, and disciplinary details. Toronto (University of Toronto) and Thunder Bay (Northern Ontario School of Medicine) were excluded from this study to ensure even distribution among the cohort. Relative to Toronto, which has a larger population, and Thunder Bay, which has a smaller population, the 4 cities chosen for the study are of a similar size and socioeconomic profile. Ottawa (University of Ottawa), Hamilton (McMaster University), Kingston (Queen’s University), and London (Western University) were the 4 sites that comprised the basis of data collection (Table 1).

RateMDs profiles of the cohort were identified based on the CPSO search results. To capture all RateMDs profiles, all possible name permutations were searched, such as the physician’s first name and surname, middle name and surname (if applicable), or only the surname. Launched in 2004, RateMDs is a common platform for patients to post reviews of physicians. WPRs on RateMDs [1] include an average number of stars ranging from 1 to 5, which accounts for the patient’s rating of the staff and their punctuality, helpfulness, and knowledge, as well as written comments. WPRs are posted anonymously and voluntarily (as such, there is no information available on the sociocultural levels of the WPR authors in this study). WPRs consist of a numerical ranking and often include a written comment. WPR authors are asked to select whether the visit they are reviewing occurred in-person, via real-time video, telephone, or other means. They are then prompted to describe their experience through text. The prompt reads, “Please leave a comment with more detail about your experience” [24]. RateMDs provides the following additional information: physician specialty (based on an existing list created by RateMDs), overall specialty numerator and denominator based on ratings (eg, 1 of 200), posting date, and the number of comments marked “helpful” by anonymous readers on the web.

All data in this study were preexisting, and there was no collection of personal information beyond that publicly available in the CPSO database on the web. The reviews of physicians identified as eligible through the CPSO search were downloaded to a spreadsheet. Multiple profiles (eg, using name variations) of the same physician were merged. Reviews were anonymized for reviewer and physician information. Data collection occurred in August 2020.

The CanMEDS Framework, established by the Royal College of Physicians and Surgeons of Canada, identifies and describes the physician competencies required for the effective delivery of health care. CanMEDS competencies are divided into the roles of medical experts, communicators, collaborators, leaders, health advocates, scholars, and professionals, with key concepts and enabling competencies constituting the definitions of each role.

This study considers validity not in terms of the correlation between physician reputation and clinical outcomes but between patient evaluation and recognized physician competencies. Validity was established through a matching process. If a WPR reflected one or more CanMEDS roles and could be associated with physician competencies, it achieved validity. The WPR validity increased based on how many roles or competencies were present within the comment. However, all WPRs, independent of their level of association with physician competencies, were considered in the subsequent qualitative analysis to identify themes or qualities that may exceed, depart from, or add nuance to the CanMEDS Framework.

Two reviewers—1 communication studies researcher and 1 health care professional—independently assessed WPRs according to the CanMEDS Framework and identified common themes using pragmatic analysis [25]. WPRs were collected, anonymized, and exported to a spreadsheet by the research assistant. The reviewers familiarized themselves with the CanMEDS Framework and then matched the WPR text with one or more CanMEDS roles. Once the roles were selected, the reviewers selected the key concepts and enabling competencies that best associated with the WPRs.

The reviewers also took note of cases in which the WPRs mentioned phenomena not specific to CanMEDS, such as punctuality or wait time. In addition to matching the WPR text with the CanMEDS Framework, the reviewers recorded whether the review was positive, negative, mostly positive, mostly negative, physician positive and ancillary negative, physician negative and ancillary positive, or ancillary or other. In this context, ancillary care refers to experiences outside the physician’s care but related to the medical experience, such as those associated with nursing or administrative staff, technicians, directions, parking, traffic, or other obstacles on the way to one’s appointment. Comment scores were then evaluated to identify κ agreement rates between the reviewers, and a descriptive quantitative analysis of the cohort was completed. The WPRs were matched with CanMEDS in December 2020.

Following the quantitative analysis, the WPR text was exported to NVivo (QSR International) for qualitative thematic analysis. An inductive thematic analysis was performed [26]. Portions of the WPR text were coded to nodes that corresponded to the themes. Drawing from López et al [27], the WPR text was also coded to a case to classify the WPR as concise or elaborate depending on the level of detail offered by the author of the WPR. Concise WPRs represented written comments that included little detail but still clearly articulated their author’s experience and perception. Elaborate WPRs are WPRs that tended to include several specific details and often described particular anecdotes. The thematic analysis was performed in October 2021.

The subsequent inductive thematic analysis contributed to the initial deductive approach using the CanMEDS Framework. CanMEDS is a standardized set of documentation; by using it as a reference point, we can identify instances when WPRs directly comment on qualities that physicians are meant to adhere to. Therefore, the subsequent thematic analysis provided knowledge on how patient concerns are expressed through WPRs and the specificities with which they are written. Our inductive approach allowed for additional nuances that could not be provided by the initial analysis. The first analysis revealed which of the CanMEDS roles and competencies are discussed in WPRs, whereas the second revealed how CanMEDS manifests in the written text of WPRs. These approaches fit together because they each offer results the other cannot; therefore, each provide complementary answers to the overarching research questions of the extent to which the CanMEDS roles and competencies are present in WPRs and the ways in which patients perceive and assess physician quality through WPRs.

All data in this study exist in the public domain; therefore, this study was deemed exempt by the research ethics board of McGill University. WPRs are authored anonymously. WPR author identities cannot be known and are not known by any authors of this paper. The identities of the physicians being reviewed were known only by the first author at the time of data collection. When the first and second authors scored the WPRs, they had no access to the identities of the physicians for whom the WPRs were written. Any WPR that mentioned a physician’s name was deidentified. All deidentification processes were completed by a research assistant.

This study identified the RateMDs profiles of 49 actively practicing university-affiliated medical oncologists in midsized urban areas in Ontario, Canada (Table 2). Of the included physician profiles, 71% (35/49) were those of men. All the physicians spoke English. French was the most common second language (13/49, 27%). Overall, 73% (36/49) of the physicians studied at a domestic university, whereas 27% (13/49; 6/13, 46% women and 7/13, 54% men) studied at a foreign university. Furthermore, 31% (15/49) of the physicians obtained their undergraduate degree between 1990 and 1999.

We identified a total of 473 WPRs reviewing these 49 physicians. Of the included physician profiles, 37% (18/49) had 5 to 9 WPRs. The mean word count per physician was 52.15 (SD 23.7) words. Agreement levels between the coders (κ scores) were high in all roles (weighted κ=0.71-1.00). In addition to commenting on physician experience, 20.9% (99/473) of the WPRs contained a mention of the support staff, and 23% (109/473) of the WPRs were clearly identifiable as being written by care providers.

Among the CanMEDS competencies, those defining the roles of medical experts, communicators, and professionals were the most prevalent (303/473, 64%; 182/473, 38%; and 129/473, 27%, respectively). The identified themes were similar between positive and negative evaluations. The most-discussed positive themes were knowledge and compassion (Table 3). Less prominent positive themes pertained to follow-up quality, referral, family relationship, current knowledge, and research participation. The most common negative theme was poor interpersonal skills. Negative themes also pertained to lateness, punctuality, or overall wait time for appointments. As expected, negative comments tended to represent the opposite of the most common positive themes, such as a lack of compassion or communication skills, a perceived lack of knowledge, and an unprofessional behavior. Overall, 38.1% (180/473) of the WPRs were marked as helpful by anonymous readers, and 27.9% (132/473) of the WPRs provided a high (4-5 stars out of 5) rating.

The inductive thematic analysis revealed the themes in Table 4. The percentages listed refer to the amount of coded text in relation to all WPR text. All themes included WPRs that were considered either concise or elaborate. In total, 78% (369/473) of the comments were coded as concise, and 19% (90/473) were coded as elaborate.

This section includes excerpts from the WPRs that correspond with the themes of interpersonal skills, medical skill and knowledge, and medical-personal duality, with general and specific examples of each. The first 2 themes represent the 2 most common themes identified, whereas medical-personal duality, a less common theme that combines the interpersonal and medical skills, provides insight into what patients interpret to be the most important elements of care. Concise WPRs regarding interpersonal skills included comments such as “He is very friendly and approachable” and “She can be negative and non-supportive.” Elaborate and interpersonal-themed WPRs described the physician’s communication skills and personality in greater detail:

Regarding medical skill and knowledge, concise WPRs included “extremely knowledgable,” “Knows his stuff,” and “Great oncologist!” Elaborate WPRs that discussed medical skill and knowledge tended to comment on clinical trials, treatment decisions, and prolonged life rather than particular scientific or technical competence:

Less frequent were the WPRs that considered the balance between medical knowledge and personality. Generally, these WPRs would mention a strength in one and a lack in the other:

In more specific terms, the following WPR offers a description of both the physician’s interpersonal skills and medical expertise. The author interprets what may be perceived as a lack of compassion as evidence of the physician’s increased attention to detail and competence in cancer care:

Although WPRs do evaluate characteristics found in the CanMEDS Framework, they offer detailed experiences that illustrate how the CanMEDS roles function in practice and are perceived by patients, thereby contributing to our understanding of the significance of physician assessment tools. The themes, stories, details, and patterns revealed by attending to the WPRs qualitatively provided the following insights: (1) confirmation that patients evaluate characteristics found in the CanMEDS Framework, (2) descriptions of how the CanMEDS roles manifest in practice as perceived by patients, and (3) patients’ perceptions of important and expected physician qualities in cancer care. These results indicate that the CanMEDS roles and competencies that are explicitly patient facing (ie, those that are directly experienced by patients in their interactions with physicians and through the care physicians provide) are the most likely to be present and reported on in WPRs.

This study uses both quantitative and qualitative methods to assess WPRs as a potential source of information on physician competency using a validated framework such as CanMEDS. To date, studies of WPRs were limited by large data sets spanning several specialties and a focus on numerical star ratings. This work differs because of its focus on a defined subset of physicians within a particular specialty. While the CanMEDS Framework was created for physicians practicing in Canada and has been used in this study to assess the WPRs of physicians in a Canadian context, its competencies represent qualities that are standard and commonplace in medicine, making our observations transposable to other clinical practice contexts. As such, similar assessments could be performed in other countries using the CanMEDS Framework or an equivalent set of guidelines as a reference point.

We found that most RateMDs WPRs of medical oncologists address a limited number of themes, several of which speak directly to CanMEDS competency areas and indicate what patients consider to be important, reportable, and shareable qualities within the context of cancer care. Our findings suggest that WPRs may serve as an additional tool for evaluating physician competence, may be relevant to physician training or continuing development, and may be used to flag or prevent lapses in care. In sum, WPRs emphasize experiential competencies related to communication and interpersonal skills and suggest an alternative format for evaluating the care provided by medical oncologists.

Studies of WPRs tend to be quantitative. Previous studies of web-based evaluations have used methods such as word frequency, surveys, and regression models. These studies were retrospective, observational, and comparative (for instance, comparing data from 2 sources, such as physician peer reviews or institutional reviews with patient-provided web-based reviews). Qualitative studies of WPRs include the use of interviews and content analysis. Many studies of WPRs draw on large cohort sizes, often comprising thousands of web-based reviews. Moreover, research shows that WPRs consider not only doctor-patient encounters but also ancillary forms of care, such as those provided by support staff [27], and the context of care. Urgent care reviews tend to mention operational elements such as staff competence [28]. Moreover, the validity of patient ratings (in terms of their consistency with physician peer reviews) varies by medical specialty [29]. Physician-based and office-based web-based physician reviews do not necessarily correlate highly with each other [30]. For example, Quinones et al [31] also found that discussions of pain management were present in both positive and negative WPRs of neurosurgeons; top-rated WPRs demonstrated the importance of compassion for levels of patient satisfaction. The study by Seltzer et al [32] on patient experiences in web-based reviews of obstetric care revealed that narratives found in WPRs related to obstetrics provide meaningful information toward the improvement of obstetric care experiences.

Our focus on a local or midsized cohort allowed us to manually analyze individual comments and match them with the CanMEDS competencies to evaluate their meaning and importance. By reducing the scale, we have the capacity to test this study’s assessment methods within a relatively homogeneous cohort of physicians, allowing for comparison, which, in future research, can be applied to cohorts of different sizes or characteristics. By matching the WPR text to specific CanMEDS competencies, we were able to identify what competencies could potentially be captured by WPR authors. Most physicians in our cohort were trained, passed certification examinations, and now practice in a Canadian context where the CanMEDS Framework is in effect. Therefore, it is reasonable to expect that they would meet the competencies under which they are expected to practice.

As demonstrated by scholarship on physician ratings, communication and respect between physicians and patients are important elements of patient care [33]. Little is known about the written text that accompanies WPRs [34]. The results of our thematic analysis respond to this gap and suggest that WPRs, when detailed, tend to include the following elements: (1) experience and connection, including information regarding the author of the WPR or their family or friend and the type of cancer and a timeline indicating how long they have been the physician’s patient; (2) discussion and evaluation of the physician’s knowledge, professionalism, interpersonal skills (often described as “bedside manner”), and punctuality; (3) in the case of positive reviews, the expression of feelings of gratitude and thankfulness; and (4) in the case of positive reviews, a recommendation or in the case of negative reviews, discouragement from seeking the physician’s care.

The WPRs in this study reflect that patients value time management, access to information, and whole person care or comprehensive care. In this context, time management refers to instances when patients do not feel rushed during their appointments and do not feel as if they are asking too many questions or taking up too much of the physician’s time. Access to information comprises patients being able to receive clear answers to all of their questions or concerns and not being condescended to for raising them. Comprehensive care refers to patients feeling that their physician has taken a genuine interest in them and that they are treated as a whole person and not as a number or disease. Taken together, these are especially valuable and important elements of care in the context of medical oncology, where patients experience pain and side effects as a result of toxic medications and may be overwhelmed with new information.

Although patients’ perception of medical skills is less specific than their perception of interpersonal qualities, medical skills are the most commented-on element of care in the RateMDs WPRs assessed in this study. Patients’ perception of interpersonal skills (listening, compassion, and overall caring demeanor) and other experiential phenomena, such as feeling rushed during appointments, is often specific, detailed, and described with more nuance than what is offered by the CanMEDS Framework. The findings of this study indicate that patients are highly perceptive, particularly of their physician’s interpersonal skills. Patients recognize and describe in WPRs instances when their physician does not remember them, seems unprepared, is not thorough in their examination or questions, and does not seem to care about or value them. This suggests that details about a physician’s interpersonal skills or “bedside manner” are highly perceived, valued, and shareable in a WPR context such as RateMDs.

In addition, we found that only a small number of WPRs reflected a distinction between the value of medical skills and that of interpersonal skills. The authors of these WPRs claimed that, for them, a physician’s medical skills and competence are more important than their interpersonal skills (ie, being a “good doctor” is more important than whether they are polite). Although not representative of most WPRs, this is an important perception and articulation of what qualities matter the most in one’s medical oncologist. However, compassion and politeness remain highly appreciated qualities. Owing to the subjectivity of WPRs, there is an ongoing debate on how the information these evaluations provide should be measured, interpreted, and potentially applied to the improvement of health care. The prevalence of WPRs that evaluate interpersonal communication skills reveals the subjective and experiential nature of what WPRs discuss, part of which may not be testable using standardized assessment tools. Although WPRs provide insight into the patient-doctor relationship, their importance in relation to the quality of health care is understudied [35,36]. Research indicates that the validity of WPRs varies according to medical specialization [29]. For instance, medical practitioners in obstetrics and gynecology face a high risk of patient complaint [37]. It is unclear whether WPRs have enforceable legal value, although they have appeared in lawsuits before, for instance, in cases of libel where a physician considered their WPRs to be defamatory [38,39].

Patients who leave reviews tend to be women with postsecondary education who have a chronic illness [40]; however, demographic variables and health status do not act solely as predictors for a patient’s use of physician evaluation tools. WPR websites such as RateMDs [1] have a substantial influence on patients who read or write reviews [16,39]. For example, Emmert et al [35] demonstrated that 65% of survey respondents went to a physician based on that physician’s WPRs, whereas 52% did not go to a physician because of their WPRs. Similarly, Li et al [16] demonstrated that patients were considerably more likely to be swayed by web-based ratings than by objective report cards when selecting a cardiac surgeon. Although not completely indicative of health care quality, our findings signify that WPRs are a strong indicator of experiential qualities, which themselves may influence care, particularly in terms of trust and communication between the patient and the physician, and may represent opportunities for feedback and improvement.

Importantly, WPRs exist within a wider environment of sponsorships and reputation management systems. Physicians have the option to pay for additional features on their RateMDs pages, such as banner advertisements for their profiles on other physicians’ profiles [41]. Patients are more likely to share the experiences they had with physicians who have “a higher medical quality and service attitude and who work in hospitals with a higher online reputation” [42]. Social influence, in the form of credibility and performance expectancy, has, at minimum, an indirect impact on the “behavioral intention” to use physician rating websites [43].

Overall, there is consensus that WPRs are important and comprise a valuable source of data, although there is less consensus on how the importance and validity of these ratings can be measured and interpreted to better understand their meaning. Although WPRs continue to grow in popularity, they constitute a minimal proportion of the physician’s actual patients [44]. Moreover, WPRs have been shown to have a poor association with the physician’s clinical performance scores [6]. Qualitative analysis can help validate rating tools [45], as the information provided by WPRs is most valuable when considered in the appropriate context [46]. Therefore, although the available WPRs do consist of valuable data, their small sample size is considered to decrease their overall validity [2,3,47]. It is also recommended that WPRs not be published until a sufficient number of reviews are reached, such as 15 reviews [47]. Outside health care, rating systems have a major impact: if an Uber (Uber Technologies, Inc) driver’s star rating (provided by passengers) drops below the established average for that city, the driver will lose temporary or permanent access to the Uber platform and will not be allowed to drive for Uber [48]. Altogether, the benefits afforded by WPRs, which include a sense of patient empowerment and the perception of informed decision-making, must be considered alongside their flaws, which include a small sample size and poor association with clinical outcomes [6,49]. As shown by our study, WPRs can potentially be used to assess experiential qualities while remaining aware of the possible drawbacks of relying on anonymous web-based content.

The limitations of this study include a cohort of physicians skewed toward men, uneven numbers of WPRs per physician, and the unavailability of the demographic data of the WPR authors. However, the skewed physician gender distribution in this study is representative of the actual gender disparities in medical oncology within the area in which the study cohort was built [50]. The number of WPRs a particular physician has is partially dependent on the number of years the physician has been practicing, which suggests that physicians with more years of experience will have amassed more WPRs than those with fewer years of experience. However, the impact of WPRs on modulating physician behavior remains unknown. Moreover, although this study moves toward an analysis of patient expectations, further analysis using the expectancy-disconfirmation paradigm would be necessary to sufficiently conclude on patient expectations within this study’s results [51].

This study gestures toward the potentially valuable content found in WPRs. It demonstrates the opportunity to learn from WPRs, not simply to discern physician popularity but to grasp what patients may expect from their physicians. In this context, WPRs could potentially represent a method for assessing patient-facing physician competencies. The next steps include conducting a survey to determine patients’ own assessment of this study’s results and patient engagement with WPRs in the context of medical oncology, as well as further analysis to identify patient expectations. Future research is required to assess WPRs’ role in and influence on cancer care, medical education, and experiential competency assessment.