The pilot program included a broad cross section of public and private hospitals from all Australian states, of all sizes (small [<100 beds], mid-range [100-499 beds], and large [>500 beds]) and from urban and nonurban areas. Hospital representatives either volunteered to participate in the pilot PROMs program, were recruited following surgeon recommendation, or were invited to participate based on previous collaborative projects. In total, 44 hospitals provided pilot data.

Patients scheduled for primary or revision hip, knee, and shoulder replacement procedures were eligible. Patients under the age of 16 and patients with a cognitive impairment that impacted their ability to provide informed consent were excluded. Initial patient recruitment involved registration into RAPID via the collection of limited data including patient name, date of birth, postcode, contact details (1 or more of the following: email, mobile phone, and home phone), joint, side, surgeon (optional, if known), and hospital. Patients registered themselves or were registered by an administrator, via the RAPID platform. Administrators were provided training in RAPID by the project team and had on-site visits (or when not possible on-site, online training). Once registered, patients were able to provide electronic consent and then PROMs could be completed, either immediately or at a later time. Once registered, patients were provided with an electronic patient information sheet that detailed information about the Registry and the pilot program. A hard copy study information card was also provided to patients. If patients did not wish to participate, they were able to select “decline to participate” within RAPID, which deleted all identifying information recorded at registration from the system. However, only a small number of patients declined participation (944/14,890, 6.34%). SMS text messages and email reminders were effective in maintaining patient participation once patients were successfully registered in the electronic system [5]. Feedback provided by administrators and patients highlighted the efficiency of the patient registration and data collection system, indicating that it took 20-30 seconds to register a patient and 10-12 minutes for patients to complete their PROMs.

Automated email and SMS text message reminders were sent to patients via the RAPID platform as a reminder to complete their PROMs both preoperatively and 6 months postoperatively. The system allowed for a set number of reminders to be sent pre- and postoperatively, depending on the contact details provided at registration. Patients who had not completed their PROMs after 2 automated reminders appeared on a list for phone call follow-up. Phone call follow-up was trialed as part of the pilot and patients’ responses were entered directly into RAPID by a member of the phone call follow-up team. Comparisons of demographic information determined no substantial difference between patients who required phone call follow-up and those that responded to electronic reminders. Given this finding and associated costs, phone call follow-up was later phased out.

Throughout the data collection phase, the project manager communicated with orthopedic surgeons and personnel assisting with patient registration to ensure hospitals were well supported. A data summary was distributed regularly (containing both patient registration and procedure registration at each hospital), and hospitals were also directed to their RAPID reporting dashboards. If a hospital was identified as having low registration of patients, it was contacted so that processes could be reviewed and refined. Through this process, the AOANJRR determined that hospital registration improved over the pilot period, where more than 60.2% (441/733) of procedures undertaken at pilot hospitals were registered within the RAPID platform after 12 months of data collection, compared with 44.85% (2366/5275) in the first 3 months of data collection [5]. By examining learnings from hospitals that improved registration rates, the AOANJRR was able to implement the same processes to improve the performance of less satisfactory hospitals.

Surgical procedure forms received from each hospital were entered into the AOANJRR database as part of routine Registry processes. The patient information from the procedure form was then matched with the patient information in RAPID. This process assisted with determining the proportion of joint procedures registered at each hospital as well as identifying any data entry errors. It was also required for electronically triggering postoperative data collection reminders based on the procedure date. Reports were produced on patient registration and procedure registration rates at each hospital.

A multistakeholder approach to funding was established to foster a wide leadership base and to maximize engagement across the health care sector. Expenditure for the pilot PROMs program (n=14,890) was largely attributed to staffing costs as well as to the establishment of RAPID.

While the PROMs pilot was in the design and development phase, staffing costs were kept to a minimum by drawing on the expertise and experience of existing Registry staff. As the pilot developed and the scope increased, additional resources were required to establish PROMs collection within the pilot hospitals. This included the project manager (1.0 full-time equivalent [FTE]), software developers (2.5 FTE), data manager (0.8 FTE), and a statistician (0.6 FTE).

A large portion of establishment costs was associated with the software development phase of RAPID. In addition to the information and communications technology (ICT) staffing costs, running costs included software licensing, servers (including nonproduction servers used for development and testing), and website certificates (Aus $27,000 [~US $20,000] over 2 years). Furthermore, to ensure secure storage of sensitive patient data, a security design review was conducted during the design phase and a follow-up software penetration test was conducted prior to launch (Aus $14,000 [~US $10,000] over 2 years). While these costs substantially contribute to the overall budget, the security of patient data is at the forefront of consideration in health care settings [7].

A total of 70 tablet devices (specifically iPads) were purchased by the Registry and provided to 41 pilot hospitals to mitigate potential barriers relating to patients not having access to electronic devices and to encourage hospital participation. The total cost including providing cellular data for the devices was Aus $31,000 (~US $23,000). Data indicated that hospitals only intermittently provided patients with the tablets for patient self-registration and survey completion [5]. With this in mind, it has been decided that tablets will not be provided when the PROMs program rolls out nationally in Australia.

During the pilot period, 1148/14,926 (7.7%) procedures registered in RAPID had a landline telephone only (predominantly patients aged ≥85 years) [5], precluding patient follow-up via SMS text messaging. Extrapolating the cost of Aus $65,000 (~US $48,000) for telephone follow-up in the pilot (involving about 15% [n=44] of available hospitals) to a broader national rollout produces an estimate exceeding Aus $450,000 (~US $336,000) annually. This was deemed to be financially unsustainable. Throughout the pilot and since its completion, AOANJRR has encouraged hospitals to obtain the contact details of a proxy individual (eg, a family member or friend) to receive reminders on the patient’s behalf and assist with the electronic completion of PROMs.

Nesting the PROMs pilot project within the AOANJRR was key to successfully reducing costs as it allowed for shared staff resources throughout the lifecycle of the project, particularly in the planning and implementation phases of the project. This included utilizing the existing expertise, skills, and relationships of Registry staff to liaise with surgeons and clinicians directly at hospital sites. These relationships assisted with approval for hospitals to participate and reassuring surgeons regarding any process or security concerns. The well-established reputation of the AOANJRR allowed for confidence in data security by patients, surgeons, and stakeholders.

Electronic PROMs collection has proven to be a successful means of outcome data collection, as evidenced by other Registries, such as the Functional and Outcomes Research for Comparative Effectiveness in Total Joint Replacement (FORCE-TJR) Registry that has reported high levels of enrollment and data completion via electronic collection [8]. Our pilot has proven similar success in that of the patients who consented within RAPID, preoperative PROMs collection was obtained for 97.77% (12,871/13,164) of registered procedures, and 79.05% (4184/5293) of postoperative PROMs were completed [5]. Wilson et al [2] highlighted that paper-based questionnaires can be relatively easy for patients to complete, but issues persist with mail-out, having patients mail them back, following up on missing data, possible data entry errors, or duplications [2]. These issues were mitigated with the use of the RAPID platform. Another consideration is potential patient response differences when 1 question is viewed at a time (versus traditional paper-based collection) and the order in which the questions are answered. Some research has indicated that displaying 1 question at a time improves response rates; however, equivalence studies compared with paper-based collection would provide benefit to the orthopedic community [9]. In our pilot study, the small number of patients who were unable to access RAPID or could not be contacted via electronic means were encouraged to seek assistance from family or friends to complete their PROMs or to contact the Registry for assistance with online completion. In the pilot study, preoperatively, 10.34% (782/7562) of patients reported seeking assistance from a family member and 0.45% (34/7562) of patients reported seeking assistance from a friend. Technology is increasingly affordable and accessible with internet access in Australian private homes recorded at 86% [10]. Access and technological familiarity will likely continue to improve over time, including for older patients.

The ability to adapt RAPID functionality following user feedback and respond quickly to address any identified issues was a critical enabler to improve engagement with hospital sites and optimize patient registration. Hospital staff feedback identified that patients in preadmission clinics are required to attend a variety of appointments during their preadmission review and this made it difficult to complete the survey on the supplied iPad in a single sitting. A system adjustment was then made to implement a “resume” function in RAPID. This allowed patients to exit RAPID part way through the survey and return to complete the survey within 14 days in their own time on any device.

Administrators identified during the pilot that the electronic consent format was proving onerous for patients as they had to click “I agree” to each consent statement (9 in total) before providing final consent. The system was updated so that the information sheet could be reviewed on a single page with patients consenting to participate in the study once the information was reviewed by pressing 1 button.

Registry staff identified that it would be helpful to generate additional PROMs completion reminders manually when patients requested. RAPID was updated to include a manual link that generated additional SMS text messages or email reminders on request.

Such enhancements proved important in facilitating data collection and reducing the burden on patients. This also underscores the importance of continuous software developer resourcing to support additional minor platform refinements, even after a project has launched.

One of the main considerations for this program was to develop an electronic system that would not overburden hospital and administrative staff. Two pathways for patient registration were therefore developed; patient self-registration or registration of a patient by an administrative user. During the pilot, participant recruitment increased when the patient was registered by hospital staff. For the national implementation of this program, the Registry will continue to encourage the registration of patients by hospital staff.

Frequently reported barriers for patients to complete PROMs are the length of time to completion and difficulty using electronic devices [11]. Early reports from patients suggested that the time taken to complete the PROMs instruments was satisfactory (10-12 minutes for PROMs completion) [5]; however, further exploration of patient burden and preferences is necessary through seeking consumer representation [6].

Staff turnover at the hospitals was identified as an issue impacting patient recruitment. Registry staff were often unaware that a hospital staff member trained in using RAPID had left the hospital until a decrease in patient registration was identified. In these instances, it was identified that key project details were not communicated or understood when hospital staff handed over the task to new personnel. To alleviate this potential barrier, the Registry has implemented additional reporting strategies to monitor hospital recruitment as well as continuing to provide specific hospital training, education material, and induction documentation to meet the needs of new hospital staff.