Patient self-reported follow-up for radiation oncology patients during COVID-19: feasibility and patient-clinician agreement

Introduction: The COVID-19 global pandemic required health services to be innovative and quickly adapt their health service delivery, including adopting health technology in cancer clinical practice. COVID-19 restrictions forced our health service to introduce follow-up consultations for many patients via telehealth. At the same time, we explored an alternative follow-up model of care in preparation for unknown future restrictions and changes to health resources. We adapted an existing Patient Reported Outcome messaging service that linked to the patient’s medical record. Clear and meaningful interpretation of patient-reported outcome scores is fundamental to their use to determine if they could become a means of follow-up care when service delivery is impeded. Therefore, this study aimed to evaluate the feasibility of a patient self-reported follow-up model of care for radiation oncology that was opportunistically introduced during COVID. Methods: This was a cross-sectional clinical practice study set in Wollongong, Australia. Patients on radiation oncology follow-up care were sent an unannounced text message with a weblink to a survey to self-report their health before their radiation oncology telehealth appointment. Radiation oncologists completed the same set of questions during or within a day of the telehealth follow-up consultation. Descriptive statistics were analysed to evaluate the uptake of self-reporting. Percent agreement and Cohen’s Kappa were used to determine patient-clinician agreement. Results: A moderate response rate of 62% was achieved from the 142 patients. Percent agreement between the patient-reported and the clinician-reported for weight change, appetite, physical performance, side effects was acceptable (>75%). However, percent agreement was moderate for pain and sleep. For most items, Cohen’s Kappa indicated moderate agreement, with pain, side effects, and recurrence being fair. Patients were more likely to report themselves worse than the clinician for all items, except for side effects. Conclusions: Based on the findings of this study, a standalone patient-reported follow-up model of care is not feasible due to the lower than ideal response rate and fair to moderate patient-clinician agreement. However, we show the importance of capturing the patient perspective for radiation oncology follow-up care as complementary information for clinicians prior to telehealth consultations. Patient-reported information could triage phone consultation from a standard to a long consultation or triage patients requiring physical consultation and immediate attention. With further research, patients self-reporting before their telehealth consultation holds promise for future models of follow-up care, particularly for rural and remote patients and during pandemics and other disasters where clinic attendance is not possible.


Introduction
Patients who complete active cancer treatment require ongoing follow-up care to manage ongoing and late side effects, monitor recurrence and provide psychosocial care [1][2][3].When the novel SARS-CoV-2 2019 virus caused the COVID-19 pandemic, health services were forced to rapidly change how they delivered cancer follow-up care.There was a need to minimise cancer patients' exposure to the virus, as they were twice as likely to die from the first variant of COVID-19 than the general population [4].The American Medical Association encouraged the use of telehealth and technology [5], and Cancer Australia recommended that hospitals minimise outpatient visits and find alternative methods to deliver care [6].
In Australia, telehealth substituted face-to-face follow-up consultations and provided a means to continue care and maintain the safety of both patients and health care workers during the pandemic [7].When faced with restrictions, a regional hospital in Australia saw the COVID-19 pandemic as an opportunity to adopt new healthcare innovations, harness existing online health technologies, and explore alternative follow-up models of care as recommended by Cancer Australia [6].An alternative follow-up model of care considered was the use of patients' self-reporting, also known as Patient Reported Outcomes (PROs) [8].
PROs are defined as a measurement based on "any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else." [9].PROs can be measured in absolute terms, such as a pain rating scale of zero to 10, or changes in reported nausea [10].The collection of PROs in cancer care has become an important and frequent clinical practice activity to understand the impact of the disease on the patient and develop appropriate support and screening measures [11][12][13].However, whilst there has been a proliferation of validated PRO tools used in cancer care [14][15][16][17], there is no gold standard for measuring PROs in radiation oncology follow-up care [18,19].
As per the definition above, a PRO is without amendment or interpretation by the clinician.However, when the clinician interprets the PRO, it is often not well understood because of insufficient data or lack of experience or clinical understanding [20].Additionally, some clinicians are sceptical of the role PROs play and whether the information reported represents their patient's current situation [21].Patient-reported responses are subject to social and environmental conditions [21], individual motivation, interpretation of the questions, expectations and personality [22], and clinicians should not rely on PROs data to fully represent a patient's experience or condition [21].Clear and meaningful interpretation of patient-reported outcome scores is fundamental to their use [20].
In recent years, the ability to electronically collect, report and use PRO data in cancer care has become increasingly important [23]; however, inadequate health technology and poor integration of PROs with hospital electronic medical records are barriers to its integration [24].The internet has been increasingly leveraged to enable and enhance supportive care services for cancer survivors, using websites, support groups and a broad range of mobile applications to collect patient-reported outcomes [25,26].Web-based technology has allowed patients to self-report their health, screen patients and reduce hospital admissions [13].In addition, there is growing recognition that combining health technology with good measurement properties and shorter instruments could be more userfriendly and facilitate better translation of research into clinical practice [27].
Despite the barriers to interpreting and collecting PROs, there are many advantages to using PROs during COVID-19 including: monitoring the clinical trend of symptoms and side effects; prevention of the occurrence of severe adverse events; efficient screening of patients who need further phone assistance or direct medical intervention; prompt management of medical needs; positive psychological impact on patients; and increased patient satisfaction with health care services [8].
Understanding the discordance of the PRO information is essential to help interpret the data and support clinical care remotely during a pandemic or in other circumstances where access to care is impeded, such as living in rural or remote areas.Given the change and limited access to radiation oncology follow-up consultations due to hospital restrictions during COVID-19, this study aimed to evaluate the feasibility of a rapidly deployed patient self-reported followup model of care for radiation oncology.The objectives were to a) develop a set of patient-reported questions specific to radiation oncology follow-up consultations, b) evaluate patients' ability to self-report their current health status via an unannounced text message, c) determine the level of agreement between patient self-report and clinician assessment.

Method
This study was a cross-sectional clinical practice study conducted at the Wollongong hospital, Australia.Radiation oncology was selected as the service was familiar with and had implemented the web-based health technology to collect patient-reported outcomes to screen patients commencing treatment [15].Ethics approval from the the Joint University of Wollongong and Illawarra Shoalhaven Local Health District Health and Medical Human Research Ethics committee (2020/ETH01427).

Sample
This study used a convenience sample.Patients were eligible if they were scheduled for a radiation oncology follow-up telehealth consultation between June and September 2020 in the Oncology Information System (OIS -MOSAIQ®).The end date coincided with the announced cessation of the Australian Government's funding for telehealth consultations, even though this was subsequently extended.The sample size target was a minimum of 32 sets of patient-clinician data [28].Patients were not recruited or provided any training, reflecting real-time clinical practice.

Web-based technology
Web-based technologies are important as they allow patients to complete a survey online in their own time and have them subsequently transferred into the patient's oncological medical record.The information technology infrastructure (PROsaiq® 1 ) [29] consists of a webserver that uses the surveys existing within the OIS to produce a webpage in Xform format with a specific Uniform Resource Locator (URL) that can be shared.The webform contains placeholders for patient identifiers and survey assessment items.When the form is submitted, the webserver alters the returned survey from a JavaScript Object Notation (JSON) format into HL7 format and imports it into the OIS through the usual HL7 gateway.The submitted answers are stored and appear as if the survey had been completed entirely within the OIS.
The PROsaiq platform was specifically designed to obtain PRO data from patients during their cancer journey.The system has been trialled for collecting quality of lifebased patient-reported outcomes and deemed feasible in terms of use [13,30].For this study, the system was piloted for one month to monitor the condition of data being returned; no changes were required to the assessment or process.The PROsaiq platform acts only as a server of empty forms and a converter of returned forms; it does not store patient data, and deliberately cleans RAM after completing data conversion and transfer.

Tool development
At the time of writing, the hospital cancer centre used validated tools to collect PROs to screen radiotherapy patients commencing treatment and review patients on active treatment (Distress Thermometer, Problem Checklist, Edmonton Symptoms Assessment scale, Common Terminology Criteria for Adverse Events).As no specific tool that addressed radiation oncology follow-up consultations was available, a patient questionnaire that reflected standard questions addressed and recorded in follow-up consultations was developed after extensive consultation with radiation oncologists across two cancer centres.In addition to this, a document review of a sample of 20 follow-up consultation letters from radiation oncologists to general practitioners was performed to ascertain the most frequently documented items during a follow-up consultation.The questions were developed to be broad and relevant to all tumours, with plans to individualise based on tumour groups, pending the results of this study.
The final clinical assessment included performance level, sleep, appetite, weight, pain, side effects and recurrence (see Table 1).When the patients selected the weblink from the text message, it took them to the survey.Patients were asked the specific questions shown in Table 1, for example, "Are you eating well?".In contrast, the clinician was only prompted by a single word for that assessment area, for example, 'Appetite'.The reporting scales were the same for both the patient and the radiation oncologist. 1 DidymoDesigns Ltd (https://www.didymodesigns.com.au/)

Data collection
Patients: A list of radiation oncology patients scheduled for their follow-up telehealth consultation was extracted from the OIS.Using the Telstra TIM messaging system2 via email, the patient's mobile number was entered, and they were sent the following personalised message with the link to the assessment the day before their scheduled telehealth consultation (see Figure 1).
The initial text message was sent unannounced, that is, without pre-warning the patient or providing the patient with training.If the patient opened and completed the survey, the data from the completed clinical assessment was sent from the patient's mobile phone into the hospital's OIS via a secure webserver.The PROsaiq system also included a module to monitor rejected incoming assessments to allow for manual correction of contained errors, for example, incorrect spelling of the surname, switching of first and last names, or incorrect medical record number.The first author monitored this portal daily.
Radiation oncologists: were provided with a list of patients sent the clinical assessment prior to their clinic to remind them to enter the data at that point in time.During the radiation oncology telehealth follow-up consultation, the clinician-reported data were entered directly into the OIS as standard practice so that the record contained two sets of the same survey (that is, patient and clinician).When clinical needs interfered, some data was entered retrospectively based on the patient's progress notes; how much was entered retrospectively cannot be ascertained.To ensure that the patient's self-report data did not influence the oncologist's assessment, the oncologist was unaware of where the patient's entered data was located.

Data analysis
The response rate and identified errors will evaluate patients' ability to self-report via a text message.A response To minimise errors associated with manual data transfer, the patient self-reported data and the clinician-reported data were extracted separately from the OIS, copied and pasted into the same Microsoft Excel spreadsheet for analysis.The patient and clinician data were matched using the patient's medical record number, and de-identified.Typically, percent agreement and kappa should be calculated if there are four or fewer discrete ratings [32].Therefore, percent agreement and Cohen's Kappa were used to analyse the patientclinician agreement.Percent of agreement is the simplest measure of inter-rater agreement, with values >75% demonstrating an acceptable level of agreement [32].Cohen's Kappa is a more rigorous measure of the level of agreement, as it is a measure of agreement in excess of chance and interpreted as: <0.00 as poor, 0.00-0.20 slight, 0.21-0.40fair, 0.41-0.60moderate, 0.61-0.80substantial, and 0.81-0.99 as almost perfect agreement [32].If chance agreement is high, then the percentage of absolute agreement will overstate how much agreement occurred [32].

Results
A total of 167 patients were extracted from the OIS.Fifteen percent (n=25) did not have a mobile phone number and were therefore ineligible.The average age of the excluded patients without a mobile number recorded was 84 years (range 77 to 91 years); the average age of patients with a mobile number recorded was 71 years (range 24 to 92 years).
Of the 142 eligible patients, the response rate was 62% (n=88).There was no significant difference in age for patients who could self-report via text message (average age 70 years) compared to those who did not send back (average age 71).Twenty-two females were sent text messages, compared to 120 males.This difference in male and female samples resulted from the radiation oncologists that primarily treated prostate cancer having scheduled clinics on the days selected for this research, which was unknown to the research team when the sampling strategy was created and performed.
Despite the lower number of females, their response to self-reporting was slightly higher (68%, n=16) than males (60%, n=67).There were seven errors where the patient either entered a letter in their surname incorrectly or the wrong number for their medical record number; this was manually corrected.In addition, there were five instances where there was no data from the clinician due to late cancellation, no show or clinician unavailability; this resulted in a study population of 83 patient-clinician matched datasets.Patient characteristics, shown in Table 2, present that the sample comprised more prostate cancer and head and neck cancer patients, and 70% of the sample was aged between 60 and 79.
Figure 2 and Table 3 presents that of the 83 patientclinician matched datasets, there was acceptable percent agreement for most items: performance status (ECOG), appetite, weight, side effects and recurrence; with pain and sleep below the 75% threshold for acceptability.The Cohen's Kappa accounted for chance, with many items resulting in moderate agreement: ECOG, appetite, weight and side effects.Similar to the percent agreement results, sleep and pain were below an acceptable level of agreement with Kappa being fair.Despite having a high percent agreement (83%), the item for recurrence had the lowest Kappa of 0.230, and this result was not significant.
For all variables, excluding side effects, the patient selfreported their condition as poorer than the clinician-reported (see Figure 3).Sleep had the largest variance of reporting, with 37 patients reporting their sleep as being poorer than what the clinician reported.Pain was also scored as being worse by the patient on 24 occasions (15 out of the 24 occasions where the clinician rated the patients' pain lower, the variance was by one point).For the treatment-related side-effects variable, four patients reported that they had side effects when the clinician reported nil.Conversely, there were 12 occasions when the patient reported nil side effects, and the clinician reported that side effects were present.This study rapidly implemented innovative health technology for cancer follow-up patients invoked by the COVID-19 restrictions.The feasibility of using text messages for cancer patients to self-report during their follow-up period is good, and the patient-clinician level of agreement is fair to moderate.Our results are consistent with other studies, where patients were more likely to score themselves as being more impacted than clinicians' ratings for disease severity, physical performance, pain and quality of life [33][34][35].Although this study found a fair to moderate patient-clinician agreement, the questions were broad and not specific to individual tumours.However, a study that was individualised to breast cancer treated with radiotherapy also found low to poor patient-clinician concordance (for example, breast hardness, shrinkage, and veins) [36].
The response rate of 62% was above the target goal of 60% for researchers [31].However, our response rate is lower than two other studies that asked patients to self-report their health [37,38].The noticeable difference was that the other two study's participants were recruited and willing to be involved.In contrast, our study provided no notice,  While the overall agreement between patient-clinicians was measured as 'Fair' to 'Moderate', the value of each assessment is not equally important or valuable.The discordance between patients and clinicians is known to be substantial for pain scales [39] since the patient evaluation is based on perception.In contrast, the clinician's assessment includes patient language, sleeping, activity levels and pain relief use.Some of the variance in scores, specifically for pain, could be attributed to the time frame between when the patient self-reported and when the clinician entered the data during the consultation; this time frame was usually 24 hours in advance, with some minor variation due to changes in consultation time.The assessments of performance status, appetite and weight change are based on more discrete and substantive parameters.These findings are similar to a systematic review that sought to find the association of clinician-reported common toxicity scales (such as those used in this study) against patient-reported outcomes of the same toxicity items, which found there was poor to moderate association [40].
Some of the lack of agreement in sleep is understandable and demonstrates the need for slower implementation.The patient assessment asked the question, "Apart from going to the toilet, are you sleeping well?", while the OIS assessment for the clinician simply stated the word "Sleep".Given that most patients had a prostate cancer diagnosis with a reasonable frequency of prostatism symptoms in this group, this difference in wording is likely to produce a variance in answers between patient and clinician.The analysis of level of agreement should therefore downplay the influence of pain and sleep.The fair correlation of cancer recurrence belies its importance in the dialogue between patient and clinician, identifying and addressing the patient's real concerns.While these assessments are very broad, their use by the patient to identify real clinical concerns can allow the clinician to focus on these issues.
Clinicians were more likely to score the patient as having treatment-related side effects when the patient reported that they had no side effects.There is no qualitative data to understand further why the clinicians were more likely to report that the patients had treatment-related side effects.However, it is hypothesised that during the telehealth consultation, the clinician asked additional questions about the patient's health and well-being specific to their cancer type (breast, colorectal, prostate, lung, etc.) where there are other toxicities to monitor, such as skin irritation, fatigue, dysuria, cosmesis, telangiectasia, proctitis and so on.Given that the number of questions was kept to a minimum to prevent survey fatigue, there is scope to individualise the questions to the different cancer types to ascertain individualised patient information.
While the text message to the patient was manually prepared in this setting, the automatic sequencing, preparation and sending of these messages is imminently achievable within the existing PROsaiq system and would allow for more frequent and variable contact.

Limitations
The moderate response rate shows that surveys delivered by text message are acceptable for many patients; however, the study did not explore the reasons for the 38% of patients that did not respond.Possible causes for not responding are that patients do not own a smart device, the text size was too small for the phone survey, or patients may have had difficulty reading and interpreting the questions from a health literacy level.Some patients may not have been able to enter their medical record number that was provided in the text message into the survey.Since this study, PROsaiq now has been enhanced with a module that generates a URL link with a hashed identifier specific for the patient in question so that the patient does not need to add identifying information that might need manual review.
It is acknowledged that the reporting timeframe may be a factor to consider, as the pain that a patient reports 24 hours before the telehealth appointment can change quickly.However, reporting in advance would allow the patient to be triaged, and support put in place before the consultation.Additionally, responses may not be directly related to their cancer follow-up or radiation oncology toxicities, especially if the patient had concurrent treatment or other comorbidities.

Strengths
A text message-based survey administered via weblink may offer a convenient and reliable method of measuring patient-reported outcomes, particularly for weight change, appetite, physical function and side effects, and allow clinicians to triage radiation oncology patients to earlier telehealth or face-to-face appointments for clinical review.

Future implications
To better assist clinicians in supporting their patients long-term and remotely, future research should systematically correlate clinician-reported and patientreported data and qualitatively review patient preference for clinical interaction use of text messages.Once the discordance is known, the data from the patient ratings can be interpreted with more knowledge to assist the patient better.It would be beneficial to analyse results on other demographic data, such as gender.In addition, expansion to tumour-specific items is suggested, such as breast, prostate, and colorectal, as clinical questions would be more specific and potentially reduce the disparity of patient-clinician report of side effects.

Conclusion
Oncologists needed to balance the logistics of the healthcare service and patient care during COVID-19.This study showed that rapid implementation of this existing technology (PROsaiq) has benefited in catering for rapidly changing needs in follow-up cancer care.The lower than ideal response rate and fair to moderate patient-clinician agreement found in this study means that the results of this study alone cannot say that a standalone patient self-reported follow-up model of care is feasible.However, we recognise the importance of capturing the patient perspective for radiation oncology follow-up care as complementary information for clinicians prior to telehealth consultations.
Outside of the COVID-19 global pandemic, patients' self-reporting for their follow-up care can provide useful information to clinicians.Instead, this information could efficiently screen patients who need further phone assistance or direct medical intervention.Patients' self-reporting before their telehealth consultation holds promise for future models of follow-up care, particularly for rural and remote patients, and during pandemics and other disasters where clinic attendance is not possible.

Table 1 :
Clinician and patient assessment questions In addition, analysing the patient-reported and clinician-reported data provides evidence to understand if the information reported can be used to support follow-up care when patients are unable to access 'usual care' (i.e.face to face consultations, pre/post-pandemic).

Table 2 :
Patient characteristics