Minimizing Interobserver Variation: The Achilles' Heel of Positron Emission Tomography/Computed Tomography Response-Adapted Treatment Approaches.

In the past 2 decades, F-fluorodeoxyglucose positron emission tomography/computed tomography (PET/CT) imaging has become essential to response-adapted treatment algorithms in some lymphoma types, in addition to its established roles in staging and final remission assessment. Response-adapted treatment protocols use early response assessment by PET/CT to intensify treatment and improve outcomes where response is suboptimal or to reduce treatment burden without compromising the cure potential when response is optimal. Randomized controlled trials have helped to define which specific treatment modifications are effective in which clinical scenarios and which are not, and the results of many of these trials have now been implemented in routine clinical practice.1Andre MPE Girinsky T Federico M et al.Early positron emission tomography response-adapted treatment in stage I and II Hodgkin lymphoma: Final results of the randomized EORTC/LYSA/FIL H10 trial.J Clin Oncol. 2017; 35: 1786-1794Crossref PubMed Scopus (343) Google Scholar, 2Johnson P Federico M Kirkwood A et al.Adapted treatment guided by interim PET-CT scan in advanced Hodgkin's lymphoma.N Engl J Med. 2016; 374: 2419-2429Crossref PubMed Scopus (546) Google Scholar, 3Mauz-Korholz C Landman-Parker J Balwierz W et al.Response-adapted omission of radiotherapy and comparison of consolidation chemotherapy in children and adolescents with intermediate-stage and advanced-stage classical Hodgkin lymphoma (EuroNet-PHL-C1): A titration study with an open-label, embedded, multinational, noninferiority, randomised controlled trial.Lancet Oncol. 2022; 23: 125-137Abstract Full Text Full Text PDF PubMed Scopus (39) Google Scholar With the increased reliance on imaging to modify treatment, interobserver variation (IOV), which has been considered the Achilles’ heel of imaging,4Robinson PJ Radiology's Achilles' heel: Error and variation in the interpretation of the Röntgen image.Br J Radiol. 1997; 70: 1085-1098Crossref PubMed Scopus (255) Google Scholar has more effect than ever before. Efforts to quantify IOV, analyze its sources, and minimize its frequency are very important, both in clinical trials and routine clinical practice. These efforts include standardizing response criteria, observer training, double reporting, quality assurance, and benchmarking of reporting, and in clinical trials, central review of imaging. Standardization of response assessment criteria in lymphoma has undoubtedly improved the accuracy of response designation and reduced IOV. Initial reports of PET in lymphoma just differentiated between negative and positive PET corresponding to complete metabolic response and residual disease. Subsequently, the International Harmonisation Project criteria and later the Deauville score (DS) were introduced and are based on comparing the uptake in previous sites of involvement to the physiological uptake in the mediastinal blood pool and the liver. Although a significant improvement over simple positive/negative designation, the DS is essentially visual and remains liable to IOV. Using semiquantitative methods, as an aid to visual assessment, to report the DS more accurately and reduce IOV has been recommended by the Lugano guidelines.5Barrington SF Mikhaeel NG Kostakoglu L et al.Role of imaging in the staging and response assessment of lymphoma: Consensus of the International Conference on Malignant Lymphomas Imaging Working Group.J Clin Oncol. 2014; 32: 3048-3058Crossref PubMed Scopus (1103) Google Scholar They suggest that standardized uptake value is measured in lesion(s) and liver and that score 4 is applied to uptake above the maximum standardized uptake value in a large region of normal liver. However, this recommendation is not always adopted in routine reporting. Even with standardization of response criteria the problem of IOV remains, and this led to including central imaging review as a gold standard in clinical trials. Most of the influential clinical trials used a central review process to ensure consistency of response assessment and reduction of IOV. However, the necessity of central imaging review in trials is continuously questioned, especially with wide availability of PET/CT imaging in clinical practice. In the article by Hoppe et al,6Hoppe et al (current study)Google Scholar the effect of central review of imaging in a large PET-guided clinical trial in pediatric Hodgkin lymphoma was assessed. PET was performed after 2 cycles of chemotherapy, and its result determined whether patients receive radiation therapy after completion of chemotherapy. Those with DS 1-3 did not receive radiation therapy and those with DS 4-5 received radiation therapy. The study included a central review by 2 nuclear medicine physicians. Overall, the central review changed the designation of 1 to 3 versus 4 to 5 in 59 of 573 cases (10.3%). There was more change from positive to negative (38/126 = 30.2%, 6.6% of total) than from negative to positive (21/447 = 4.7%, 3.7% of total). This is consistent with other reports showing that overcalling is more common than undercalling in routine reporting. Interestingly, the 2 central reviewers (drawn from a panel of 8 with varying levels of experience) agreed between themselves on all cases apart from 2, which required adjudication by a third reviewer. In the largest published study of IOV performed on more than 1000 scans in diffuse large b-cell lymphoma (DLBCL),7Burggraaff CN Cornelisse AC Hoekstra OS et al.Interobserver agreement of interim and end-of-treatment F-18-FDG PET/CT in diffuse large B-Cell lymphoma: Impact on clinical practice and trials.J Nucl Med. 2018; 59: 1831-1836Crossref PubMed Scopus (21) Google Scholar the overall agreement between 2 reviewers (drawn from a central review panel of 10) was 87.7% for interim PET (iPET) and 91.7% for end-of-treatment PET. The disagreement level of 8.3% to 12.3% is close to the 10.3% in the Hoppe study,6Hoppe et al (current study)Google Scholar although in the DLBCL study7Burggraaff CN Cornelisse AC Hoekstra OS et al.Interobserver agreement of interim and end-of-treatment F-18-FDG PET/CT in diffuse large B-Cell lymphoma: Impact on clinical practice and trials.J Nucl Med. 2018; 59: 1831-1836Crossref PubMed Scopus (21) Google Scholar the disagreement is measured between central reviewers. Similarly, the positive agreement was lower than the negative agreement in iPET (73.7% vs 92%) and end-of-treatment PET (76.3% vs 95%). In the response-adapted therapy in advanced Hodgkin Lymphoma study,8Barrington SF Kirkwood AA Franceschetto A et al.PET-CT for staging and early response: Results from the Response-Adapted Therapy in Advanced Hodgkin Lymphoma study.Blood. 2016; 127: 1531-1538Crossref PubMed Scopus (124) Google Scholar the central review changed DS 1-3 to DS 4-5 in 4% (9/231 cases) and changed DS 4-5 to DS 1-3 in 22% (15/69 cases). The total change was 8% (24/300). In a retrospective international validation study in advanced Hodgkin lymphoma,9Biggi A Gallamini A Chauvie S et al.International validation study for interim PET in ABVD-treated, advanced-stage Hodgkin lymphoma: Interpretation criteria and concordance rate among reviewers.J Nucl Med. 2013; 54: 683-690Crossref PubMed Scopus (244) Google Scholar the expert review panel changed the response designation of local reports (DS 1-3 vs 4-5) in 24 of 260 cases (9.2%). The progression-free survival was better predicted by the central review than the local response designation. Another small study of 106 patients reported a central review change from positive to negative in 38% (20/52) and from negative to positive in 9% (5/54) when using iPET in DLBCL (total change 24%).10Torka P, Pederson LD, Knopp MV, et al. Is local review of positron emission tomography scans sufficient in diffuse large B-cell lymphoma clinical trials? A CALGB 50303 analysis [e-pub ahead of print]. Cancer Med. doi:10.1002/cam4.5628.Google Scholar The consistent finding of significant change by the central review process across studies highlights the importance of central review. The range in the studies listed previously is 8% to 24%, and the current study rate of 10% probably represents the average incidence. It is worth noting that this level is of a similar magnitude to the differences expected between treatments, or the noninferiority margins tested, in lymphoma trials. Introducing IOV of about 10% could well interfere with the accuracy of outcome and jeopardize the conclusions. Hoppe et al conclude that central imaging review should remain as an essential part of imaging-guided trials, a conclusion that is well supported by their and other similar studies. The cost of central review is usually small compared with the overall costs of running trials in modern times, and the technology of secure imaging transfer and remote access is improving all the time, which should facilitate the process further. The central review process also has some unintended but positive consequences, including fostering expert discussions, developing consensus to move the field forward, and enabling secondary studies on centrally stored imaging studies. The problem of IOV remains an issue in routine clinical practice. The probability of allocating the wrong response category and consequently a different treatment in 1 in 10 patients would not be acceptable to most clinicians. Measures to reduce the IOV in clinical practice are therefore of paramount importance. Training courses focusing on sources of IOV are needed. Burggraaff et al7Burggraaff CN Cornelisse AC Hoekstra OS et al.Interobserver agreement of interim and end-of-treatment F-18-FDG PET/CT in diffuse large B-Cell lymphoma: Impact on clinical practice and trials.J Nucl Med. 2018; 59: 1831-1836Crossref PubMed Scopus (21) Google Scholar identified frequent areas of IOV (eg, Waldeyer ring, spleen, skeletal, and gastro-intestinal sites). Some departments provide routine double reporting. In other situations, a review of the imaging in tumor boards before treatment change is decided offers another quality assurance step to reduce IOV. Internal quality processes to retrospectively review and analyze reporting discrepancies can also be very useful learning tools to improve quality of reporting. PET/CT response-adapted treatment approaches can only improve clinical outcomes if based on accurate image reporting, and every effort should be made to reduce the interobserver variation both in clinical trials and routine practice. Importance of Central Imaging Review in a Pediatric Hodgkin Lymphoma Trial Using Positron Emission Tomography Response Adapted Radiation TherapyInternational Journal of Radiation Oncology, Biology, PhysicsVol. 116Issue 5PreviewWe investigated the effects of central review of the interim fluorodeoxyglucose−positron emission tomography/computed tomography (FDG-PET/CT) scan response (iPET) assessment on treatment allocation in the risk-based, response-adapted, Children's Oncology Group study AHOD1331 (ClinicalTrials.gov identifier: NCT02166463) for pediatric patients with high-risk Hodgkin lymphoma. Full-Text PDF