Abstract
The Centers for Medicare and Medicaid Services (CMS) has provided a mechanism for expanded coverage of selected promising technologies under its “coverage with evidence development (CED)” policy. The National Oncologic PET Registry (NOPR) was designed to address the CED requirements for collection of clinical and demographic data to allow for CMS coverage of PET for previously noncovered cancer types and indications. The NOPR opened in May 2006. This report reviews the NOPR's data collection and analysis plan. Methods: NOPR is a nationwide prospective internet-based registry. All PET facilities that are participating providers in the Medicare program may enroll in NOPR. The PET facility is responsible for collecting and entering patient data into the NOPR database through a Web application at http://www.cancerPETregistry.org/. Data are collected from the requesting physician on Pre-PET and Post-PET forms. The primary research goal is to assess the effect of PET on referring physicians' plans of intended patient management across the spectrum of expanded cancer indications (diagnosis, staging, restaging, suspected recurrence, and treatment monitoring). The NOPR investigators will have access to data only on cases in which both the patient and the referring physician have consented to allow their data to be used for research. Data will be analyzed and compared in aggregate for all cancers by category (e.g., staging) and then for specific high-impact types and indications (e.g., staging of pancreatic cancer) when 200 patients have been accrued to a specific combination or after the NOPR has been operational for 1 y. Conclusion: The NOPR will allow an accurate assessment of the impact of PET on intended patient management across a wide spectrum of cancer indications.
PET performed with the radiopharmaceutical 18F-FDG is a diagnostic imaging procedure that allows for the assessment of regional glucose metabolism in normal and diseased organs and tissues. Its use in cancer imaging is based on the principle that most malignant neoplasms exhibit increased use of glucose (and accordingly, increased uptake of 18F-FDG) in comparison with normal tissues (1).
Between 1998 and 2005, the Centers for Medicare and Medicaid Services (CMS) approved reimbursement of PET performed on Medicare beneficiaries for specific indications in 9 malignancies (2). In 2005, CMS indicated its intent to establish coverage for PET for essentially all other cancers and indications when the provider is participating in and patients are enrolled in one of certain prospective clinical studies (“coverage with evidence development” [CED]), including a prospective registry (2).
The National Oncologic PET Registry (NOPR) provides a mechanism for CMS coverage of PET for patients with currently noncovered cancers and indications in the context of a registry. The data from this registry will be used to evaluate the impact of PET on intended patient management across all of these cancer type/clinical indication categories, and within specific cancer type/clinical indication categories. CMS can use this information to make evidence-based decisions with regard to subsequent coverage of PET use for each cancer type/clinical indication category. The process leading to the development of the NOPR and an overview of the mechanics of registry operation have been reported recently (3). In this report, we describe the data collection and analysis plan of the NOPR.
METHODS
The NOPR is a prospective data registry that collects information from the facility that performs the PET scan, from the physician requesting the PET scan, and from the interpreting physician's PET report. The NOPR is managed by the American College of Radiology (ACR) through the American College of Radiology Imaging Network (ACRIN) in Philadelphia, PA. All NOPR data are entered by participating PET facilities via a secure Web-based interface; the data are stored on secure servers at ACR headquarters in Reston, VA. The NOPR home page and Web application, including all forms, are located at http://www.cancerPETregistry.org/. A NOPR Working Group composed of representatives from the Academy of Molecular Imaging, ACRIN, ACR, and the Center for Statistical Sciences at Brown University directs the research operations.
The NOPR's design is modeled after a recent cohort study that prospectively collected data on a continuous series of PET scans performed for all cancer-related indications at one U.S. academic medical center (4). Similar methodology has been used at other institutions to assess the impact of PET on clinicians' intended patient management plans (5–12).
Institutional Review Board Approval
The NOPR Working Group asked the ACR Institutional Review Board (IRB Federal Assurance #0002271) to determine whether the activities of the NOPR constituted research and, if so, what informed consent procedures would be required. The ACR IRB determined that data collection by the NOPR itself is not research, because it is required by CMS for reimbursement purposes for all patients having PET under the conditions of the registry. However, the IRB determined that the activities of the Working Group to analyze the data are considered research. The ACR IRB granted approval to the Working Group to engage in research using a dataset that includes data only from those patients who provide oral consent to staff at the PET facility and referring physicians who indicate consent on the form completed after the PET scan. Under Title 45 CFR § 46.117(c)(2) (2005) regarding minimal risk of harm to subjects, written documentation of consent was waived by the IRB. In addition, the ACR IRB concluded that individual PET facilities and referring physicians and their staffs are not engaged in research and, therefore, are not required to have IRB review and approval before their participation in NOPR under Title 45 CFR § 46.117(c)(2). The ACR IRB also determined that CMS is not conducting research. The Office of Human Research Protections separately reviewed the NOPR procedures for protection of human research subjects and found them in compliance with all applicable Department of Health and Human Services (DHHS) regulations. The DHHS also reviewed the NOPR procedures and found that they met the requirements for protection of patient confidentiality required by the Health Insurance Portability and Accountability Act.
Registry Workflow
Figure 1 illustrates the NOPR operations schematically. The responsibilities of the facility and the referring physician are outlined. Patients referred to NOPR-registered PET facilities for NOPR-eligible cancer type and indication categories are registered in the database via a secure Web-based application. Each eligible PET study receives a unique NOPR case number. For each study, the facility reports the type of scan done (PET vs. PET/CT), the specific scanner used that has already been profiled in the database, and the interpreting physician.
NOPR schema. See text for details.
For the PET procedure to be covered by CMS, the referring physician must complete both a Pre-PET form and a Post-PET form. The Pre-PET form must be submitted to the PET facility for entry into the database by the PET facility by midnight on the day the study is performed. If not, the PET study is ineligible for CMS reimbursement. The information collected on the Pre-PET form and the Post-PET form is summarized in Table 1. The Pre-PET form collects the following information: (i) the specific reason for the PET study, (ii) the patient's cancer type (if known) and working stage assessment, (iii) the patient's Eastern Cooperative Oncology Group (ECOG) performance status, (iv) an indication of whether the referring physician will also be the treating physician, and (v) the referring physician's planned management if PET were not available.
Question Timing and Types in NOPR Case Report Forms
The PET scan must be completed within 2 wk of case registration. If not, the registration is cancelled and the PET facility is notified by e-mail. If the PET study is performed later, the patient must be reregistered, and the information on the Pre-PET form must be updated, as necessary.
Before the PET study or when the patient arrives for the PET scan, the PET facility provides the patient with the ACR IRB-approved Patient Information Sheet (Appendix). The patient may indicate his or her consent orally to the staff at the PET facility on the day of the PET study or by telephone within 2 working days after the PET study is completed. Written consent is not required, as noted above. The PET facility records in the database whether the patient provides or withholds consent for use of his or her data in future NOPR research. If consent is withheld, data for this PET study are not included in the research database used by the Working Group.
Once the PET scan has been completed, the PET facility uploads the PET report to the database. The final step is the completion of a Post-PET form by the referring physician, who sends it to the PET facility for data entry. There are several Post-PET forms. These forms differ slightly from each other, depending on the reason for the PET study; however, all Post-PET forms assess the referring physician's planned management of the patient in light of the PET findings (Fig. 2). The physician is asked on this form for consent to use the data for NOPR research. If his or her consent is withheld, data for that PET study are not included in the research database used by the Working Group. The Post-PET form data must be entered via the Web site within 30 d after the PET study.
Primary endpoint: change in intended management strategy.
Case Completion and Reimbursement
Once all required NOPR forms have been entered into the database in a timely fashion, the PET facility is notified that it may submit its claim to CMS for the PET study (global claim or separate professional and technical claims). As explained earlier, the dataset compiled for use by the Working Group contains the data for those PET scans for which both the patient's and the referring physician's consent have been obtained. The entire dataset, regardless of consent decisions, is provided to CMS.
Statistical Analysis
Several prospective and retrospective studies have examined the change in patient management after a PET scan. Reported frequencies have ranged from 10% to 40% (4–12). Most of these publications were from single centers that assessed oncologic PET in aggregate or focused on common specific cancers and indications currently covered by CMS for Medicare beneficiaries.
The Working Group will evaluate the change in intended management for the currently noncovered cancers and indications for all patients as well as for each cancer type/clinical indication category individually. PET studies obtained for diagnosis of breast cancer in patients with a suspicious lesion on mammography or physical examination and those obtained for axillary nodal staging in patients with breast cancer or for regional nodal staging in patients with malignant melanoma are not included in the NOPR, because these indications are explicitly excluded by CMS (2).
To determine whether a category should be recommended to CMS for coverage, we will determine the proportion of cases in that category for which a change in management would be clinically meaningful. The “clinically important” change in management may differ when cases are assessed in aggregate as opposed to assessment for each specific cancer type/indication category. If all cases of initial staging are considered, a minimal threshold may be a 25% change in management. Yet, for particular cancers and particular indications, a clinically important effect size may differ. For example, a change in intended management in only 5% of patients with hepatocellular carcinoma being staged for planned liver transplantation might be judged as both very important and sufficiently robust to warrant requesting CMS coverage for this indication. On the other hand, a change of 30% might not be judged important if the intended management change was from first-line to second-line chemotherapy for pancreatic cancer. What constitutes a clinically important change will be determined primarily by the Working Group on the basis of available literature for the specific disease category and advice from outside experts for any disease category in which a consensus cannot be reached.
Table 2 illustrates, at different potential sample sizes, the 1-sided 95% confidence intervals (CI) for true proportions of changes in intended clinical management of 5%, 10%, 20%, 30%, and 50%, respectively. The change in intended management was modeled as a binary variable (parameter P) following a binomial distribution. The lower 95% CI bounds for the binomial proportion based on a normal approximation are shown.
95% Confidence Intervals for Different Probability Estimates of Minimally Important Difference in Change in Management
Assessment Plan
The NOPR Working Group intends to evaluate the data in 3 phases using a single definition of change in intended management. In the first phase, we will evaluate the impact of PET when all cancers are aggregated together across all indications and then subsequently stratified by cancer indication. This “big picture” approach takes advantage of the NOPR's large sample size and should provide estimates with narrow CI. In the second phase, we plan to examine the impact of PET on individual cancer type and indication combinations. In the third phase, we will evaluate the impact of PET when used for treatment monitoring because its use for this indication under the NOPR includes all cancers (except breast cancer, where this use is already covered) and where changes in the types or duration of therapy may be important and unique endpoints.
In the second phase, we will examine the data for each cancer type/indication pairing once 200 patients have been accrued to that category or after NOPR has been operational for 1 y. The proportion of patients for whom a change in intended management is observed will be determined. If the estimated proportion of change were below the predetermined minimum clinically important change, coverage would not be recommended to CMS. If the estimated proportion were above the minimum clinically important change, the lower bound for a 1-sided 95% CI based on that proportion would then be calculated. The predetermined minimum clinically important change will then be compared with this lower 95% CI bound. If its value for the minimum clinically important change falls below the lower confidence bound, then the Working Group will recommend that CMS consider coverage outside of the NOPR for that cancer type/clinical indication category. If its value is greater than the lower confidence bound, then under the same estimated true proportion of change, the number of additional patients that must be accrued to obtain an estimate of the desired lower 95% CI bound will be computed. A decision will then be made after the additional patients have been recruited.
For example, if the estimated true proportion of change is 20% after the first 200 patients for a specific category have been accrued, then—on the basis of the numbers in Table 2—the lower bound for a 1-sided 95% CI is 15.3%. If the minimally clinically important change were judged to be 15%, then that category would be recommended to CMS for coverage because the minimum clinically important change is below the estimated lower 95% CI bound. If 17% were judged to be the minimum clinically important change, then under the assumption that the estimated true proportion of change is 20%, an additional 481 patients need to be recruited to determine that the estimated lower 95% CI bound is above 17%.
Priority Cancer and Indication Combinations
The Working Group reviewed 30 principal cancer types (e.g., pancreatic cancer) and 4 indications (diagnosis, staging, restaging/recurrence, treatment monitoring) for a total of 120 combined data cells. Table 3 lists the 15 priority areas considered by the NOPR Working Group to warrant early evaluation. Table 3 includes 2 conditions for diagnosis, 4 for staging, 4 for restaging or suspected recurrence, and 5 for therapeutic monitoring. Note that there is no overlap of cancer types between therapeutic monitoring priorities and the other 3 indications.
NOPR Working Group's Priority Areas for Early Evaluation
Definitions of Change in Intended Management
Change in intended management is measured using the 3 different approaches shown in Table 4. The first approach categorizes management intent before and after PET as being either treatment (surgery, chemotherapy, or radiation therapy alone or in combination) or nontreatment (watch, noninvasive imaging, tissue biopsy, and supportive care). This provides a simple, but powerful, 2 × 2 grid of the impact of PET on patient management.
Approaches to Defining Change in Management
The second approach splits the intent of the therapeutic management into curative or palliative intent. In this approach, a meaningful change includes a change in intent, even if the specific mode of therapy does not change. The third approach includes considering a change in therapeutic mode as a consequence of PET. This is illustrated in Table 5, where, for example, changing from surgery to chemotherapy would be a major change in mode even if the treatment goal were unchanged. For illustration purposes, Table 5 does not include treatment plans including 2 or more modes of therapy.
Supplemental Data: Matrix of Combinations of Pre-PET and Post-PET Management Plans and Their Classification
The “treatment monitoring” indication will include additional unique elements since treatment has already been initiated. Categories of change in management include adjustments in dose or duration of therapy, switching to another form of therapy, and stopping current therapy. Referring physicians are also asked “if PET were not available, would you have done some type of alternative assessment at this time?”
In secondary analyses, logistic regression models will be used to assess the impact of PET on intended management by controlling for several covariates. In these models, the dependent variable, change or no change of intended management after PET, will be modeled as a function of cancer type, cancer indication, PET facility characteristics (fixed vs. mobile), type of scan (PET vs. PET/CT), patient performance status, and referring physician as treating physician. Also, regression models for multi-category outcomes will be used in secondary analyses to further explore moderate changes in intended management. For example, a cumulative logit model for ordinal response could be applied to a 3-category treatment change: no change, intramode change, intermode change. To account for potential clustering effects in the data, such as physician or hospital effect, mixed-effect models or marginal models based on generalized estimating equations will also be considered in our future analyses.
DISCUSSION
The NOPR began accepting patient registrations on May 8, 2006. Minor difficulties in the initial implementation were identified and corrected, including security concerns with unencrypted protected health information in e-mail communications, programming errors, and ambiguity in specific items in the Pre-PET and Post-PET forms. Unwillingness of referring physicians to complete the required Pre-PET and Post-PET forms has not been a prominent problem. As of July 31, 2007, there were 1,596 PET facilities nationwide participating in the NOPR, and complete data had been obtained for 42,852 patient studies. Both patients and referring physicians consented to the use of their data for research purposes in about 86% of cases.
There are several important limitations of the data being collected by the NOPR. The quality of the primary endpoint data is largely dependent on the care with which referring physicians or their staff members complete the Pre-PET and Post-PET forms. These forms were designed to be essentially self-explanatory, as it was not possible to provide specific training to the large number of individuals involved in this activity; accordingly, variability in the accuracy of form completion was an expected limitation of the data. To assess the magnitude of this potential problem, the Working Group has periodically evaluated samples of the data. The reviews included evaluation of the internal consistency of answers to related questions on the forms (e.g., comparing the summary stages reported before and after PET with the question asking whether PET showed the disease extent to be more, less, or the same as that judged before PET). The reviews also included comparisons of the Pre-PET and the Post-PET form responses with the somewhat more complete information in the clinical reports of the PET examinations. These reviews generally showed the data to be of reasonable quality but made some problems evident. For example, it became apparent that there was confusion about the meaning of the term “diagnosis” as a reason for ordering a PET scan, as it seemed that some respondents selected this reason when diagnosis of a recurrence of a known cancer was the intent of the examination. Confusion between “treatment monitoring” and “restaging” also was noted in some cases. These observations led us to include detailed definitions of the reasons for ordering PET scans on the Pre-PET form. We also found that some cases were being entered into the NOPR for cancers and indications that were covered by CMS; we communicated with participating PET facilities to increase their awareness of this concern. Several similar concerns were addressed by broadcast communications to participating PET facilities (these remain posted on the NOPR Web site for ready consultation). Additionally, the Working Group investigators have responded to many individual requests for clarification about subject eligibility or completion of answers to specific questions on the Pre-PET and Post-PET forms.
The NOPR will provide data for a very large number of Medicare patients and from a large fraction of all PET facilities in the United States about the changes in intended patient management after PET. It is not possible to determine from the data in the registry whether the reported changes in intention result in an actual change in patient care. The NOPR investigators hope to link the registry data with CMS billing records in the future to assess the level of agreement between the intended and actual patient management.
Even if PET does change patient management, it is unclear whether such changes will have a positive impact on patient outcomes. Although information on certain patient outcomes may be obtained from Medicare billing records, because of the large range of potential outcomes, targeted studies must be designed to assess the impact of PET on specific positive or negative patient outcomes. For example, PET may facilitate more effective treatment of disease if it detects potentially treatable cancer metastases not identified by other imaging modalities and may lead to a reduction in cancer mortality. On the other hand, PET may detect previously unsuspected incurable metastatic disease. In these situations, a shift in intended management from curative to palliative care may result in no mortality reduction but result in improved patient quality of life. Changes in patient management after PET may also result in adverse outcomes for patients, especially if the imaging has low sensitivity or specificity for certain cancers and indications or if the altered course of patient management is ineffective or potentially risky.
The NOPR differs from the hierarchic model of diagnostic efficacy described by Fryback and Thornbury (13) because it is a compendium of observational data rather than a specific study designed to answer a specific question. The NOPR may be used to document that PET changes patient management and to generate hypotheses for future research. If PET is found to have a large impact on changes in patient management in the registry, then these findings can be validated in the CMS billing data, and if the agreement between reported changes in intended management and actual management is high, then the change in patient outcomes can be assessed for specific cancer types and indications.
CONCLUSION
Our multi-step analysis plan allows us to address current uncertainty about subsequent CMS decision making based on data collected under the CED program. The first “big picture” analysis will focus on the cancer indication, aggregating all cancers together. This approach may show sufficient impact of PET for CMS to make a universal coverage decision. If CMS requires a more narrowed focus, the Working Group has identified 15 high-impact combinations of cancer types and indications and an early assessment plan that evaluates the first 200 cases for each combination. By linking access to PET for virtually all Medicare beneficiaries to the collection of clinically valuable data, NOPR represents the cutting edge of CMS' new approach to payment determinations.
APPENDIX
NOPR Home Page (http://www.cancerPETregistry.org/)
NOPR Operations Manual (http://www.cancerpetregistry.org/pdf/nopr_opsman.pdf)
NOPR Forms (http://www.cancerpetregistry.org/forms.html)
Acknowledgments
We thank the many individuals at the Academy of Molecular Imaging, ACR, ACRIN, Society of Nuclear Medicine, American Society of Clinical Oncology, and CMS who have contributed to the development of the NOPR. We also thank Dr. Peter Bach, Dr. Constantine Gatsonis, Dr. Bruce Hillman, Dr. Louis Jacques, Dr. Barbara J. McNeil, and Charles Apgar for their critical review of the manuscript. Funding for development of the NOPR was provided by the Academy for Molecular Imaging; the registry is otherwise self-supported by the fees paid by participating PET facilities.
Footnotes
-
COPYRIGHT © 2007 by the Society of Nuclear Medicine, Inc.
References
- Received for publication May 17, 2007.
- Accepted for publication July 30, 2007.
Jump to section
Related Articles
Cited By...
- The Australasian Radiopharmaceutical Trials Network: Clinical Trials, Evidence, and Opportunity
- Hospice Admission and Survival After 18F-Fluoride PET Performed for Evaluation of Osseous Metastatic Disease in the National Oncologic PET Registry
- The Injustice of Being Judged by the Errors of Others: The Tragic Tale of the Battle for PET Reimbursement
- Impact on Patient Management of [18F]-Fluorodeoxyglucose-Positron Emission Tomography (PET) Used for Cancer Diagnosis: Analysis of Data From the National Oncologic PET Registry
- A Flexible, Multifaceted Approach Is Needed in Health Technology Assessment of PET
- Impact of 18F-Fluoride PET on Intended Management of Patients with Cancers Other Than Prostate Cancer: Results from the National Oncologic PET Registry
- Impact of 18F-Fluoride PET in Patients with Known Prostate Cancer: Initial Results from the National Oncologic PET Registry
- Intended Versus Inferred Care After PET Performed for Initial Staging in the National Oncologic PET Registry
- A Broader Importance of NOPR to the Future of Nuclear Imaging
- Impact of 18F-FDG PET Used After Initial Treatment of Cancer: Comparison of the National Oncologic PET Registry 2006 and 2009 Cohorts
- New Pathways to Medicare Coverage for Innovative PET Radiopharmaceuticals: Report of a Medical Imaging & Technology Alliance (MITA) Workshop
- Economic Evaluation of PET and PET/CT in Oncology: Evidence and Methodologic Approaches
- Economic Evaluation of PET and PET/CT in Oncology: Evidence and Methodologic Approaches
- PET and PET/CT Reports: Observations from the National Oncologic PET Registry
- Prospective Study of [18F]Fluorodeoxyglucose Positron Emission Tomography/Computed Tomography for Staging of Muscle-Invasive Bladder Carcinoma
- NCCN Task Force: Clinical Utility of PET in a Variety of Tumor Types
- Relationship Between Cancer Type and Impact of PET and PET/CT on Intended Management: Findings of the National Oncologic PET Registry
- Registries That Show Efficacy: Good, but Not Good Enough
- Impact of Positron Emission Tomography/Computed Tomography and Positron Emission Tomography (PET) Alone on Expected Management of Patients With Cancer: Initial Results From the National Oncologic PET Registry