Controlling the false positive rate for lp-ntPET: A correction to goodness of fit metrics for ''effective'' number of parameters

Objectives: Linear parametric neurotransmitter PET¹ (lp-ntPET) is a novel kinetic model that uses discrete basis functions to estimate the temporal characteristics of neurotransmitter (NT) release. The model contains seven total parameters: three describing tracer delivery—identical to the multilinear reference tissue model² (MRTM)—and four implicitly describing NT release through basis functions. Goodness of fit (GOF) metrics evaluate the significance of improvement in a fit caused by including the NT variables, over MRTM alone. These metrics expect precise knowledge of the number of parameters in lp-ntPET. However, the basis function implementation means that NT parameters cannot take on all values (do not span the entire parameter space). We assert that proper use of GOF metrics must be formulated using an ‘effective number of parameters’. We hypothesize that the effective number of parameters increases with number of basis functions. Here, we investigated how the size of the basis function library and measurement noise affected model selection. We also used model selection metrics to determine the “effective” number of parameters in a complex model for a stipulated false positive rate (FPR).

Methods: We performed null simulations of PET data using the MRTM model (mean values: R1=1, k2=0.42 min^-1, BP=3), which included no NT effects during the scan. We applied varying levels of measurement noise within the scan, ranging from noiseless to voxel-level. 10,000 time-activity curves (TACs) were simulated for each noise level, with 10% population variance in kinetic parameters across scans. All TACs were fitted with MRTM and lp-ntPET using 4 different basis function libraries containing 1, 12, 84, and 396 curves, respectively. Because all data were generated by MRTM, any significant improvement in fit by lp-ntPET must be overfitting. An F-statistic was computed from each pair of fits, assuming 7 full parameters in lp-ntPET. We compared the 95^th percentile F-statistic from all TACs to the theoretical F_critical threshold for p<0.05, assuming 7 full parameters. We determined FPR from the F_critical threshold, for each noise level and size of basis function library. To determine the “effective” number of parameters, we solved for p_lpntPET iteratively from the formulas for the Akaike information criterion (AIC) and Bayesian information criterion (BIC) such that lp-ntPET was selected as the superior model 5% of the time (FPR=5%). Mathematically: ∑[GOF(WSSR_MRTM, n, p_MRTM) - GOF(WSSR_lpntPET, n, p_lpntPET) ≥ 0] ÷10000 = 0.05 — where GOF is BIC or AIC; WSSR is weighted sum of squared residuals produced by a fit; n is number of frames; p is (effective) number of parameters in a model.

Results: The FPR calculated from F-values ranged from 7-31%, increasing with noise and number of bases. The 95^th percentile of calculated F-values was consistently greater than the F_critical threshold (p<0.05) assuming 7 lp-ntPET parameters (Fig. A). For both AIC and BIC, the “effective” number of parameters increased with more noise and more basis functions (Fig. B). Discussion: When fitting null data, we expect a 5% FPR if the F_critical threshold is set at p<0.05; that is: lp-ntPET should emerge as the superior model by chance for 5% of fitted curves. Our finding of a consistently higher FPR suggests the need for a more stringent threshold when using the F-test with lp-ntPET. Further, we demonstrate that in order to properly use GOF metrics as a means of model selection, one must determine the “effective” number of parameters, which may not simply be equal to the number of parameters in the model. For models using a basis function implementation, the number of bases must be considered as well as the noise. Our results caution against naïve application of model selection criteria when considering models implemented with discrete numbers of basis functions.

• Download figure
• Open in new tab
• Download powerpoint