Progressively growing GANs for realistic synthetic brain MR images

Objectives: Monte Carlo simulation is a useful tool in quantitative SPECT and PET imaging for the assessment of reconstruction algorithms, compensation methods, and quantification. A large digital phantom population with realistic anatomic variation is needed for such simulation studies. Anatomic information from MR images of patients can be used to generate realistic digital phantoms for PET and SPECT simulation. Generative adversarial networks (GANs) have shown promise for generating clinically realistic MR images (1). However, generating high resolution images is difficult due to the exploding gradient problem which is amplified during GAN training (2). We aim to use a progressively growing (PG)-GAN to generate synthetic brain MR images with realistic anatomy.

Methods: Data from 223 brain MR images of healthy controls and patients with various neurological disorders were first segmented using FreeSurfer, an open-source software package (3). The images were resampled to have a voxel size of 1 mm³ and spatial dimensions of 256³. The trans-axial cross-sections of these MR images containing striatal structures were used to train the approach resulting in a training set of 8,564 2D image slices. Images from the training set were resampled to have spatial dimensions of 4², 8², 16², 32², 64², 128² and 256². The PG-GAN consisted of generator and discriminator networks that were trained in an adversarial manner to generate synthetic MR images. The generator was first trained to produce images starting with a low resolution with spatial dimensions of 4². The generator and discriminator were progressively grown throughout training by adding new layers to the models as the generator was trained to provide images of increasing resolution with spatial dimensions of 4², 8², 16², etc. This progressively growing training scheme was used to reduce training time and improve training stability (2). The PG-GAN yielded the final generated images with spatial dimensions of 256². The PG-GAN was trained by minimizing a Wasserstein GAN loss function with gradient penalty using the Adam optimizer (2). The generated images were quantitatively evaluated with commonly used evaluation metrics, including Fréchet Inception Distance (FID) and multi-scale structural similarity index measure (MS-SSIM) (2). The FID measures the distance between the distribution of generated and real images where a lower value indicates higher quality images. The MS-SSIM was used to measure the structural similarity between pairs of generated and real images. The MS-SSIM ranges from 0 to 1 where a higher value indicates higher structural similarity. FID and MS-SSIM were computed on a random sample of 5,000 generated synthetic images and 5,000 real brain MRI images. As a baseline for comparison, the training set was randomly partitioned into two splits each with 4,282 images. The FID and MS-SSIM were measured between the two splits of training images.

Results: Representative examples of generated synthetic brain MR images are shown in each row of Figure 1A where each column corresponds to generated images with spatial dimensions of 4², 8², 16², 32², 64², 128² and 256², respectively, for each stage of training. Real MR images are shown in Figure 1B for comparison. The anatomic variation of the synthetic brain MR images generated by the PG-GAN are visually similar to that of real patient data. The PG-GAN yielded an FID of 61.42 and a MS-SSIM value of 0.50±0.09 between a random sample of 5,000 synthetic and real MR images. The FID and MS-SSIM values between sets of real images in the training set were 2.14 and 0.50±0.10, respectively. Results on FID are consistent with GANs trained on natural images. The histogram of MS-SSIM values between real image pairs is very closely matched to that of synthetic and real images pairs shown in Figure 1C.

Conclusions: The PG-GAN-based approach can generate synthetic brain MR images that are visually indistinguishable from real images providing a sufficient set of digital phantoms for PET and SPECT simulation.