We describe a comprehensive framework for performing rapid and automatic nonlabel-based nonlinear spatial normalizations. The approach adopted minimizes the residual squared difference between an image and a template of the same modality. In order to reduce the number of parameters to be fitted, the nonlinear warps are described by a linear combination of low spatial frequency basis functions. The objective is to determine the optimum coefficients for each of the bases by minimizing the sum of squared differences between the image and template, while simultaneously maximizing the smoothness of the transformation using a maximum a posteriori (MAP) approach. Most MAP approaches assume that the variance associated with each voxel is already known and that there is no covariance between neighboring voxels. The approach described here attempts to estimate this variance from the data, and also corrects for the correlations between neighboring voxels. This makes the same approach suitable for the spatial normalization of both high-quality magnetic resonance images, and low-resolution noisy positron emission tomography images. A fast algorithm has been developed that utilizes Taylor's theorem and the separable nature of the basis functions, meaning that most of the nonlinear spatial variability between images can be automatically corrected within a few minutes.