We review current approaches to predicting tumor growth and treatment response that combine non-invasive imaging data with mathematical models of cancer progression, and propose some new directions for integrating quantitative imaging measurements with such numerical analyses. Historically, tumor modeling has been described by parameters that are measurable by invasive methods only or in isolated in vitro or ex vivo systems. This limits the practical usefulness of such models because it is not possible to test their predictions experimentally. Recent advances in three-dimensional magnetic resonance imaging, single photon emission computed tomography, and positron emission tomography techniques provide new opportunities to acquire measurements of relevant molecular and cellular features of tumors non-invasively and with high spatial resolution. Such data can be incorporated into mathematical models of tumors. We highlight some recent examples of this approach and identify several simple examples that allow for conventional mathematical models of tumor growth to be recast in terms of parameters that can be measured by imaging, thus raising the possibility of designing and constraining models that can be tested in clinical practice. It is our hope that this Perspective will stimulate further work in this evolving and exciting field.