Patients with many common tumors are treated with chemotherapy despite limited evidence of treatment effectiveness. To determine if chemotherapy trials reporting effectiveness actually demonstrated increased survival in treated patients, we reviewed trials published over a two-year period involving four common solid tumors. Of 80 studies, 95% reported response to chemotherapy as an end point. Of 38 studies demonstrating 15% or greater objective response, 76% reported significantly greater survival of responders than of nonresponders. Of 21 studies containing statements supporting treatment effectiveness, 95% based this claim at least in part on the superior survival of responders compared with nonresponders. Because responders may have lived longer without treatment, such comparisons are not valid and may lead to overly optimistic views of chemotherapy effectiveness. Journal editors should be wary of allowing survival comparisons between responders and nonresponders in published reports.