It remains unclear whether uncontrolled epilepsy causes mental decline. This longitudinal study contrasts change of memory and nonmemory functions in 147 surgically and 102 medically treated patients with temporal lobe epilepsy. All participants were evaluated at baseline (T1) and after 2 to 10 years (T3). Surgical patients underwent additional testing 1 year postoperatively (T2). Data were analyzed on an individual and group level. Sixty-three percent of the surgical and 12% of the medically treated patients were seizure-free at T3. Fifty percent of the medically treated and 60% of the surgical patients showed significant memory decline at T3 with little change in nonmemory functions (difference not significant). Surgery anticipated the decline seen in the medically treated group and exceeded it when surgery was performed on the left, or if seizures continued postoperatively. Seizure-free surgical patients showed recovery of nonmemory functions at T2 (p < 0.001) and of memory functions at T3 (T3, p = 0.03). Multiple regression indicated retest interval, seizure control, and mental reserve capacity as predictors of performance changes. In addition, psychosocial outcome was better when seizures were controlled. In conclusion, chronic temporal lobe epilepsy is associated with progressive memory impairment. Surgery, particularly if unsuccessful, accelerates this decline. However, memory decline may be stopped and even reversed if seizures are fully controlled.