Invasive lobular carcinoma (ILC) has been reported to be less responsive to neoadjuvant chemotherapy (NAC) than invasive ductal carcinoma (IDC). We sought to determine whether ILC histology indeed predicts poor response to NAC by analyzing tumor characteristics such as protein expression, gene expression, and imaging features, and by comparing NAC response rates to those seen in IDC after adjustment for these factors. We combined datasets from two large prospective NAC trials, including in total 676 patients, of which 75 were of lobular histology. Eligible patients had tumors ≥3 cm in diameter or pathologic documentation of positive nodes, and underwent serial biopsies, expression microarray analysis, and MRI imaging. We compared pathologic complete response (pCR) rates and breast conservation surgery (BCS) rates between ILC and IDC, adjusted for clinicopathologic factors. On univariate analysis, ILCs were significantly less likely to have a pCR after NAC than IDCs (11 vs. 25 %, p = 0.01). However, the known differences in tumor characteristics between the two histologic types, including hormone receptor (HR) status, HER2 status, histological grade, and p53 expression, accounted for this difference with the lowest pCR rates among HR+/HER2- tumors in both ILC and IDC (7 and 5 %, respectively). ILC which were HR- and/or HER2+ had a pCR rate of 25 %. Expression subtyping, particularly the NKI 70-gene signature, was correlated with pCR, although the small numbers of ILC in each group precluded significant associations. BCS rate did not differ between IDC and ILC after adjusting for molecular characteristics. We conclude that ILC represents a heterogeneous group of tumors which are less responsive to NAC than IDC. However, this difference is explained by differences in molecular characteristics, particularly HR and HER2, and independent of lobular histology.