Glutamate carboxypeptidase II (GCPII, EC 3.4.17.21) is a zinc metallopeptidase that hydrolyzes N-acetylaspartylglutamate (NAAG) into N-acetylaspartate (NAA) and glutamate in the nervous system. Inhibition of GCPII has the potential to reduce extracellular glutamate and represents an opportune target for treating neurological disorders in which excess glutamate is considered pathogenic. Furthermore, GCPII was found to be identical to a tumor marker, prostate-specific membrane antigen (PSMA), and has drawn significant interest as a diagnostic and/or therapeutic target in oncology. Over the past 15 years, tremendous efforts have been made in the discovery of potent GCPII inhibitors, particularly those with phosphorus-, urea- and thiol-based zinc binding groups. In addition, significant progress has been made in understanding the three-dimensional structural characteristics of GCPII in complex with various ligands. The purpose of this review article is to analyze the structure-activity relationships (SAR) of GCPII inhibitors reported to date, which are classified on the basis of their zinc-binding group. SAR and crystallographic data are evaluated in detail for each of these series to highlight the future challenges and opportunities to identify clinically viable GCPII inhibitors.