Figures
Abstract
Gastroenteropancreatic (GEP) neuroendocrine neoplasms (NENs) are increasing in both incidence and prevalence. A delay in correct diagnosis is common for these lesions. This reflects the absence of specific blood biomarkers to detect NENs. Measurement of the neuroendocrine secretory peptide Chromogranin A (CgA) is used, but is a single value, is non-specific and assay data are highly variable. To facilitate tumor detection, we developed a multi-transcript molecular signature for PCR-based blood analysis. NEN transcripts were identified by computational analysis of 3 microarray datasets: NEN tissue (n = 15), NEN peripheral blood (n = 7), and adenocarcinoma (n = 363 tumors). The candidate gene signature was examined in 130 blood samples (NENs: n = 63) and validated in two independent sets (Set 1 [n = 115, NENs: n = 72]; Set 2 [n = 120, NENs: n = 58]). Comparison with CgA (ELISA) was undertaken in 176 samples (NENs: n = 81). 51 significantly elevated transcript markers were identified. Gene-based classifiers detected NENs in independent sets with high sensitivity (85–98%), specificity (93–97%), PPV (95–96%) and NPV (87–98%). The AUC for the NEN gene-based classifiers was 0.95–0.98 compared to 0.64 for CgA (Z-statistic 6.97–11.42, p<0.0001). Overall, the gene-based classifier was significantly (χ2 = 12.3, p<0.0005) more accurate than CgA. In a sub-analysis, pancreatic NENs and gastrointestinal NENs could be identified with similar efficacy (79–88% sensitivity, 94% specificity), as could metastases (85%). In patients with low CgA, 91% exhibited elevated transcript markers. A panel of 51 marker genes differentiates NENs from controls with a high PPV and NPV (>90%), identifies pancreatic and gastrointestinal NENs with similar efficacy, and confirms GEP-NENs when CgA levels are low. The panel is significantly more accurate than the CgA assay. This reflects its utility to identify multiple diverse biological components of NENs. Application of this sensitive and specific PCR-based blood test to NENs will allow accurate detection of disease, and potentially define disease progress enabling monitoring of treatment efficacy.
Citation: Modlin IM, Drozdov I, Kidd M (2013) The Identification of Gut Neuroendocrine Tumor Disease by Multiple Synchronous Transcript Analysis in Blood. PLoS ONE 8(5): e63364. https://doi.org/10.1371/journal.pone.0063364
Editor: Yuan-Jia Chen, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, China
Received: January 24, 2013; Accepted: April 1, 2013; Published: May 15, 2013
Copyright: © 2013 Modlin et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Funding for this work was supplied by Clifton Life Sciences. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: We received funding from a commercial source: Clifton Life Sciences. This does not alter our adherence to all the PLOS ONE policies on sharing data and materials.
Introduction
Although previously considered rare, gastroenteropancreatic neuroendocrine neoplasms (GEP-NENs) are common (incidence: 3.6/100,000), occurring as frequently as testicular tumors, Hodgkin’s disease, gliomas and multiple myeloma [1] and are estimated to have a prevalence of 35/100,000 [2]. They represent a significant clinical issue since 50–70% are metastatic at diagnosis and there is a paucity of effective therapy. Two common agents, everolimus and sunitinib, only increase progression free survival by ∼6 months, while somatostatin analogs have a marginal impact. The lack of sensitive and robust biomarkers to establish diagnosis, assess disease progress and monitor treatment efficacy has been identified as key unmet needs [3].
Strategies including staging at surgery, pathological grading, blood Chromogranin A (CgA) measurements, detection of circulating tumor cells (CTCs) or other products e.g. serotonin are currently used [1]. Their utility is highly variable and often insensitive for small tumors or metastasis detection, may require tissue and depends on non-standardized tests. Despite that CgA has been proposed as a marker of disease and tool for evaluating treatment efficacy [4], it is not FDA-accepted as a supportable biomarker [5]. This reflects limitations in sensitivity, specificity and reproducibility.
Identification of a peripherally accessible, molecular fingerprint using PCR-amplification of target genes, has successfully been undertaken in other cancers e.g., breast and colon. In the former, this is used in prognosis, identification of metastasis and recurrence, prediction of therapy response and metastasis-free survival for node-negative, untreated primary cancers [6], [7]; for the latter, utility has been determined for staging [8]. We report the initial assessment of our hypothesis that a neoplasia-associated circulating signature is identifiable in GEP-NENs and can be used to accurately identify disease. We have previously evaluated tissue-derived gene markers for GEP-NENs [9]–[11] and demonstrated their utility for detecting NEN malignancy [12]. In this study, we extended this strategy developing a blood-based PCR test using REMARK (REporting of tumor MARKer studies) criteria [13] to detect circulating mRNAs that facilitate GEP-NEN diagnosis and management.
Materials and Methods
Detailed methods are available in the online supplement including computational analyses, collection methodology, sampling and handling. All samples were collected and analyzed according to an IRB protocol (Yale University School of Medicine). The protocol was specifically approved for this study. Written consent was obtained from all study participants.
In silico Identification of 51 Marker Genes
Human cancer and normal tissue microarray datasets were obtained (ArrayExpress database [14]). Two GEP-NEN gene expression datasets were analyzed (GEP-NEN-A, GEP-NEN-B). The former included small intestinal tissue (n = 3; macroscopically normal mucosa collected at surgery), primary GEP-NENs (n = 6), and metastatic GEP-NENs (n = 3) [15]; the latter, normal ileal mucosa (n = 6), primary midgut NENs (n = 3), and liver metastases (n = 3) [16]. Additionally, a compendium of public cancer microarray datasets (three hepatocellular carcinoma (HCC) datasets [Alcohol-HCC (n = 65 arrays), Viral-HCC (n = 124 arrays), and Progression-HCC (n = 75 arrays)]; breast (n = 86 arrays, colon (n = 47 arrays, and prostate (n = 154 arrays) cancer profiles and normal human tissue arrays (n = 158 arrays)] were analyzed (Table S1).
Thereafter, we examined gene expression in peripheral blood. For this, fourteen samples (controls: n = 7; GEP-NENs: n = 7) were examined. Samples were processed using the Applied Biosystems Tempus Spin RNA Isolation t Kit (RNA quality >1.8 A260∶280 ratio, RIN>7.0). RNA was hybridized on Affymetrix platforms [11], [12] and analysis performed as described previously [17].
Details of the microarray analyses and gene identification including normalization using Robust Multi-array Average (RMA) [18], identifying probes and mapping to Ensembl gene identifiers, assessment of gene co-expression network inferences, network partitioning and functional enrichment analyses [19] are included in the Supplemental methods. This computational approach (Figure 1, Figure S2) resulted in the identification of 75 candidate genes. Preliminary screening detected 51 marker genes which were then included in the current study.
Step 1: Gene co-expression networks inferred from GEP-NEN-A and GEP-NEN-B datasets are intersected, producing the GEP-NEN network. Step 2: Co-expression networks from neoplastic and normal tissue microarray datasets are combined to produce the normal and neoplastic networks. Step 3: Links present in normal and neoplastic networks are subtracted from the GEP-NEN network. Step 4: Concordantly regulated genes in GEP-NEN-A and GEP-NEN-B networks are retained; other genes are eliminated from the GEP-NEN network, producing the Consensus GEP-NEN network. Step 5: Upregulated genes in both the GEP-NEN-A and GEP-NEN-B dataset are mapped to the Consensus GEP-NEN network. Step 6: Topological filtering, expression profiling, and literature-curation of putative tissue-based markers, yielding 21 putative genes further examined by RT-PCR. Step 7: Identification of mutually up-regulated genes in GEP-NEN blood transcriptome and GEP-NEN-A and GEP-NEN-B datasets, yielding 32 putative genes further examined by RT-PCR. Step 8: Literature-curation and cancer mutation database search, yielding a panel of 22 putative marker genes for further RT-PCR analysis.
RNA Isolation and cDNA Synthesis from Peripheral Blood
Training set: Transcripts (mRNA) were isolated from 130 blood samples (controls: n = 67; GEP-NENs: n = 63) using the mini blood kit (Qiagen: RNA quality >1.8 A260∶280 ratio, RIN>5.0) with cDNA produced using the High Capacity Reverse transcriptase kit (Applied Biosystems: cDNA production 2000–2500ng/ul).
Independent Validation sets: Two sets were used: the first included 115 samples (controls: n = 43; GEP-NENs: n = 72), and the second, 120 samples (controls: n = 49; GEP-NENs: n = 71) (Table 1). The clinical characteristics are included in Table 2. Small intestinal and pancreatic tumors (67–86%) were predominant, lesions were grade 1 and 2 (Ki67≤20%) (76–89%) and metastases (73–85%) were prevalent. Patients were older than controls, and sex-matching was not undertaken.
Real-time PCR Analysis of Peripheral Blood Gene Expression
Real-time PCR was performed using Applied Biosystems products (details in Supplemental Methods). PCR values were normalized to ALG9 (ΔΔCT) [15], using the control group as the population control (calibrator sample).
Chromogranin A Measurement
CgA was measured using the DAKO ELISA kit (K0025, DAKO North America, Inc., Carpinteria, CA) [20] in a set of 176 samples (controls: n = 95; GEP-NENs: n = 81). A cut-off of 19 Units/L (DAKO) was used as the upper limit of normal.
Classification Algorithms
Expression values were log-transformed and mapped to the range (1–100). GEP-NEN classifiers were built and optimized on the training set (n = 67 controls, n = 63 GEP-NENs) using 10-fold cross-validation design. In the internal training set, differentially expressed genes (control versus tumor) were calculated by a t-test. Four different learning algorithms [support vector machine (SVM), linear discrimination analysis (LDA), K-Nearest Neighbor (KNN), and Naive Bayes (Bayes)] were trained on the internal training set using the up-regulated features (uncorrected p<0.05). To control for over-fitting, the classifier was verified in 2 validation sets. A consensus labeling of “control” or “GEP-NEN” was generated by a “majority vote” approach [21], whereby a sample with <2 “control” predictions was designated as “GEP-NEN”. Detailed description of all classification algorithms is in Supplementary Methods.
All analyses were carried out using MATLAB’s Statistics and Bioinformatics toolboxes (2009a, The MathWorks, Natick, MA).
Results
Pipeline for Identifying and Defining Candidate Genes in GEP-NENs
1. Gene co-expression network inference in GEP-NENs.
We hypothesized that comparison of co-expression networks between GEP-NEN and other cancer datasets would provide additional biological insight. We utilized two independent GEP-NEN microarray datasets [15], [16] and compared them with well-characterized cancer datasets chosen for prevalence and represented by comprehensive microarray collections. Additionally, an independent normal human tissue dataset (79 different healthy tissues and cell types [2 replicates/tissue/cell type including liver, brain and heart, totaling 158 arrays] was included to eliminate co-expressions that may occur due to healthy tissue in malignant biopsies (Table S1).
Gene co-expression networks were constructed for all microarray datasets by linking genes whose expression correlated above a predefined PCC threshold (Supplementary Methods, Figure S1). Subsequently, the inference of a GEP-NEN network consisted of: 1) retaining co-expression pairs that recurred in both GEP-NEN datasets; 2) eliminating genes and co-expressions present in other cancer and normal tissue gene networks from the consensus GEP-NEN network; and 3) eliminating genes from the consensus GEP-NEN network that exhibited divergent changes in GEP-NEN-A and GEP-NEN-B datasets (Figure 1, Figure S2). This analysis produced 2892 genes and 30444 co-expressions. We focused on the largest connected component of this network (2545 genes and 30249 links), which contained 99% of all co-expressions (Figure 2A). It is important to note that a gene co-expression network does not attempt to identify “direct gene interactions” but rather contain “gene neighborhood relations” that are usually overlooked in conventional microarray analysis [22] and is used to identify genes that play distinct roles in a common pathway or biological process [23]. Therefore, functional characterization of a co-expression network should be regarded as a descriptive analysis aimed to generate additional testable hypotheses.
A) Visualization of the GEP-NEN gene co-expression network (2545 genes, 30249 edges). Each node represents a gene, while a link represents a GEP-NEN-specific co-expression. Nodes that localized to the same network community are marked in the same color. The community structure of the GEP-NEN network is further visualized in the 3 dimensional inset, whereby each node represents a community while edges are drawn between communities that contain co-expressed genes. Larger nodes indicate bigger gene communities. B) Heatmap visualizing enrichment for over-represented Gene Ontology (GO) Biological Process (BP) terms assigned to the 10 largest clusters (>20 genes). Heatmap colors represent the significance of the enrichment.
2. Functional analysis of GEP-NEN gene co-expression network.
To provide insight into molecular pathways captured by the GEP-NEN network, the DAVID functional enrichment tool was used to identify over-represented Biocarta, KEGG, and Reactome pathways (see Supplemental Methods). The most abundant pathways were Reactome pathways including “Integration of energy metabolism” (n = 58 genes, p = 4.2×10−5) and the “Diabetes pathway” (n = 68, p = 2.7×10−4), and KEGG pathways like “Pathways in cancer” (n = 72, p = 0.003) (Table S2). Other pathways included genes involved in immune responses, nervous system development, and metabolism. An important characteristic of most biological networks is that they tend to naturally organize into modules. We used the Louvain algorithm, a “greedy” method for iterative grouping of nodes into communities through modularity maximization [24], to partition the GEP-NEN network into 62 clusters with 800 and 3 genes in the largest and smallest clusters respectively (Figure 2A). Enrichment for over-represented GO-BP terms in clusters with >20 genes, revealed presence of processes including “Apoptosis” (p = 2.9×10−26, Cluster 1), “Oxidation reduction” (p = 2.3×10−36, Cluster 2), and “Nervous system development” (p = 7×10−20, Cluster 4) (Figure 2B). These processes are consistent with the known biology of GEP-NENs [25].
3. Marker gene selection.
We generated three panels of putative marker genes that were further examined by RT-PCR: 1) tissue-based panel, 2) peripheral blood-based panel and, 3) literature-curated panel. A detailed description of the methods is in Supplementary Methods.
To generate the tissue-based gene panel, we identified significantly (false discovery rate [FDR] adjusted p<0.025) up-regulated genes in both GEP-NEN-A and GEP-NEN-B datasets and retained only genes that were also present in the GEP-NEN gene co-expression network. Subsequently, we retained genes with high network clustering coefficient (≥0.25), based upon their increased likelihood of an association with tumorigenesis [26]. Finally, we examined a set of 369 genes that passed our filtering threshold using a manual literature-curated search. Our search criteria involved implication in: a) neuroendocrine axis, b) tumor formation, or c) metastasis. Using these constraints, 21 of the 369 “putative” marker genes were selected for PCR validation.
To derive a peripheral blood-based “putative” marker gene panel, we generated a transcriptome consisting of 14 peripheral blood samples (n = 7 controls, n = 7 GEP-NENs). There were 1382 significantly up-regulated (unadjusted p<0.05, FC>0) genes in GEP-NENs (details in Supplemental Methods). All genes with expression values in the lower 25th quantile were excluded and only those genes with positive FC in both tissue datasets (GEP-NEN-A/B) were retained. This analysis produced 306 “putative” marker genes. A manual literature-curated search focusing on relevance to neuroendocrine biology/neoplasia identified 32/306 as putative targets for PCR validation.
The literature-curated panel consisted of 22 genes. Thirteen marker genes previously associated with GEP-NENs, either in our studies [11], [12] or in others [16], [27], were identified using queries of the Catalogue of Somatic Mutations in Cancer (COSMIC v60) database [28]. The additional 9 genes were included given their association with tumor initiation and metastasis.
Thus, based upon these analyses, 75 “putative” marker genes were selected for PCR analysis (Figure 1, Figure S2).
Validation of GEP-NEN Marker Gene Panel in Test Set and Independent Sets
To validate a “putative” marker panel, transcript levels of mRNA isolated from a subset of the training set (controls: n = 49 and GEP-NENs: n = 28) was measured. This identified that 51 of the 75 candidate markers produced detectable product (CT<40 cycles) in blood. The 51 gene panel is listed (Table 3, Table S3).
1. Utility of the 51 marker panel to identify GEP-NENs.
The GEP-NEN classifiers were built on a training set (controls: n = 67, GEP-NENs: n = 63) and significantly up-regulated features between control and tumor cases were calculated by t-test (n = 27, p<0.05, Figure 3A). Four classification algorithms (SVM, LDA, KNN, and Bayes) and a 10-fold cross-validation design were used to build a classifier for the diagnosis of GEP-NENs. The average accuracy of the SVM, LDA, KNN, and Bayes algorithms to distinguish GEP-NEN from control samples using 27 genes was comparable –0.89 (0.85–1.0), 0.89 (0.86–0.93), 0.88 (0.85–0.93), and 0.86 (0.85–0.93) respectively. The “majority voting” combination of the four classifiers achieved an accuracy of 0.88 (Figure 3B). To control for over-fitting and to evaluate classifier performance, we examined two validation sets (see Methods). The “majority vote” classification was used to generate final predictions. In these validation sets, the performance metrics for differentiating GEP-NENs from controls exhibited sensitivities of 85–98% with specificities of 93–97%, PPVs of 95–96% and NPVs of 87–98%. The AUC for the diagnostic test in first and second validation sets were 0.98 and 0.95 respectively (Figure 3C). These results indicate the signature was effective at distinguishing between GEP-NENs and controls.
A) Unsupervised hierarchical clustering of the 130 samples in the training set (n = 67 controls, n = 63 GEP-NENs). Tree was generated with an average agglomeration method and 1-(sample correlation) was used as a measure of dissimilarity. Unique colors under the dendrogram represent sample cluster assignments, computed by cutting the hierarchical tree at height = 0.99 (black line), 0.85 (blue line), or 0.50 (red line) using a dynamic tree cutting approach [77]. B) Prediction accuracy of each classifier using sequential addition of 27 significantly up-regulated genes (p<0.05) in the GEP-NEN samples obtained using results of the 10-fold cross validation. C) Receiver operating characteristic (ROC) curves for “majority vote” classifier applied to validation sets 1 (AUC = 0.98, p<0.0001) and 2 (AUC = 0.95, p<0.0001) compared to ROC curve for utility of the plasma CgA values (AUC = 0.64, p<0.002). Direct comparisons of AUCs between set 1 or set 2 and CgA identified estimated Z-scores of 10.57 and 11.42 respectively, confirming the significant differences between the two detection systems (calculations detailed in Supplementary Methods).
2. Comparison of the 51 marker panel with Chromogranin A for GEP-NEN identification.
To examine the utility of the peripheral blood PCR signature, we compared it to measurements of CgA in a set (n = 176 samples). Levels of CgA were elevated (p<0.002) in GEP-NENs compared to controls (Figure 4A). Using the DAKO cut-off of 19 Units/L as the ULN, a total of 26 (32%) of 81 GEP-NENs were positive compared to 1 (1.0%) of 94 controls for performance metrics of 32% (sensitivity), 99% (specificity), 96% (PPV) and 63% (NPV). The correct call rate was 68%. A direct comparison of the molecular test and CgA ELISA identified that the PCR-based method had a significantly more accurate call rate compared to CgA levels (χ2 = 12.3, p<0.0005) (Figure 4B). The specificities were similar for detecting a GEP-NEN (94% versus 99%) but the sensitivity of the PCR test was significantly higher than for CgA (85% versus 32%).
A) CgA levels were significantly elevated in the GEP-NEN group (n = 176; *p<0.002) but an overlap with normal values was identified. B) Comparison of the PCR-based approach with CgA protein measurement identified that call rates were significantly higher for the PCR-based test (*p<0.0005, χ2 = 12.3). The PCR blood test was significantly more accurate than measurement of CgA levels to detect GEP-NENs. ULN = upper limit of normal (19U/L – DAKO).
Additional Utility of GEP-NEN Marker Gene Panel
To further evaluate the potential utility of this marker panel, we undertook a sub-analysis of the data to examine whether there were any differences in sensitivity or specificity for detecting P-NENs versus GI-NENs and whether non-metastatic tumors could be detected. In addition, we wanted to determine how well the test performed in the patients with low CgA expression. We examined each of the validation sets (independent set 1 and 2) individually as well as the combination of the two sets.
The performance metrics for identifying P-NENs were: sensitivity 64–100% and specificity 92–95%; overall 79% of the 43 pancreas NETs (in both sets) were positive by the test (specificity: 94%) (Figure 5A). For GI-NENs, this was 74–98% and 92–95%, respectively. Overall, 88% of the 95 GI-NENs (both sets) were positive (specificity: 94%). There was no significant difference (Chi-square = 1, p = 0.31, 2-tailed) indicating that the PCR test could identify these two tumor types with a similar efficacy.
A) The sensitivity and specificity of the test to detect GI-NENs (90%, 94%) and P-NENs (80%, 94%) was similar. B) The PCR-based approach could detect patients with no metastases as well as patients with metastases. C) The PCR-based test could accurately identify GEP-NENs even when plasma CgA were low (<19U/L). Overall, the PCR blood test was significantly more accurate than measurement of CgA levels to detect GEP-NENs (*p<10−13, χ2>50).
Assessment of tumors with metastases identified an overall sensitivity and specificity of 85% (specificity: 94%) while 91% of the 11 documented patients with no metastases were positive (specificity: 94%). The PCR test therefore identified patients equally well irrespective of metastases (Figure 5B).
Using the 176 sample dataset for CgA and the DAKO cut-off of 19U/L, 55 patients were identified with low circulating levels of CgA. The PCR score in these patients was >2 in 50 (91%). For the 26 patients with elevated CgA, the PCR score was elevated in 22 (85%). Using diagnosis of GEP-NEN as a “standard”, the PCR score significantly outperformed measurements of CgA (Chi-square: >50, p<10−13) for the identification of the disease (Figure 5C).
Discussion
We have developed and validated a PCR-based, blood-derived, molecular signature test that is based on 51 genes and identifies GEP-NENs with a high specificity and sensitivity. This test significantly outperforms the current CgA blood test that is used to confirm the clinical suspicion of a NEN. Since the blood PCR signature comprises 51 NEN-based transcripts that cover a wide biological spectrum, it is also more effective than a single peptide-based ELISA that identifies a secretory peptide unrelated to tumor cell proliferation and not produced by ∼25% of NENs [29], [30]. Such a multi-transcript approach is generally more effective than single parameter analyses [31], [32].
A key limitation of CgA measurement is that it only measures one variable of NENs namely a secretory peptide and the ELISA technique used is based on a number of different antibodies used by various commercial laboratories (e.g. Cisbio, DAKO or Eurodiagnostica). Measurements are thus not only mono-dimensional but not readily comparable if different assays utilizing different antibodies are used [29], [30], [33]. To ensure a broader biological coverage and diminish reliance on a single variable, we developed a multiple parameter PCR test.
A compendium of tissue-based and peripheral blood transcriptomes was used to develop a signature which exhibited GEP-NEN specificity and was biologically related to GEP-NENs. To generate a rational basis for integrating multiple transcripts a series of mathematical algorithms were utilized to derive the marker signature, namely the GEP-NEN classifier. These included gene co-expression network profiling and functional gene community detection, all robust methods previously used in the development of gene-based molecular protocols [34]. Experimental artifact was minimized and robustness amplified through the use of two independent GEP-NEN microarray datasets and seven normal and neoplastic tissue transcriptomes (total 551 arrays). To further assure the biological relevance of the analysis functional enrichment of genes associated with GEP-NENs (inclusion of GO-BP terms such as “Chromatin organization”, “Negative regulation of gene expression”, and “Cell surface receptor linked signal transduction” [25] [e.g., chromogranin A/B (CHGA/CHGB: secretion], glutamate decarboxylase 1 [GAD1: GABA production] [25] and Aurora kinase B [AURKB: mitosis] [35] was undertaken.
Since a key component of accuracy was dependent on accurate and reproducible mathematical analysis we utilized supervised learning methods, SVM, LDA, KNN, and Bayes to build the GEP-NEN classifier. These strategies have previously been used as broad applications in two-class classification problems in biomedicine. SVM has been utilized to predict grading in astrocytomas [36] (>90% accuracy), and prostatic carcinomas (74–80% accuracy) [37]. LDA can detect non-small cell lung cancer in peripheral blood [38], while KNN models have been used to predict outcome in neuroblastoma [39]. The Bayes classifier has been used to predict prostate cancer recurrence [40]. Each therefore has utility for identifying individual or multi-variable alterations in neoplasia. Combining these techniques with a “majority vote” strategy in two independent validation cohorts, the PCR-based test exhibited correct call rates of 91–97% with sensitivities and specificities of 85–98% and 93–97% respectively for the identification of GEP-NENs. These performance metrics are comparable to similar algorithms that were successfully used clinically to detect CTCs e.g. cutaneous T-cell lymphoma (90%) [41].
To assess the efficacy of this signature index, we then compared it to CgA which is the current NEN marker used to establish diagnosis and disease status [30], [42]–[46]. CgA elevations are considered a sensitive, ∼60–85% accurate marker for GEP-NENs [1]. Measurements are, however, non-specific (10–35%) since CgA is elevated in a wide variety of diverse conditions [30], [33], [43]. These include non endocrine neoplasia (pancreatic and prostate) and a wide variety of cardiac, endocrine and inflammatory diseases [47], as well as in patients undergoing acid suppressive therapy with the proton pump inhibitor (PPI) class of drugs [48] and in renal failure [49]. CgA is constitutive component of neuroendocrine secretion, not proliferation, and therefore its use as a surrogate marker for tumor growth has obvious limitations [1]. In the current study we compared the PCR test with a widely available commercial CgA kit (DAKO: K0025) [30], [33]. Values were, as expected, elevated in GEP-NENs but exhibited a significant overlap with controls with an accuracy of 60% and sensitivity of 32%. It is likely that use of other kits to measure CgA would generate similar numbers given their published concordance (∼40–70%) [30], [33]. In comparison, the PCR-based test exhibited a sensitivity of >85% with a correct call rate of >90%. Evaluation of the ROCs was similarly significantly effective for the PCR-based test, which exhibited an AUC of 0.95–0.98 compared to 0.64 for the CgA. AUCs for CgA have ranged as high as 0.8–0.9 in other studies [50], [51], but this is dependent both on the kits used, the patient inclusion criteria e.g., undergoing treatment or type of GEP-NEN, but most importantly, the cut-off chosen, which is often population-dependent [30]. In comparison to other molecular-based tests, the performance metrics for the NEN-PCR-based test are substantially higher than for prostate (PSA or PMSA (0.75, both single target test) [52] or colon cancer (0.51–0.72, a two target PCR test) [53]. Given the utility of these latter cancer tests in clinical management [54], it is probable that application of this PCR multi-transcript measurement strategy to GEP-NENs will be similarly effective.
It has been noted that the majority of biomarker studies may not translate into clinically relevant tests [55]. For example, peripheral blood screens for colorectal cancer are not routine practice [56]. This is paradoxically associated with the sensitivity of PCR per se. Substantial differences in final yield can occur if there are minor variations in reaction components and thermal cycling conditions and/or mispriming events during PCR [57]–[61]. To minimize these potential issues, we have chosen to use a TAQMAN approach. In other studies, this has been demonstrated to have a low variability between runs ranging between 0–5% [62], have small coefficient of variations (CVs) for the cycling threshold (CT) of 1–3% [63] and results in acceptable CVs for normalized data between 10–24% [64], [65]. A consistent protocol for RNA isolation, cDNA synthesis and real-time PCR is considered appropriate to provide a stable platform for target and housekeeping gene analyses [57], [62], [63], [66], [67]. Stringent quality control [68], standardization of sample acquisition [69] and processing [70] therefore are a prerequisite for use of this molecular tool which makes it likely that any PCR approach will require dedicated, specialized facilities.
Irrespective of the potential limitations, our study identifies that a PCR-based test is significantly more sensitive than that currently utilized, namely CgA measurements, and can detect the majority (∼95%) of patients with disease irrespective of the location, extent, grade or metastasis. It is therefore likely that the test would be useful in a number of areas, following appropriate study. One is as a “rule-out” diagnostic test (to confirm absence of a GEP-NEN or residual disease). The low incidence of GEP-NENs in the population makes it unlikely to be cost-effective as a screening tool for tumor detection. The high sensitivity of the PCR test, in contrast, renders it a more effective tool to rule out a diagnosis. This will eliminate the relatively large number of “borderline” abnormal CgA results, particularly when different types of kits are used. Any future studies, should, in addition, assess whether medications or conditions associated with non-specific elevations in neuroendocrine cell numbers, e.g., PPIs, increase transcript expression. Given the similarities in biology (i.e., expression of receptors, pathways involved in secretion, molecular pathways e.g., MEN-I) [71]–[73] between GEP-NENs and other NENs e.g., pheochromytomas or medullary thyroid cancers, it would be useful to assess whether the PCR test can accurately identify these lesions. The existence of tumors with a significant neuroendocrine component e.g., prostate tumors [74] or colorectal cancers [75], [76], provides additional clinical samples in which to evaluate the efficacy of the PCR test.
Currently, CgA is used to evaluate treatment protocols [4], [46] as expression levels are considered to relate to tumor burden [46]. However, issues remain with the use of different measurement protocols as well as how to accurately assess CgA in monitoring disease if values are low or within the normal range. Given the high rate of detection even when plasma CgA levels are low (91% of these samples could be accurately identified by the PCR test), we anticipate that the PCR test can potentially be used as a prognostic. Future studies examining whether the PCR test results alter in response to therapy e.g., debulking or targeted therapy, would answer this possible indication.
In conclusion, using computational and machine learning approaches, including analysis and integration of tumor tissue and circulating peripheral blood transcripts, we identified a panel of 51 marker genes selectively associated with GEP-NENs. The test can differentiate between GEP-NENs and controls and has a high PPV and NPV (>90%). It is more accurate than the currently used clinical standard CgA assay, which identifies a single peptide related only to tumor secretion. The PCR-based signature measures multiple transcripts which reflect the diverse biological profile of a proliferating NEN and may, with further examination in appropriate studies, be tested as a measure of tumor responsiveness and, potentially, as a prognostic.
Supporting Information
Figure S1.
Selection of Pearson correlation coefficient thresholds for gene co-expression network inference.
https://doi.org/10.1371/journal.pone.0063364.s001
(TIF)
Figure S2.
Computational pipeline used to derive a set of 51 markers that identify GEP-NEN disease.
https://doi.org/10.1371/journal.pone.0063364.s002
(TIF)
Table S1.
Microarray Datasets used in GEP-NEN network analysis.
https://doi.org/10.1371/journal.pone.0063364.s003
(DOCX)
Table S2.
Functional enrichment of 2545 genes in the GEP-NEN network for Biocarta, KEGG, and Reactome pathways.
https://doi.org/10.1371/journal.pone.0063364.s004
(DOCX)
Table S3.
List of 51 marker genes used in the study.
https://doi.org/10.1371/journal.pone.0063364.s005
(XLS)
Acknowledgments
We would like to acknowledge the following for providing samples and clinical data for analysis: M. Pavel (Charité, Berlin; n = 48), M. Banck (Mayo Clinic, Rochester, MN; n = 56), D. Alaimo for technical support in sample processing, and B. Lawrence, S. Schimmack, B. Svejda for clinical analyses and constructive discussion. ID is currently an employee at Bering Ltd. The company provided no financial or material support for this study.
Author Contributions
Conceived and designed the experiments: IMM ID MK. Performed the experiments: ID MK. Analyzed the data: IMM ID MK. Contributed reagents/materials/analysis tools: IMM ID MK. Wrote the paper: IMM ID MK.
References
- 1. Modlin IM, Oberg K, Chung DC, Jensen RT, de Herder WW, et al. (2008) Gastroenteropancreatic neuroendocrine tumours. Lancet Oncol 9: 61–72.
- 2. Yao JC, Hassan M, Phan A, Dagohoy C, Leary C, et al. (2008) One hundred years after “carcinoid”: epidemiology of and prognostic factors for neuroendocrine tumors in 35,825 cases in the United States. J Clin Oncol 26: 3063–3072.
- 3. Kulke MH, Siu LL, Tepper JE, Fisher G, Jaffe D, et al. (2011) Future directions in the treatment of neuroendocrine tumors: consensus report of the National Cancer Institute Neuroendocrine Tumor clinical trials planning meeting. J Clin Oncol 29: 934–943.
- 4.
Yao JC, Pavel M, Phan AT, Kulke MH, Hoosen S, et al. (2011) Chromogranin A and neuron-specific enolase as prognostic markers in patients with advanced pNET treated with everolimus. J Clin Endocrinol Metab 96: 3741–3749. Epub 2011 Oct 3712.
- 5.
AACC (2010) Chromogranin A. Lab Tests Online: American Association for Clinical Chemistry. pp. Details regarding Chromogranin A as a test.
- 6. van ’t Veer LJ, Dai H, van de Vijver MJ, He YD, Hart AA, et al. (2002) Gene expression profiling predicts clinical outcome of breast cancer. Nature 415: 530–536.
- 7.
Hess KR, Anderson K, Symmans WF, Valero V, Ibrahim N, et al. (2006) Pharmacogenomic predictor of sensitivity to preoperative chemotherapy with paclitaxel and fluorouracil, doxorubicin, and cyclophosphamide in breast cancer. J Clin Oncol 24: 4236–4244. Epub 2006 Aug 4238.
- 8. Frederiksen CM, Knudsen S, Laurberg S, Orntoft TF (2003) Classification of Dukes’ B and C colorectal cancers using expression arrays. J Cancer Res Clin Oncol 129: 263–271.
- 9. Kidd M, Modlin IM, Mane SM, Camp RL, Eick GN, et al. (2006) Utility of molecular genetic signatures in the delineation of gastric neoplasia. Cancer 106: 1480–1488.
- 10. Modlin IM, Kidd M, Latich I, Zikusoka MN, Eick GN, et al. (2006) Genetic differentiation of appendiceal tumor malignancy: a guide for the perplexed. Ann Surg 244: 52–60.
- 11.
Kidd M, Modlin IM, Mane SM, Camp RL, Eick G, et al. (2006) The role of genetic markers–NAP1L1, MAGE-D2, and MTA1–in defining small-intestinal carcinoid neoplasia. Ann Surg Oncol 13: 253–262. Epub 2006 Jan 2020.
- 12. Drozdov I, Kidd M, Nadler B, Camp RL, Mane SM, et al. (2009) Predicting neuroendocrine tumor (carcinoid) neoplasia using gene expression profiling and supervised machine learning. Cancer 115: 1638–1650.
- 13. McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, et al. (2005) Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst 97: 1180–1184.
- 14. Parkinson H, Kapushesky M, Kolesnikov N, Rustici G, Shojatalab M, et al. (2009) ArrayExpress update–from an archive of functional genomics experiments to the atlas of gene expression. Nucleic Acids Res 37: D868–872.
- 15. Kidd M, Nadler B, Mane S, Eick G, Malfertheiner M, et al. (2007) GeneChip, geNorm, and gastrointestinal tumors: novel reference genes for real-time PCR. Physiological genomics 30: 363–370.
- 16. Leja J, Essaghir A, Essand M, Wester K, Oberg K, et al. (2009) Novel markers for enterochromaffin cells and gastrointestinal neuroendocrine carcinomas. Mod Pathol 22: 261–272.
- 17. Shou J, Dotson C, Qian HR, Tao W, Lin C, et al. (2005) Optimized blood cell profiling method for genomic biomarker discovery using high-density microarray. Biomarkers 10: 310–320.
- 18. Irizarry RA, Hobbs B, Collin F, Beazer-Barclay YD, Antonellis KJ, et al. (2003) Exploration, normalization, and summaries of high density oligonucleotide array probe level data. Biostatistics 4: 249–264.
- 19. Drozdov I, Ouzounis CA, Shah AM, Tsoka S (2011) Functional Genomics Assistant (FUGA): a toolbox for the analysis of complex biological networks. BMC research notes 4: 462.
- 20. Modlin IM, Gustafsson BI, Drozdov I, Nadler B, Pfragner R, et al. (2009) Principal component analysis, hierarchical clustering, and decision tree assessment of plasma mRNA and hormone levels as an early detection strategy for small intestinal neuroendocrine (carcinoid) tumors. Ann Surg Oncol 16: 487–498.
- 21. Sarac OS, Atalay V, Cetin-Atalay R (2010) GOPred: GO molecular function prediction by combined classifiers. PloS one 5: e12382.
- 22. Horvath S, Dong J (2008) Geometric interpretation of gene coexpression network analysis. PLoS computational biology 4: e1000117.
- 23. Stuart JM, Segal E, Koller D, Kim SK (2003) A gene-coexpression network for global discovery of conserved genetic modules. Science 302: 249–255.
- 24.
Blondel VD, Guillaume J-L, Lambiotte R, Lefebvre E (2008) Fast unfolding of communities in large network. J Stat Mech P10008.
- 25. Ippolito JE, Xu J, Jain S, Moulder K, Mennerick S, et al. (2005) An integrated functional genomics and metabolomics approach for defining poor prognosis in human neuroendocrine cancers. Proceedings of the National Academy of Sciences of the United States of America 102: 9901–9906.
- 26. Li L, Zhang K, Lee J, Cordes S, Davis DP, et al. (2009) Discovering cancer genes by integrating network and functional properties. BMC Med Genomics 2: 61.
- 27. Muscarella LA, D’Alessandro V, la Torre A, Copetti M, De Cata A, et al. (2011) Gene expression of somatostatin receptor subtypes SSTR2a, SSTR3 and SSTR5 in peripheral blood of neuroendocrine lung cancer affected patients. Cell Oncol 19: 19.
- 28. Forbes SA, Bindal N, Bamford S, Cole C, Kok CY, et al. (2011) COSMIC: mining complete cancer genomes in the Catalogue of Somatic Mutations in Cancer. Nucleic acids research 39: D945–950.
- 29. Zatelli MC, Torta M, Leon A, Ambrosio MR, Gion M, et al. (2007) Chromogranin A as a marker of neuroendocrine neoplasia: an Italian Multicenter Study. Endocr Relat Cancer 14: 473–482.
- 30. Ramachandran R, Bech P, Murphy KG, Dhillo WS, Meeran KM, et al. (2012) Improved diagnostic accuracy for neuroendocrine neoplasms using two chromogranin A assays. Clin Endocrinol (Oxf) 76: 831–836
- 31. Palmieri G, Pirastu M, Strazzullo M, Ascierto PA, Satriano SM, et al. (2001) Clinical significance of PCR-positive mRNA markers in peripheral blood and regional nodes of malignant melanoma patients. Melanoma Cooperative Group. Recent Results Cancer Res 158: 200–203.
- 32.
Van der Auwera I, Peeters D, Benoy IH, Elst HJ, Van Laere SJ, et al. (2010) Circulating tumour cell detection: a direct comparison between the CellSearch System, the AdnaTest and CK-19/mammaglobin RT-PCR in patients with metastatic breast cancer. Br J Cancer 102: 276–284. Epub 2009 Dec 2001.
- 33. Stridsberg M, Eriksson B, Oberg K, Janson ET (2003) A comparison between three commercial kits for chromogranin A measurements. J Endocrinol 177: 337–341.
- 34. Aggarwal A, Guo DL, Hoshida Y, Yuen ST, Chu KM, et al. (2006) Topological and functional discovery in a gene coexpression meta-network of gastric cancer. Cancer Res 66: 232–241.
- 35. Vischioni B, Oudejans JJ, Vos W, Rodriguez JA, Giaccone G (2006) Frequent overexpression of aurora B kinase, a novel drug target, in non-small cell lung carcinoma patients. Molecular cancer therapeutics 5: 2905–2913.
- 36. Glotsos D, Tohka J, Ravazoula P, Cavouras D, Nikiforidis G (2005) Automated diagnosis of brain tumours astrocytomas using probabilistic neural network clustering and support vector machines. Int J Neural Syst 15: 1–11.
- 37. Mattfeldt T, Gottfried HW, Wolter H, Schmidt V, Kestler HA, et al. (2003) Classification of prostatic carcinoma with artificial neural networks using comparative genomic hybridization and quantitative stereological data. Pathol Res Pract 199: 773–784.
- 38. Zander T, Hofmann A, Staratschek-Jox A, Classen S, Debey-Pascher S, et al. (2011) Blood-based gene expression signatures in non-small cell lung cancer. Clinical cancer research : an official journal of the American Association for Cancer Research 17: 3360–3367.
- 39. Parry RM, Jones W, Stokes TH, Phan JH, Moffitt RA, et al. (2010) k-Nearest neighbor models for microarray gene expression analysis and clinical outcome prediction. The pharmacogenomics journal 10: 292–309.
- 40. Demsar J, Zupan B, Kattan MW, Beck JR, Bratko I (1999) Naive Bayesian-based nomogram for prediction of prostate cancer recurrence. Studies in health technology and informatics 68: 436–441.
- 41.
Nebozhyn M, Loboda A, Kari L, Rook AH, Vonderheid EC, et al. (2006) Quantitative PCR on 5 genes reliably identifies CTCL patients with 5% to 99% circulating tumor cells with 90% accuracy. Blood 107: 3189–3196. Epub 2006 Jan 3110.
- 42. Stridsberg M, Oberg K, Li Q, Stridsberg M, Eriksson B, et al. (1995) Measurement of chromogranin A, chromogranin B (secretogranin I), chromogranin C (secretogranin II) and pancreastatin in plasma and urine from patients with carcinoid tumours and endocrine pancreatic tumours. Journal of Endocrinology 144: 49–59.
- 43. Modlin IM, Gustafsson BI, Moss SF, Pavel M, Tsolakis AV, et al. (2010) Chromogranin A–biological function and clinical utility in neuro endocrine tumor disease. Ann Surg Oncol 17: 2427–2443.
- 44.
Jianu CS, Fossmark R, Syversen U, Hauso O, Waldum HL (2010) A meal test improves the specificity of chromogranin A as a marker of neuroendocrine neoplasia. Tumour Biol 31: 373–380. Epub 2010 May 2018.
- 45.
Namwongprom S, Wong FC, Tateishi U, Kim EE, Boonyaprapa S (2008) Correlation of chromogranin A levels and somatostatin receptor scintigraphy findings in the evaluation of metastases in carcinoid tumors. Ann Nucl Med 22: 237–243. Epub 2008 Jun 2006.
- 46.
Arnold R, Wilke A, Rinke A, Mayer C, Kann PH, et al. (2008) Plasma chromogranin A as marker for survival in patients with metastatic endocrine gastroenteropancreatic tumors. Clin Gastroenterol Hepatol 6: 820–827. Epub 2008 Jun 2010.
- 47. Sciarra A, Monti S, Gentile V, Salciccia S, Gomez AM, et al. (2005) Chromogranin A expression in familial versus sporadic prostate cancer. Urology 66: 1010–1014.
- 48. Giusti M, Sidoti M, Augeri C, Rabitti C, Minuto F (2004) Effect of short-term treatment with low dosages of the proton-pump inhibitor omeprazole on serum chromogranin A levels in man. Eur J Endocrinol 150: 299–303.
- 49. Hsiao RJ, Mezger MS, O’Connor DT (1990) Chromogranin A in uremia: progressive retention of immunoreactive fragments. Kidney Int 37: 955–964.
- 50. Campana D, Nori F, Piscitelli L, Morselli-Labate AM, Pezzilli R, et al. (2007) Chromogranin A: is it a useful marker of neuroendocrine tumors? J Clin Oncol 25: 1967–1973.
- 51. Chou WC, Hung YS, Hsu JT, Chen JS, Lu CH, et al. (2012) Chromogranin A is a reliable biomarker for gastroenteropancreatic neuroendocrine tumors in an Asian population of patients. Neuroendocrinology 95: 344–350
- 52. Yates DR, Roupret M, Drouin SJ, Comperat E, Ricci S, et al. (2012) Quantitative RT-PCR analysis of PSA and prostate-specific membrane antigen mRNA to detect circulating tumor cells improves recurrence-free survival nomogram prediction after radical prostatectomy. Prostate 6: 22488.
- 53. Mead R, Duku M, Bhandari P, Cree IA (2011) Circulating tumour markers can define patients with normal colons, benign polyps, and cancers. Br J Cancer 105: 239–245
- 54.
Bai VU, Hwang O, Divine GW, Barrack ER, Menon M, et al. (2012) Averaged differential expression for the discovery of biomarkers in the blood of patients with prostate cancer. PLoS One 7: e34875. Epub 32012 Apr 34876.
- 55. Alymani NA, Smith MD, Williams DJ, Petty RD (2010) Predictive biomarkers for personalised anti-cancer drug use: discovery to clinical implementation. Eur J Cancer 46: 869–879.
- 56. Schuster R, Max N, Mann B, Heufelder K, Thilo F, et al. (2004) Quantitative real-time RT-PCR for detection of disseminated tumor cells in peripheral blood of patients with colorectal cancer using different mRNA markers. Int J Cancer 108: 219–227.
- 57. Wu DY, Ugozzoli L, Pal BK, Qian J, Wallace RB (1991) The effect of temperature and oligonucleotide primer length on the specificity and efficiency of amplification by the polymerase chain reaction. DNA Cell Biol 10: 233–238.
- 58. Dingemans AM, Brakenhoff RH, Postmus PE, Giaccone G (1997) Detection of cytokeratin-19 transcripts by reverse transcriptase-polymerase chain reaction in lung cancer cell lines and blood of lung cancer patients. Lab Invest 77: 213–220.
- 59. Zippelius A, Kufer P, Honold G, Kollermann MW, Oberneder R, et al. (1997) Limitations of reverse-transcriptase polymerase chain reaction analyses for detection of micrometastatic epithelial cancer cells in bone marrow. J Clin Oncol 15: 2701–2708.
- 60. Henke W, Loening SA (1998) Detection of illegitimate transcripts of prostate-specific antigen mRNA in blood by reverse transcription-polymerase chain reaction. Int J Cancer 77: 164–165.
- 61. Lambrechts AC, van ’t Veer LJ, Rodenhuis S (1998) The detection of minimal numbers of contaminating epithelial tumor cells in blood or bone marrow: use, limitations and future of RNA-based methods. Ann Oncol 9: 1269–1276.
- 62. Bustin SA (2002) Quantification of mRNA using real-time reverse transcription PCR (RT-PCR): trends and problems. J Mol Endocrinol 29: 23–39.
- 63. Bustin SA (2000) Absolute quantification of mRNA using real-time reverse transcription polymerase chain reaction assays. J Mol Endocrinol 25: 169–193.
- 64. Schmittgen TD, Zakrajsek BA, Mills AG, Gorn V, Singer MJ, et al. (2000) Quantitative reverse transcription-polymerase chain reaction to study mRNA decay: comparison of endpoint and real-time methods. Anal Biochem 285: 194–204.
- 65.
Ding C, Cantor CR (2003) A high-throughput gene expression analysis technique using competitive PCR and matrix-assisted laser desorption ionization time-of-flight MS. Proc Natl Acad Sci U S A 100: 3059–3064. Epub 2003 Mar 3056.
- 66. Liu W, Saint DA (2002) Validation of a quantitative method for real time PCR kinetics. Biochem Biophys Res Commun 294: 347–353.
- 67. Lekanne Deprez RH, Fijnvandraat AC, Ruijter JM, Moorman AF (2002) Sensitivity and accuracy of quantitative real-time polymerase chain reaction using SYBR green I depends on cDNA synthesis conditions. Anal Biochem 307: 63–69.
- 68. Keilholz U, Willhauck M, Rimoldi D, Brasseur F, Dummer W, et al. (1998) Reliability of reverse transcription-polymerase chain reaction (RT-PCR)-based assays for the detection of circulating tumour cells: a quality-assurance initiative of the EORTC Melanoma Cooperative Group. Eur J Cancer 34: 750–753.
- 69. Jung R, Ahmad-Nejad P, Wimmer M, Gerhard M, Wagener C, et al. (1997) Quality management and influential factors for the detection of single metastatic cancer cells by reverse transcriptase polymerase chain reaction. Eur J Clin Chem Clin Biochem 35: 3–10.
- 70. Schittek B, Blaheta HJ, Florchinger G, Sauer B, Garbe C (1999) Increased sensitivity for the detection of malignant melanoma cells in peripheral blood using an improved protocol for reverse transcription-polymerase chain reaction. Br J Dermatol 141: 37–43.
- 71. Saveanu A, Muresan M, De Micco C, Taieb D, Germanetti AL, et al. (2011) Expression of somatostatin receptors, dopamine D(2) receptors, noradrenaline transporters, and vesicular monoamine transporters in 52 pheochromocytomas and paragangliomas. Endocr Relat Cancer 18: 287–300
- 72.
Reubi JC, Waser B (2003) Concomitant expression of several peptide receptors in neuroendocrine tumours: molecular basis for in vivo multireceptor tumour targeting. Eur J Nucl Med Mol Imaging 30: 781–793. Epub 2003 Apr 2018.
- 73. Dilley WG, Kalyanaraman S, Verma S, Cobb JP, Laramie JM, et al. (2005) Global gene expression in neuroendocrine tumors from patients with the MEN1 syndrome. Mol Cancer 4: 9.
- 74. Lapuk AV, Wu C, Wyatt AW, McPherson A, McConeghy BJ, et al. (2012) From sequence to molecular pathology, and a mechanism driving the neuroendocrine phenotype in prostate cancer. J Pathol 227: 286–297 .
- 75. Sun MH (2004) Neuroendocrine differentiation in sporadic CRC and hereditary nonpolyosis colorectal cancer. Dis Markers 20: 283–288.
- 76. Indinnimeo M, Cicchini C, Memeo L, Stazi A, Provenza C, et al. (2002) Correlation between chromogranin-A expression and pathological variables in human colon carcinoma. Anticancer Res 22: 395–398.
- 77. Langfelder P, Zhang B, Horvath S (2008) Defining clusters from a hierarchical cluster tree: the Dynamic Tree Cut package for R. Bioinformatics. 24: 719–720.