Methodology for the inference of gene function from phenotype data

Abstract

Background

Biomedical ontologies are increasingly instrumental in the advancement of biological research primarily through their use to efficiently consolidate large amounts of data into structured, accessible sets. However, ontology development and usage can be hampered by the segregation of knowledge by domain that occurs due to independent development and use of the ontologies. The ability to infer data associated with one ontology to data associated with another ontology would prove useful in expanding information content and scope. We here focus on relating two ontologies: the Gene Ontology (GO), which encodes canonical gene function, and the Mammalian Phenotype Ontology (MP), which describes non-canonical phenotypes, using statistical methods to suggest GO functional annotations from existing MP phenotype annotations. This work is in contrast to previous studies that have focused on inferring gene function from phenotype primarily through lexical or semantic similarity measures.

Results

We have designed and tested a set of algorithms that represents a novel methodology to define rules for predicting gene function by examining the emergent structure and relationships between the gene functions and phenotypes rather than inspecting the terms semantically. The algorithms inspect relationships among multiple phenotype terms to deduce if there are cases where they all arise from a single gene function.

We apply this methodology to data about genes in the laboratory mouse that are formally represented in the Mouse Genome Informatics (MGI) resource. From the data, 7444 rule instances were generated from five generalized rules, resulting in 4818 unique GO functional predictions for 1796 genes.

Conclusions

We show that our method is capable of inferring high-quality functional annotations from curated phenotype data. As well as creating inferred annotations, our method has the potential to allow for the elucidation of unforeseen, biologically significant associations between gene function and phenotypes that would be overlooked by a semantics-based approach. Future work will include the implementation of the described algorithms for a variety of other model organism databases, taking full advantage of the abundance of available high quality curated data.

Background

A hallmark of modern biomedical research is the generation of increasingly large amounts of scientific data. Biomedical ontologies have the potential to greatly accelerate biomedical research by enhancing our ability to integrate and access these data. A biomedical ontology is a resource that represents a controlled set of terms for entities in a particular biomedical domain and how those terms are related to one another [1]. Biocurators are scientists who review experimental data, primarily as reported in the biomedical literature, to create empirical connections between different aspects of biological data, that is to say, annotations [2]; .e.g. biocurators annotate, or tag, biological entities (e.g. proteins, functional RNAs) with ontology terms (capturing all relevant metadata as well). One of the most widely used modern bio-ontologies is the Gene Ontology (GO), a resource that describes canonical gene functions in a computable species-independent manner so that they may be used for statistical analysis of gene sets or for comparative genomic analysis [3],[4]. Another bio-ontology is the Mammalian Phenotype Ontology (MP) that provides an independently curated set of terms and relationships describing non-canonical phenotypes, primarily in mouse models, and that is used to query the effects of genetic mutations [5].

Genes and alleles (genetic variants) of genes are annotated respectively for both function and phenotype within Mouse Genome Informatics (MGI) system, a comprehensive resource for genomic research of the laboratory mouse [6]. Within the MGI curation workflow, different subsets of biocurators separately process papers identified for function (GO) and phenotype (MP) curation. As a result, although papers may be selected for curation in regards to both GO and MP, they are not processed simultaneously, leading to short-term temporal discrepancies in overall curation coverage in MGI. Because scientific literature is published much faster than biocurators can read and curate the papers, the development of methods to computationally infer annotations from one source to another would greatly add and enhance curation efficiency [7].

Recent years have seen efforts to complement curated annotation data sets with text mined and association-rule mined predicted annotations. Broadly, text mining approaches use natural language processing methods to alleviate the backlog of papers awaiting curation, while association-rule mining uses curated annotation sets to predict new annotations and to assess the validity of automated annotation methods [8]-[17]. It is worth noting that several of these same approaches have been used to improve and expand the ontology structure and relationships as well [18]-[21]. These efforts have been used both to predict additional annotations from curated annotations in the same ontology and to predict across ontologies, as we do here.

The prediction efforts mentioned above may include lexical matching [8], semantic similarity measures [9]-[11], ontology matching [18],[19], and so-called ‘guilt-by-association’ methods [12]; several efforts use a combination of these approaches [13]. Some prediction methods are primarily ontology based and others are annotation, or instance, based. Our method uses an extension of ‘guilt-by-association’ and is annotation based.

Lexical matching methods, including text mining and text clustering, have been used to infer gene function from phenotype and vice versa. Semantic matching is facilitated by the use within the OBO community of equivalence axioms and logical definitions and by curated inter-ontology links. Semantic similarity measures based on ontology structure or information theory between phenotype and GO have been used to predict additional GO annotations. In a departure from semantic similarity approaches, various groups have performed network analyses to align GO terms with protein association networks to predict protein function [14]-[16].

Other efforts follow a more empirical approach such as instance based ontology matching and other so-called ‘guilt-by-association’ methods: annotation co-occurrence pairs, knowledge-based annotation inference based on, for example, protein-protein interactions or pathway term enrichment.

Our approach is strictly empirical and makes no assumptions about lexical matching, semantics, or ontology structure, except to infer annotations according to the true-path rule. The rationale behind our approach is to make a simplifying assumption that in some cases ‘interesting’ biology could be missed by limiting the analysis to include an alignment of ontology structure or by attempting to compare the ‘meaning’ of phenotype versus GO terms. Using this simplified approach there is no underlying assumption that ‘similar’ areas of the MPO and the GO should correlate. Instead, we examine the feasibility of constructing rules based only on conjunctions and disjunctions of high-quality phenotype annotations made by MGI curators to predict GO annotations. A crucial difference in our approach is that, where most empirical methods group annotations based on gene entities, our analysis is allele-specific, and therefore addresses the potential that a given set of varied phenotypes may be the result of a single underlying genetic perturbation. Additionally, mouse phenotypes can vary widely for different alleles of the same gene on different strain backgrounds. Indeed, this is the reason for the detailed study of spontaneous and targeted mutations in specific strains of the laboratory mouse. Consider, for example, the phenotypes of three alleles of the mouse Pax3 gene. One spontaneous allele, Pax3 ^Sp-d manifests phenotypes in many areas: embryogenesis, integument, limbs/digits/tail, mortality/aging, nervous system, pigmentation. This allele is present in the mouse model for the human disease Waardenburg Syndrome, Type 1; WS1 (OMIM:193500) [22],[23]. Another targeted allele, Pax3 ^tm1Mrc manifests a different set of phenotypes: craniofacial, growth/size, mortality/aging, muscle, nervous system, respiratory, skeleton, tumorigenesis, vision/eye. This allele is present in the mouse model for another human disease Rhabdomyosarcoma 2; RMS2 (OMIM:268220) [24]. Yet another targeted allele, Pax3 ^tm2.1Joe [25], is present in a mouse in which no abnormal phenotype is observed.

In this work, we describe an original method to predict novel GO annotations for genes associated with alleles that have existing MP annotations. We apply our derived set of rules to a set of papers that have been selected for curation for both MP and GO but that have, as yet, been annotated only for MP, but not for GO, within the MGI system. The algorithms draw inspiration from set and graph theory as they attempt to mathematically analyze relationships between GO and MP term(s). The approach used is as follows: First, gather relevant data and align MP and GO terms based on co-curated literature and shared alleles as our training set; Second, analyze the data such that rules can be made to predict a GO term from MP term(s); And finally, Third, apply the rules to a new set of papers/alleles that are currently annotated for MP and selected for but not yet annotated for GO. The goal of the work is to complement and facilitate the work of biocurators.

Methods

Consolidation of datasets

Both MP and GO are used in MGI, an open source resource that freely publishes its datasets. We collected all data used in this study from the MGI ftp site (ftp://ftp.informatics.jax.org/pub/reports/index.html, retrieved 10 June 2013). First, the set of literature that was annotated for both GO and MP in MGI was gathered and formatted; this set provided our training data set (3662 publications with both GO and MP annotations to the same allele). Then, the set of literature selected for both MP and GO, but annotated only for MP (63,028 publications) was collected from internal reports used by curators in their workflow; this set provided the test set for our derived inference rules.

Data alignment and processing

A base set of MP-GO pairs was generated by matching the set of all collected GO terms to the set of all collected MP terms used to annotate the same allele (we use the allele here as a proxy for full genotype) in the same study, knowledge of which is reflected by shared PubMedID (PMID) and alleleID. The GO terms were filtered to select only those terms from the biological process (BP) subontology, and only those GO annotations with evidence as “inferred from a mutant phenotype” (IMP). These selection criteria provide a defined set of papers restricted formally to those tagged for phenotype-based GO annotation for biological process, provided a first level of quality control on the dataset. There were found to be 81,245 MP-GO pairs sharing PMID/alleleID, and 67,424 unique MP-GO pairs, indicating that many of the MP-GO pairs occurred more than once. The MP, GO and the PMID/alleleID data were modeled as a network with PMID/alleleID nodes connecting MP and GO nodes derived from the same study.

All network visualizations were created using Cytoscape v2.8 [26] and all calculations were performed using Numerical Python (Python 2.7.3, NumPy 1.6.2, SciPy 0.10.1, Matplotlib 1.2) [27]. The network was modeled with an adjacency-like matrix. The term “adjacency matrix” is used loosely here, as the matrices presented in this work are quite different than those traditionally used—the matrices used here are rectangular matrices with the columns representing the annotated terms and rows representing PMID/alleleIDs. This approach is necessary since we wish to track individual studies for which both MP and GO annotations have been made. Two matrices were created, one with all unique PMID/alleleIDs (3662 rows) by all unique GO IDs (2472 columns); the other with all unique PMID/alleleIDs by all unique MP IDs (4978 columns). The network was found to be very sparse—that is, the number of all possible edges far exceeded the number of actual edges (graph density of approximately 6e-04).

Calculation and evaluation of statistical significance

After constructing our networks, we calculated the probability that a connection between an MP and GO node was statistically significant rather than due to chance alone. Some gene function-phenotype connections might be supported by many shared annotations, implying that there is some underlying connection between the gene function and the mutant phenotype, while others might be connected by only a few connections relative to the number other connections and yet also be informative.

From a set of unique integers of size N representing all the PMID/alleleID combinations, the probability that a selection of n at random will include j or more ‘successes’ – that is, a PMID/alleleID that is shared between a particular MP-GO node pair, can be defined as a modification of the cumulative binomial distribution (for values of N large compared to n):

$$\mathrm{p}\left({\mathrm{X}}_{\mathrm{n}}\ge \mathrm{j}\right)={\sum }_{\mathrm{i}=\mathrm{j}}^{\mathrm{n}}\left({}_{\mathrm{i}}^{\mathrm{n}}\right){\left(\frac{1}{\mathrm{N}}\right)}^{\mathrm{i}}{\left(1-\frac{1}{\mathrm{N}}\right)}^{\mathrm{n}-\mathrm{i}}\text{.}$$

(1)

In our case, N = 3662, and n, the maximum number of PMID/alleleID combinations between any MP-GO node pair, is 250, so the condition of equation 1 is satisfied. Applying this definition, the total probability, p_tot, that a particular MP_i-GO_j association is due to chance is given by:

$${\mathrm{p}}_{\mathrm{tot}}=\mathrm{p}\left[\mathrm{S}\subseteq \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_{\mathrm{i}}\right)\right]$$

(2)

where PM(MP_i) or PM(GO_j) returns the set of all PMID/alleleID nodes that are connected to the MP_i or GO_j nodes, respectively, and:

$$\mathrm{S}=\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_{\mathrm{i}}\right)\cap \mathrm{PM}\left(\mathrm{G}{\mathrm{O}}_{\mathrm{j}}\right)$$

(3)

is the set of data confirmed PMID/alleleID connections between the MP_i and the GO_j nodes. The p-value was calculated in this way for each MP-GO pair found in the data and sorted. The efficacy of the p-value was evaluated by using each MP-GO connection as a prediction, or ‘rule’: MP_i → GO_j —to wit: “if an annotation of a particular allele has been made to the MP term MP_i by curation of a particular study then an IMP annotation corresponding to that allele can be made to the GO term GO_j.” The resulting true and false positives (sensitivity and specificity) were plotted on the receiver operating characteristic (ROC) curve (Figure 1). As shown, the calculated p-value shows sufficient discriminatory power (AUC = 0.733) to discriminate MP-GO pairs that are more likely to correctly predict a GO node from a given MP term. An alternative scenario was also addressed, where possible MP terms were predicted from alleles with GO annotations (but no MP annotations); it was found that this prediction route does not carry as much predictive value (AUC = .686), further validating our approach to predict GO from MP annotations. We therefore used the calculated p-values as our measure of statistical significance for the GO function prediction from the simple rule.

Figure 1

ROC curve. Evaluation of the efficacy of a rule: if MP_i then GO_j, using the receiver operating characteristic.

Generalized patterns lead to extended rules

While the calculation of the p-value is useful in discovering which MP-GO pairs have stronger connections when run against the training sets, we found that even those with the lowest probabilities of being due to chance returned many false positives. Therefore, the need arose to either better identify the true positives of an MP-GO pair, or to better exclude the false positives: that is, to improve the discriminatory power of our rules.

Two types of generalized patterns arose from the network, which we chose to identify as the + (plus) rule and the - (minus) rule. The plus-rule can be qualitatively described as inspecting the connections of an MP-GO pair and examining if there is another MP node that is connected to the true positives of the pair, but excludes all of the false positives (PMID/alleleID nodes that are connected to MP, but not to GO) (Figure 2a). We can define the plus-rule: MP₁ AND MP₂ → GO₁ —to wit: “if an annotation of a particular allele has been made to the MP term MP₁ and to the MP term MP₂ by curation of a single particular study then an IMP annotation corresponding to that allele can be made to the GO term GO₁.” The conditional statement for finding an MP₁ and MP₂ that follow the pattern of the plus-rule is as follows:

$$\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\cap \mathrm{PM}\left(\mathrm{G}{\mathrm{O}}_1\right)==\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)$$

(4)

Figure 2

A graphical representation of the plus and minus rule. The highlighted nodes represent the ‘successes’ identified by the corresponding rule. a, The plus rule-a pattern whereby if MP₁ and MP₂, then GO₁. b, The minus rule-another pattern whereby if MP₁ and not MP₂, then GO₁.

The minus-rule takes the somewhat opposite approach from the plus-rule; that is, instead of discerning if the intersection of two MP nodes can define the successes, the minus-rule examines if the set difference of two MP nodes can define the successes (Figure 2b). Similarly, we can define the minus-rule: MP₁ AND NOT MP₂ → GO₁ —to wit: “if an annotation of a particular allele has been made to the MP term MP₁ but not to the MP term MP₂ by curation of a particular study then an IMP annotation corresponding to that allele can be made to the GO term GO₁.” The statement used to find an MP₁ and MP₂ that follow the pattern of the minus-rule is as follows:

$$\left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \mathrm{PM}\left(\mathrm{G}{\mathrm{O}}_1\right)\right)\backslash \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)==\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\backslash \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)$$

(5)

The network was then searched for collections of MP and GO nodes that followed the aforementioned patterns, with 3105 instances of the plus-rule pattern and 234 instances of the minus-rule pattern. Detailed examples of both rules are illustrated in Figure 3. However, the need to differentiate between those rules that were more likely to give results to be due to chance versus those that were less likely was still present. Utilizing the demonstrated efficacy of the cumulative binomial distribution as applied to the undirected graph (equation 1), the p-values for the plus and minus-rules were calculated respectively:

$${\mathrm{S}}_{+}=\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_{\mathrm{2}}\right)$$

(6)

$${\mathrm{p}}_{+}=\mathrm{p}\left[{\mathrm{S}}_{+}\subseteq \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\right]*\mathrm{p}\left[{\mathrm{S}}_{+}\subseteq \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\right]*\mathrm{p}\left[{\mathrm{S}}_{+}\subseteq \mathrm{PM}\left(\mathrm{G}{\mathrm{O}}_1\right)\right]$$

(7)

$${\mathrm{S}}_{‐}=\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\backslash \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)$$

(8)

$${\mathrm{p}}_{‐}=\mathrm{p}\left[{\mathrm{S}}_{‐}\subseteq \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\right]*\mathrm{p}\left[{\mathrm{S}}_{‐}\subseteq \mathrm{PM}\left(\mathrm{G}{\mathrm{O}}_1\right)\right]*\mathrm{p}\left[{{\mathrm{S}}_{‐}}^{\mathrm{c}}\subseteq \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\right]$$

(9)

where S in each case is the set of ‘successes’ and S^c is the set complement. As the network is sparse, the p-values are expected to be small. Arising from the basic plus and minus-rule patterns, three other rule patterns were defined, descriptively designated as plus-plus (++) (Figure 4a), minus-minus (--) (Figure 4b) and plus-minus (+-) (Figure 4c). The mathematical statements used to find rules from three additional rule patterns respectively are as follows:

$$\mathrm{PM}\left(\mathrm{GO}\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\cup \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_3\right)\right)==\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\cup \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_3\right)\right)$$

(10)

$$\mathrm{PM}\left(\mathrm{GO}\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\backslash \left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\cup \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_3\right)\right)==\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\backslash \left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\cup \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_3\right)\right)$$

(11)

$$\left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\cap \mathrm{PM}\left(\mathrm{GO}\right)\right)\backslash \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_3\right)==\left(\mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_1\right)\cap \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_2\right)\right)\backslash \mathrm{PM}\left(\mathrm{M}{\mathrm{P}}_3\right)$$

(12)

Figure 3

Example of rule instances. The highlighted nodes represent the ‘successes’ identified by the corresponding rule. a, Plus rule instance. b, Minus rule instance.

Figure 4

A graphical representation of the three rules emerging from the plus and minus rules. The highlighted nodes represent the ‘successes’ identified by the corresponding rule. a, The plus-plus rule-if MP₁ and MP₂ or MP₃, then GO₁. b, The minus-minus rule-if MP₁ and not either MP₂ or MP₃, then GO₁. c, The plus-minus rule-if MP₁ and MP₂ and not MP₃, then GO₁.

The statements used to calculate the p-values for these rule patterns are expounded in the appendix. The network was again searched for arrangements of nodes that followed the three other patterns, with 3143, 328, and 634 instances of the plus-plus-rule, minus-minus-rule and plus-minus-rule respectively. While the composite plus- and minus-rules can be combined iteratively in theory, there seems to be a point where the statements used to calculate the probabilities for large combinations of nodes become too large to be feasibly evaluated, and where they have the potential of straying further from biological reality. Therefore, we limited our analysis to these five rules as examples. All rule instances are compiled in Additional file 1.

Evaluation of extended rules

All rules were applied to the literature set of PMID/alleleIDs that are annotated to MP term(s) but not yet annotated to a GO term to predict which GO term would be annotated to the PMID/alleleID (set may be found in Additional file 2). Validation of predictions was performed by the selection of a set of 20 papers for each rule that represented a range of the p-values. The papers were read by a GO scientific curator and the curatorial predictions for functional GO annotation were examined in the context of our rule structures. Companion software for the prediction of GO annotations is provided as Additional file 3.

Results and discussion

Evaluation of predicted rules

Many of the rules and annotation predictions are immediately intuitive in nature. Table 1 shows a selection of rules of each type (Plus, Minus, …) along with p-values and a preliminary assessment of whether the rule is biologically ‘obvious’ (O) or ‘subtle/surprising’ (S). The inclusion of depth of GO terms as a proxy for specificity provided no additional information. The average depths for various rule sets are: ‘plus’ 7.25; ‘minus’ 7.28; ‘plus-plus’ 7.30; ‘plus-minus’ 7.15; ‘minus-minus’ 7.60. The reviewed annotations showed no trend according to GO depth.

Table 1 Selection of proposed rules

For example, the first Plus-rule #0: If a study shows that an allele has both phenotypes “impaired ovarian folliculogenesis” [MP:0001129] and “absent mature ovarian follicles” [MP:0001132], that allele is predicted be curated to function annotation “ovarian follicle development” [GO:0001541]. This rule has a very low p-value of 1.47E-36 and is assessed as ‘obvious.’ As another example, consider Plus-rule #21: If a PMID/alleleID has both the MP terms “abnormal direction of heart looping” [MP:0004252] and “situs inversus” [MP:0002766], then it should be annotated to the GO term “determination of left/right symmetry” [GO:0007368]. Both phenotypes are almost always associated with a defect in the biological process of “determination of left/right symmetry”, and it makes sense that the occurrence of both phenotypes simultaneously would more strongly indicate a defect in the aforementioned process.

Perhaps the more interesting results of our analysis are in the identification of plus-rule statements that are not necessarily obvious, but make sense biologically. For example, Plus-rule #5: If a PMID/alleleID has both the MP terms “absent Peyer’s patches” [MP:0002831] and “abnormal spleen morphology” [MP:0000689] that allele is predicted to have function “lymph node development” [GO:0048535] and is assessed as ‘subtle.’ Many of these rule statements predict relationships that would not result from a purely semantic approach. Plus-plus-rule #1188 predicts that if a PMID/alleleID has the MP terms “enlarged liver” [MP:0000599] and either “hepatic necrosis” [MP:0001654] or “increased glycogen levels” [MP:0005440], then the allele should be annotated to the GO term “glycogen metabolic process” [GO:0005977]. This is interesting because while an “enlarged liver” may not be intuitively or semantically linked to “glycogen metabolic processes”, the liver is instrumental in the storing and metabolism of glycogen, reflecting the overall physiology. This result illustrates another potential use of our correlative analysis in that a perturbation of glycogen metabolism might result in an enlarged liver phenotype or that in an animal with an enlarged liver, but no other phenotype reported, the underlying defect might be in glycogen metabolism. If our method took into consideration the structure of the ontology or a semantic link between terms, the liver correlation would likely have been missed because there is nothing in the ontology itself that states glycogen metabolism and the liver are linked.

Evaluation of predicted annotations

The next step in our prediction pipeline is the application of our rules to particular allele instances. As a result of this work, 4818 unique potential annotations associated with 1796 genes have been predicted.

Plus-rule #21, discussed above, is predicted to apply to mouse gene Zic3 [MGI:106676] based on the paper titled “Zic3 is required in the extracardiac perinodal region of the lateral plate mesoderm for left-right patterning and heart development” [PMID:23184148]. The study describes a variety of congenital defects primarily due to defects in the development of embryonic left-right patterning in Zic3 ^tm1Bca mice [28]. Our prediction is that since the paper has been used to make annotations to the MP terms “abnormal direction of heart looping” [MP:0004252] and “situs inversus” [MP:0002766], then it would be annotated, or should be annotated, to the GO term “determination of left/right symmetry” [GO:0007368]. In this case, curatorial review confirmed that the rule held true.

Tables 2, 3, 4, 5, and 6 show the curator’s evaluation of the annotations derived from the Plus, Minus, Plus-plus, Minus-minus, Plus-minus-rules respectively for the 20 papers that were reviewed for each rule type.

Table 2 Validation of annotations derived from plus-rules

Table 3 Validation of annotations derived from minus-rules

Table 4 Validation of annotations derived from plus-plus-rules

Table 5 Validation of annotations derived from plus-minus-rules

Table 6 Validation of annotations derived from minus-minus-rules

In 16 out of 20 cases, the Plus-rule generated an annotation that was considered correct by the curator. In all cases, the incorrect inferences were associated with poorer p-values (Table 2), validating its utility as a score to measure confidence level. In one negative case shown here, Plus-rule 3072: “decreased body size” plus “decreased length of long bones” phenotypes implies “collagen fibril organization,” the curator specifically noted that a curator could not make this annotation based on the paper since the authors did not look at the collagen fibrils directly although they did show that the cartilage in the developing bone is affected Another interesting negative case is the application of Plus-rule #3100: “deafness” plus “decreased body weight” phenotypes implies “neuromuscular process controlling balance” to two different genes. For Kcnq1, based on the paper [PMID:15498462], the annotation could be made; while for Slc12a7, based on the paper [PMID:11976689], it could not.

In the case of the Minus-rule, only 10 of the 20 predicted annotations were correct. In the case of the Minus-rule, better p-value scores gave more reliable predictions (Table 3). The Plus-plus (16 correct of 20) and Minus-minus (7 correct of 20) rules displayed similar behavior to the plus and minus-rules respectively (Tables 4 and 5), while the Plus-minus-rule (12 correct of 20) unsurprisingly seemed at almost act as an intermediate between the Plus and Minus-rules (Table 6).

Testing the validation of the various rules showed that, in general, the Plus-rule statements are more reliable that the minus-rule statements in generating valid annotations. This is not surprising for two reasons: (1) combined phenotypes often give ‘additive’ evidence to support the GO annotation and (2) the ‘closed world’ assumption limits the predictive power of the absence of annotation. The value of ‘additive’ evidence is mentioned above for Plus-rule #0, “absent mature ovarian follicles” [MP:0001132] plus “impaired ovarian folliculogenesis” [MP:0001129] additively support that a gene product would be directly involved in the GO process “ovarian follicle development” [GO:0001541]. The ‘closed world assumption’ is the assumption that what is not currently known to be true is assumed to be false. In our case, the Minus-rules rely on the lack of phenotype annotation as if it the phenotype was tested and found not to obtain.

We did a rather simple comparison of the predictive power of our rules based on the reviewed annotation predictions. We calculated the Positive Predictive Value from the true positives (TP) and false positives (FP) in the reviewed annotation sets for each of the composite rules: PPV = TP/(TP + FP). Figure 5 shows a comparison of the PPV for various p-value cutoffs of the composite rules for the reviewed annotation predictions. The PPV for the first three composite rules is markedly better for a given p-value than for the last two rules.

Figure 5

Comparison of the Positive Predictive Value for various p-value cutoffs of the composite rules for the reviewed annotation predictions. The PPV for the first three composite rules is markedly better for a given p-value than for the last two rules.

Our analysis shows that the Minus-rule statements are not reliable at predicting specific GO annotations. However, these results are very interesting because although the rules failed to accurately predict correct GO terms, the general area of biology of the predictions was often accurate. In several cases we found that the processes predicted might lead to downstream phenotypic similarities that would fit the phenotypes given in the rules. For example, the terms that predicted the process of ‘fertilization’ (Minus-rule 5) were not entirely correct, we noted, because the gene products identified did affect sperm motility and would create defective sperm, but we could not conclude from the papers [PMID:16015579; PMID:12167408] that the predicted gene products contributed to “fertilization” [GO:0009566] which in GO is specifically defined as the union of the two gametes. Likewise, the prediction of “regulation of angiogenesis” [GO:0045765] would have been correct if the prediction were “vasculogenesis” [GO:0001570], both of which can lead to abnormal branching of blood vessels (Minus-rule 227). These types of predictions may still prove useful for curators in that they point to relevant branches of biology for annotation consideration.

Our results show that the use of correlation modeling can be used to infer biological knowledge about the processes that underlie phenotypic expression. We show that the use of the Plus-rule statements has the potential to accurately predict GO annotations, and that while the Minus-rule generally predicts an incorrect specific term, it often predicts a correct area of biology. Our results show that curators can use our correlative rules to guide manual curation however, individual instances should not necessarily be annotated without the review of a biocurator. The rules can be used to help curators decide about a general subcategorization of GO terms from which to choose when curating GO data based on mutant phenotypes. The method could also be used when biocurators are trying to identify literature that covers an area of biology of interest. As the rules are further tested and more papers are cocurated for phenotype and GO, the rules will become further refined and accurate p-value cutoffs for reliability can be determined.

Future work will use these methods to create and test more complex rules based on this strategy. Our methodology could be used in future work to help predict the biological process that underlies a given disease by correlating the disease-phenotypes with GO biological processes. This methodological approach can be adapted to any species with rich phenotypic data. Lastly, we could similarly reverse the method to use multiple GO terms to predict likely phenotypic outcomes such as the disruption of specific pathways and then test those predictions as potential new mouse models of human disease.

Conclusions

Our correlative analysis for predicting GO annotations can be used to assist biocurators in the curation of papers with no previous GO annotations while also giving potential insight into complex phenotype-gene function relationships. We believe this is the first attempt to predict a GO term from its composite relationship with MP terms independently of semantic analysis but rather through a shared allele, and to apply that method to predict GO annotations to a set of papers that have not yet been annotated for GO. Our method has the advantage in that since it does not take semantic similarity into account, it can potentially find correlations between phenotypes and biological processes that are not intuitively obvious or that are not explicitly states in either ontology. Of course we are in a good position to perform this type of analysis since there are large independent efforts in the MGI resource to annotate both phenotypes and GO. The independent annotation serves as an internal control in that the annotations that are in the current corpus are essentially ‘blind’ with respect to one another. Co-curation of phenotype and underlying biological processes could potentially change our results dramatically since a single curator could be swayed to search for a semantic similarity in terms chosen from each ontology. One interesting experiment would be to perform this same analysis on a data set from another group in which individual curators annotate to both ontologies. Since our methodology is purely correlative and does not rely on any other metric, it could potentially be used with other data sets such a GO terms and disease ontologies, or even in combination with several ontologies such as GO, phenotype and expression. The five rules analyzed in this work can be combined in various manners to create many more possible derived rule structures—our results here serve as a ‘proof of principal’ for this type of analysis and pave the way for future iterations.

In practice, we have developed a script and methodology that, given a gene/PubMed ID, will suggest a GO term(s) to a curator if it meets the requirements of one of our rule statements. We intend to integrate this immediate approach into the workflow of MGI curators and hope that others will use our methodology to explore correlative algorithms in the context of diverse data sets.