Home | Targets | Evaluation | Contact

CASP9

© 2008 Grishin lab/HHMI/UTSW

updated on 2008-11-28

Targets
515	516	517	518
519	520	521	522
523	524	525	526
527	528	529	530
531	532	533	534
535	536	537	538
539	540	541	542
543	544	545	546
547	548	549	550
551	552	553	554
555	556	557	558
559	560	561	562
563	564	565	566
567	568	569	570
571	572	573	574
575	576	577	578
579	580	581	582
583	584	585	586
587	588	589	590
591	592	593	594
595	596	597	598
599	600	601	602
603	604	605	606
607	608	609	610
611	612	613	614
615	616	617	618
619	620	621	622
623	624	625	626
627	628	629	630
631	632	633	634
635	636	637	638
639	640	641	642
643

Whole chain vs. Domain in CASP

Traditionally, CASP targets are evaluated as domains, i.e. each target structure is parsed into domains, and model quality is computed for each domain separately. This strategy makes sense for two reasons:

• Domains can be mobile and their relative packing can be influenced by ligand presence, crystal packing for X-ray structures, or be semi-random in NMR structures. Thus even a perfect prediction algorithm will not be able to cope with this adequately, for instance in the absence of knowledge about the ligand presence or crystal symmetry.
• Predictions may be better or worse for individual domains than for their assembly. This happens when domains are of a different predictability, e.g. one has a close template, but the other one does not. Even if domains of a target are of equal prediction difficulty, it is possible that the mutual domain arrangement in the target structure, while predictable in principle, differs from the template, and thus is modeled incorrectly by predictors.

Comparison of the whole-chain evaluation with the domain-based evaluation dissects the problem of 'individual domain' vs. 'domain assembly' modeling and should help in development of prediction methods.

While it is clear that a detailed look at the predictions for each domain is beneficial, it is desirable to combine predictions over targets in a meaningful way, and to rank servers by the averaged ability to predict protein structures. Combination over whole chains does not address problems with domain predictions, but combination over domains may be dominated by multi-domain easy targets with domain arrangement matching the template and thus predicted well. Therefore it makes sense to combine whole chain and domain evaluations. In this combination, some targets, in particular those without problems of domain assembly, could be evaluated as 'whole chain'; while other targets, notably those with different domain predictability or difficulties with domain assembly, could be evaluated as 'domains'. Here, we attempted to determine a natural cutoff for whether the target should be evaluated as a 'domains' or as a 'whole chain'.

For each target composed of more than 1 domain (see our domain parse), we obtained GDT-TS scores on the whole chain and individual domains for all server models. Then weighted sum of GDT-TS scores for domain-based evaluation was computed, i.e. GDT-TS for each domain was multiplied by the domain length, summed, and the sum was divided by the sum of domain lengths. Typical correlation plot between the two GDT-TSs (whole chain and weighted by the number of residues sum of GDT-TS scores for domain-based evaluation) looked like this:

Each point represents first server model. Green, gray and black points are top 10, bottom 25% and the rest of prediction models. Blue line is the best-fit slope line (intersection 0) to the top 10 server models. Red line is the diagonal. Slope and root mean square y-x distance for the top 10 models (average difference between the weighted sum of domain GDT-TS scores and the whole chain GDT-TS score) are shown above the plot.

The points lie above the diagonal. Apparently, weighted sum of domain predictions is higher than the whole chain GDT-TS. This is because domain arrangement is a bit different between target and template, thus, while individual domains are modeled well, their assembly is predicted worse. We measure the difference between the weighted sum and the whole chain GDT-TS by two parameters. The root mean square (RMS) difference between the weighted sum of GDT_TS on domains and GDT_TS on the whole chain (RMS of y−x) measures absolute GDT-TS difference. A slope of best-fit line with intercept set to 0 (slope) measures relative GDT-TS difference. These parameters are computed on top 10 (according to the weighted sum) predictions.

The target on the above plot (T0528) is a three-domain protein. The slope and the RMS of y−x are 1.22 and 13.52, respectively. Do these parameters justify splitting the target into 2 domains and using all the domain individually in the combined evaluation of predictions? To answer this question, we examined correlation plots for all targets.

Here, we illustrate two examples. First, for T0521, which is a two domain protein, the plot revealed that while individual domains are predicted reasonably well: GDT-TS above 80 for some servers, their packing was not: GDT-TS about 40, which is 2 times less than the weighted sum over 2 domains, thus indicating that domain arrangement was modeled randomly by the servers and did not match closely the target domain arrangement. Obviously, domain evaluation is beneficial for this target.

Second, for T0515, which is also a 2-domain target, the plot revealed that weighted sum and whole chain GDT-TS are about the same for all servers. Clearly, domain-based evaluation is not different from the whole chain evaluation and does not reveal any interesting features of predictions.

Before we examine all targets to find a data-dictated cutoff for domain-based evaluation, additional issue needs to be considered. Some proteins, while being evolutionarily single-domain proteins, experience domain swaps. Domain swap is defined as a structural "exchange" of protein regions between monomers in an oligomer. For instance, T0459 is a PYP-like sensor domain with 'domain swap'. See diagram below, the left panel shows the whole chain, and the right illustrats the swap:

Here is the correlation between domain-based evaluation and whole-chain based evaluation for T0600. And clearly, domain-based evaluation (evaluation on domain swap) is needed for T0600.

To find a cutoff for using 'domain-based' vs. 'whole chain' evaluation, we analyzed the correlation of RMS of the difference between GDT_TS on domains and GDT_TS on the whole chain (RMS of y−x) and the slope of the 0 intercept best-fit line (slope) for all targets:

Summary: Comparison of domain-based predictions with whole chain predictions revealed a natural, data-dictated cutoff (slope of the zero intercept best-fit line is above 1.3) to select targets that require domain-based evaluation. These targets are: T0521, T0528, T0529, T0533, T0534, T0542, T0543, T0544, T0547, T0548, T0550, T0553, T0555, T0571, T0575, T0579, T0582, T0586, T0589, T0596, T0600, T0604, T0608, T0611, T0628, T0629, T0632. Predictions for other targets follow the general trend, are of a more similar quality for 'domain' and 'whole chain' and thus domain-based evaluation may not be necessary for them. It is important to note that this cutoff was found using CASP9 targets and predictions. It is possible, even likely, that for other target/prediction sets, data may dictate a different cutoff. Therefore similar analysis should be performed on other target sets, rather that this 1.3 slope cutoff being applied to them.

We used only the "red" targets in domain evaluation to combine scores between targets, thus only these targets are split into domains in combined evaluation tables. All other targets were evaluated as whole chain in domain-based evaluation: they are considered to be single-domain targets for the purpose of CASP9 evaluation. However, all domain-based evaluation results for all targets are shown on individual target pages and are available for analysis and model visualization with PyMOL, e.g. see T0515. These "domains" include single domain proteins with certain structure regions removed, and swapped domains.