Workflow of evaluation protocols.
The method under evaluation is run on the diverse set of representative SCOP domains,
using each representative as a query for the comparison with the rest domains. The
search results are then submitted to the evaluation of the quality of similarity
detection ("Detection quality"). This evaluation, based on ROC-like sensitivity curves,
is done in both reference-dependent and reference-independent fashion. Reference-dependent
evaluation is based on a pre-computed gold standard of similarity relationships between
the domains, and on reference (structure-based) alignments. This type of evaluation is
performed in two modes that differ in the definition of a true positive hit. The first,
more traditional mode ("No alignment quality required"), considers the prediction of
similarity regardless of the quality of produced alignment, i.e. a wrong alignment of
two similar domains has the same merit as the correct one. The second mode ("Alignment
quality required") demands a certain level of alignment accuracy for the hit to be
considered a true positive. Reference-independent evaluation does not use any gold
standards but is based on the quality of structural superposition suggested by the hit.
The quality measures are based on the GDT_TS score and are applied in two modes: global
(assessing the superposition over the whole query length) and local (assessing the
superposition over the alignment length, however short it is). Both reference-dependent
and -independent evaluations are performed on the whole pool of search results
("All-to-all domains"), as well as on the separated results for individual queries
("By individual queries"). The evaluation of alignment quality regardless of assigned
statistical significance is performed on the set of structurally similar domains, in
both reference-dependent fashion (based on gold-standard structure-based alignments)
and reference-independent fashion (based on GDT_TS-like measures for the structure
superposition suggested by the evaluated alignment).