Criteria used to judge the performance of a screening tool
In this study, our main focus was on studies, and the tools researched in these studies, that are based on constructs closely related to diagnostic criteria and formal DSM-based diagnoses (criterion-related validity). The DSM-based model was selected because of its predominant and critical role in current clinical guidelines, treatment planning and ongoing research.
It was necessary to arrive at a common approach so as to have a basis for comparing tools. We opted to select and compare tools that were developed using DSM-based diagnoses in the validation process. This includes tools where the gold standard for comparison was based on the categorization of internalizing and externalizing disorders because of the proven value of this approach for children and adolescents.
Adopting another comparative approach would yield different results but we believe would be less useful in the context of the two- (or three-) staged model of screening and assessment that is the foundation of this project.
Validity and Reliability
Various types of validity are summarized by Grisso and colleagues (2005, p. 72).
- Content: How have the tool's intentions been translated into item content?
- Procedure: How is it administered?
- Norm sample: What types of young people have been involved in the tool's development?
- Internal integrity: Do the content and structure of the tool make sense?
- Dependability of measurement (usually called "reliability"): Does the tool measure consistently?
- Confidence and meaning (usually called "validity"): Does the tool measure what it claims to measure?
Reliability refers to how consistently the tool performs across different people who administer it (inter-rater reliability) and over different applications within the same time period (test-retest reliability). It also relates to how well the different items of a given tool work together (internal consistency). The reliability of a tool in a given situation is related to its validity or accuracy. It is difficult for something to be "on target" if the measurement approach itself is inconsistent.