Home | Resources | Frequently Asked Questions on Research

Frequently Asked Questions on Research

If you have additional questions about research, go to Institute of Education Sciences’ What Works Clearinghouse site at  http://ies/default.a.ed.gov/ncee/wwcspx

What is an analytic sample?

The sample on which the analysis was based.

What is a comparison group design?

A study design in which outcomes for a group receiving an intervention are compared to those for a group not receiving the intervention. Comparison group designs eligible for WWC review are randomized controlled trials, quasi-experimental designs, and regression discontinuity designs.

What is a confounding factor

A component of a study that is completely aligned with one of the study conditions. For example, a study may have one intervention school and a different comparison school. In this case, it is impossible to separate how much of the observed effect was due to the intervention and how much was due to the particular school in which the intervention was used. Therefore, a study with a confounding factor cannot meet standards, as the impact cannot be attributed solely to the intervention.

What is a counterfactual?

The comparison condition. The counterfactual may be receiving a different intervention or not receiving any services or intervention. For group design studies, groups receiving different dosage levels or different versions of a single intervention are not acceptable counterfactuals.

What is a developer/distributor?

The creator of the intervention or the person or company from whom the intervention can be purchased. The WWC solicits feedback from the developer/distributor of a product or practice when beginning a review of an intervention. The WWC notifies the developer/distributor of the WWC review and provides a list of all WWC-identified citations related to the intervention, inquires if the list is complete, invites comment on the intervention description slated for use in the report, and requests that the developer/distributor sign an agreement not to disclose any information about the review prior to its release. If the initial contact is the distributor of the intervention, but not the original developer, the distributor is asked whether they wish to name an individual to also receive the materials to review and comment.

What is an effect size?

A standardized measure of the magnitude of an effect. The effect size represents the change (measured in standard deviations) in an average student’s outcome that can be expected if that student is given the intervention. Because effect sizes are standardized, they can be compared across outcomes and studies.

What is evidence standards?

The set of standards used by the WWC to review studies. These standards are used to evaluate the strength of the evidence of the effectiveness of educational interventions. Studies are given a rating of Meets WWC Evidence Standards without ReservationsMeets WWC Evidence Standards with Reservations, or Does Not Meet WWC Evidence Standards.

What is an indeterminate effect?

An effect that is neither statistically significant nor substantively important.

What is intra-class correlation?

A measure of the relatedness of outcome measures in a cluster. A mismatch problem occurs when random assignment is carried out at the cluster level and the analysis is conducted at the student level, but the correlation among students within the same clusters is ignored in the computing the standard errors of the impact estimates. Although the point estimates of the intervention’s effects based on such mismatched analyses are not affect as a result of ignoring this feature of the study sample, the standard errors of the impact estimates generally will be underestimated, thereby leading to overestimates of statistical significance. To fairly assess an intervention’s effects, in cases where study authors have not corrected for the clustering, the WWC computes clustering-corrected statistical significance estimates. Based on empirical literature in the field of education, the WWC has adopted a default value for the intra-class correlation of 0.20 for academic outcomes and 0.10 for behavioral outcomes.

What is multiple comparison adjustment?

An adjustment to the statistical significance of results to account for multiple comparisons. The WWC uses the Benjamini-Hochberg (BH) correction to adjust the statistical significance of results within a domain when study authors perform multiple hypothesis tests simultaneously without adjusting the p-value. The BH correction is used in three types of situations: studies that tested multiple outcome measures in the same outcome domain with a single comparison group, studies that tested a given outcome measure with multiple comparison groups, and studies that tested multiple outcome measures in the same outcome domain with multiple comparison groups. Because repeated tests of highly correlated constructs will lead to a greater likelihood of mistakenly concluding that the impact was different from zero, in all three situations, the WWC uses the BH correction to reduce the possibility of making this error.

What are negative effects?

Negative:  strong evidence that intervention had a negative effect on outcomes. This occurs when two or more studies show statistically significant negative effects, at least one of which met WWC standards for a strong design, and no studies show statistically significant or substantively important positive effects.

What is an outcome?

Knowledge, skills, attitudes, and other desired benefits that are attained as a result of an activity. To examine the effectiveness of an intervention for the WWC, eligible research must compare the outcome for a group receiving the intervention to the outcome for a group not receiving the intervention. An outcome measure is an instrument, device, or method that provides data on the outcome. An outcome domain is a group of closely related outcome measures, believed to provide information on the same underlying skill or ability.

What is positive effects?

Positive:  strong evidence that intervention had a positive effect on outcomes. This occurs when two or more studies show statistically significant positive effects, at least one of which met WWC evidence standards for a strong design and no studies show statistically significant or substantively important negative effects.

What is quality assurance?

An internal review process for WWC products.

What is a quasi-experimental design?

A design in which groups are created through a process that is not random.For a quasi-experimental design to be rigorous, the intervention and comparison groups must be similar, demonstrating baseline equivalence on observed characteristics, before the intervention is started. Strong quasi-experimental designs will, at best, be rated as Meets WWC Evidence Standards with Reservations.

What is a randomized controlled trial?

A design in which groups are created through a process that is random.Carried out correctly, random assignment results in groups that are similar on average in both observable and unobservable characteristics, and any differences in outcomes between the groups are due to the intervention alone. Strong randomized controlled trials can receive the highest rating ofMeets WWC Evidence Standards without Reservations.

What is statistical significance?

The likelihood that a finding is due to chance rather than a real difference.The WWC labels a finding statistically significant if the likelihood that the difference is due to chance is less than five percent (p = 0.05).

What is a unit of analysis?

The level at which an analysis is conducted. For example, a study that looks at student outcomes will likely conduct the analysis using student-level data.

What is a well-executed randomized controlled trial?

A randomized controlled trial that uses a valid randomization procedure, has low levels of sample attrition, and includes appropriate information to test program effectiveness.