A rigorous, interactive guide to measuring, comparing, and predicting educational phenomena — the deductive logic beneath the numbers, how variables become measurable, how samples support claims about populations, and how to choose and interpret the right statistical test. Built for graduate researchers, evaluators, and educational leaders.
Quantitative research reasons deductively: it begins with theory, derives testable hypotheses, operationalizes concepts as measurable variables, and uses data from a sample to draw defensible inferences about a population. It asks how much, how many, and how related.
| Term | Meaning |
|---|---|
| Null (H₀) | No effect / no difference / no relationship |
| Alternative (H₁) | A specified effect, difference, or relationship exists |
| Population | The full group a claim is about |
| Sample | The observed subset used to estimate it |
| Parameter | A true population value (μ, σ, ρ) |
| Statistic | An estimate from the sample (x̄, s, r) |
Inferential testing asks: if H₀ were true, how surprising is our sample result?
Null hypothesis significance testing is a proof-by-contradiction. You assume the null is true, ask how surprising your data would be under that assumption, and reject the null only when the data are surprising enough. Rejecting the null never proves the alternative — it only says the null is an implausible explanation for what you saw.
Every decision faces two realities — the null is really true, or it is really false — crossed with two verdicts. Two of the four cells are errors.
| Decision \ Reality | H₀ is true | H₀ is false |
|---|---|---|
| Reject H₀ | Type I error False positive (prob = α) | Correct True positive (prob = 1 − β = power) |
| Fail to reject H₀ | Correct True negative (prob = 1 − α) | Type II error False negative (prob = β) |
Lowering α guards against Type I errors but, all else equal, raises β and lowers power. The two risks trade off.
The probability of correctly rejecting a false null — of detecting a real effect when it exists. A common target is .80 (Cohen, 1988).
Larger samples shrink the standard error, making true effects easier to detect. This is the lever the researcher controls most directly.
Large effects are easy to detect; small ones demand large samples. Power analysis must assume a plausible effect size up front.
A larger α or a more reliable, less variable measure both raise power. A one-tailed test has more power than a two-tailed test in the predicted direction.
Stevens' four levels determine which statistics are legitimate. Mistaking one for another — averaging ordinal ranks, for instance — produces numbers that look precise but mean little.
Categories with no rank. Examples: school, program, gender, treatment vs. control.
Allowed: counts, mode, chi-square
Ranked, but distances aren't equal. Examples: Likert agreement, class rank, letter grades.
Allowed: median, percentiles, rank tests
Equal intervals; zero is arbitrary. Examples: temperature (°C), many scaled test scores.
Allowed: mean, SD, correlation, t / ANOVA
A meaningful zero allows ratios. Examples: age, count correct, minutes on task.
Allowed: all of the above + ratios
Reliability is consistency; validity is accuracy. A bathroom scale that reads five pounds heavy every time is perfectly reliable yet invalid. The reverse is impossible: a measure cannot be valid without first being reliable — reliability is a necessary but not sufficient condition for validity.
| Type | What it checks | How |
|---|---|---|
| Test–retest | Stability over time | Correlate scores from the same people on two occasions |
| Parallel / alternate forms | Equivalence of two versions | Correlate scores on two equivalent forms of the instrument |
| Internal consistency | Items measure one construct | Cronbach's α (or split-half, KR-20); α ≥ .70 often acceptable |
| Inter-rater | Agreement between observers | Cohen's κ or intraclass correlation (ICC) across raters |
| Type | Question it answers | Evidence |
|---|---|---|
| Content | Do items cover the whole domain? | Expert judgment; a blueprint mapping items to the construct |
| Criterion — predictive | Does it forecast a future outcome? | SAT scores predicting later college GPA |
| Criterion — concurrent | Does it agree with a present standard? | A new screener vs. an established diagnostic given at the same time |
| Construct | Does it behave like the theory says? | Convergent & discriminant evidence; factor analysis |
Probability sampling supports generalization to a population; the design determines what kind of claim — causal or associational — the data can bear. Select a design to see its logic.
Every member has the same probability of selection. The benchmark against which others are judged.
Divide the population into subgroups (grade, SES), then sample within each — improves precision and representation.
Randomly select intact clusters (schools, classrooms), then study all within. Efficient but less precise.
Choose a random start, then take every k-th unit from an ordered list. Simple, but beware periodicity in the list.
Manipulates the IV and uses random assignment to conditions, so groups are equivalent in expectation. This is the strongest warrant for a causal claim.
Best question: "Does X cause a change in Y?"
Manipulates the IV but without random assignment — using intact groups (existing classrooms). Causal claims are weaker and must address selection threats.
Best question: "Does X plausibly cause Y when randomization isn't possible?"
Measures variables as they naturally occur and quantifies their association — no manipulation. Establishes relationship and prediction, not causation.
Best question: "How, and how strongly, are X and Y related?"
Describes the characteristics of a population or phenomenon as it exists — frequencies, distributions, trends — without testing relationships between manipulated variables.
Best question: "What is the current state of ___?"
| Type | Threat | What it undermines |
|---|---|---|
| Internal | History | An outside event, not the IV, caused the change |
| Internal | Maturation | Participants change naturally over time |
| Internal | Selection | Groups differed before treatment |
| Internal | Testing & instrumentation | The measure itself changes scores |
| Internal | Regression to the mean | Extreme scorers drift toward average |
| Internal | Attrition (mortality) | Non-random dropout biases groups |
| External | Population validity | Results may not generalize beyond the sample |
| External | Ecological validity | Results may not hold in other settings/times |
Two variables can move together for reasons that have nothing to do with one causing the other. Before an association can be read as a cause, it has to survive three classic tests — and even then, only a well-designed experiment closes the gap decisively.
The presumed cause and effect must actually be related — as X changes, Y changes. This is the part a correlation establishes; it is necessary but far from sufficient.
The cause must come before the effect in time. Correlational and cross-sectional data usually cannot establish which came first — the directionality problem.
Rival explanations — especially confounders — must be ruled out. This is where observational designs are most vulnerable (Shadish, Cook & Campbell, 2002).
A confounder is a variable associated with both the presumed cause and the outcome, manufacturing a correlation between them. Ice-cream sales correlate with drowning deaths — but summer heat drives both; neither causes the other.
Random assignment is the engine of causal inference. By allocating participants to conditions by chance, it makes the groups equivalent in expectation on every variable — measured or not, known or unknown — so any post-treatment difference can be attributed to the manipulation.
Descriptive statistics summarize the data in hand. Inferential statistics use the sample to reason — under uncertainty — about the population that produced it.
Significance tells you an effect is unlikely to be zero. Effect size tells you whether it's big enough to matter. Report both. Drag the sample size below to see why.
Difference between two means in SD units.
Benchmarks: 0.2 small · 0.5 medium · 0.8 large.
r ranges −1 to +1; r² is variance explained.
Benchmarks: .10 small · .30 medium · .50 large.
Proportion of variance in the DV explained by the grouping factor.
Benchmarks: .01 · .06 · .14.
Regression fits a line (or hyperplane) that predicts a continuous outcome from one or more predictors. It is the workhorse of prediction and of statistical control — but its output only means what it claims when its assumptions hold.
Ŷ = b₀ + b₁X. The model has a single slope. b₁ is how much Ŷ changes for a one-unit rise in X; b₀ (intercept) is the predicted Ŷ when X = 0.
Equivalent to a Pearson correlation re-expressed in the outcome's raw units.
Ŷ = b₀ + b₁X₁ + b₂X₂ + … Each slope is a partial coefficient: the effect of that predictor holding the others constant. This is how regression "controls for" covariates statistically.
Standardized β coefficients put predictors on a common scale for comparison.
| Assumption | Meaning | How to check |
|---|---|---|
| Linearity | The relationship between predictors and outcome is linear | Residuals-vs-fitted plot shows no curve |
| Independence | Residuals are not correlated with each other | Design (no clustering); Durbin–Watson |
| Homoscedasticity | Residual variance is constant across fitted values | Residual plot shows even band, not a fan |
| Normality of residuals | Errors are approximately normal | Q–Q plot / histogram of residuals |
When the outcome is binary, these break down and logistic regression (modeling log-odds) replaces OLS.
A survey is a measurement instrument, and bad items produce numbers that are precise but wrong. Good design starts before a single response is collected — with items that are clear, neutral, and mapped to the construct.
| Bias | What happens | Guard against it |
|---|---|---|
| Social desirability | Respondents answer to look good rather than truthfully | Assure anonymity; word sensitive items neutrally |
| Acquiescence | Tendency to agree regardless of content | Mix item direction; avoid agree/disagree overuse |
| Leading / loaded wording | Item nudges toward a particular response | Pilot test; use neutral, balanced phrasing |
| Order & context effects | Earlier items shape answers to later ones | Randomize item order where feasible |
| Nonresponse bias | Those who don't respond differ systematically | Maximize response rate; compare respondents to frame |
| Central tendency / straight-lining | Respondents cluster at the midpoint or repeat one column | Reverse-scored items; attention checks |
Answer three questions about your analytic goal. This suggests a common starting test — a heuristic, not a substitute for checking each test's assumptions.
1 · What is your analytic goal?
2 · How many groups or predictors?
3 · What kind of outcome variable?
All three tests below assume their conditions are met (e.g. approximate normality, independence, expected cell counts). When assumptions fail, a nonparametric counterpart — Mann–Whitney, Kruskal–Wallis, Spearman — is the defensible substitute.
A quick map from question to test. Match your goal, number of groups, and outcome type to the row that fits.
| Test | When to use | Example question | Effect size |
|---|---|---|---|
| Independent t-test | Compare the means of two independent groups on a continuous outcome | Do students in the tutoring program score higher than those who are not? | Cohen's d |
| Paired t-test | Compare two related means (pre/post, matched pairs) | Did scores rise from pretest to posttest for the same students? | Cohen's d |
| One-way ANOVA | Compare means across three or more groups | Do three teaching methods differ in average achievement? | Eta-squared η² |
| Pearson correlation | Measure the linear association between two continuous variables | Is study time related to exam performance? | r / r² |
| Linear regression | Predict a continuous outcome from one or more predictors | How well do attendance and prior GPA predict final grade? | R², β |
| Chi-square | Test association between categorical variables / goodness of fit | Is program completion related to enrollment status? | Cramér's V, φ |
| Logistic regression | Predict a binary categorical outcome | What predicts whether a student is retained vs. promoted? | Odds ratio |
| Factorial / two-way ANOVA | Test two or more factors and their interaction on a continuous outcome | Do method and grade level interact to affect achievement? | Partial η² |
| Repeated-measures ANOVA | Compare three or more related means (same subjects over time) | Do scores change across three testing occasions? | Partial η² |
| ANCOVA | Compare group means while statistically controlling a covariate | Do methods differ once we adjust for prior GPA? | Partial η² |
| Mann–Whitney U | Nonparametric alternative to the independent t-test (ordinal / non-normal) | Do two groups differ in ranked satisfaction? | Rank-biserial r |
| Wilcoxon signed-rank | Nonparametric alternative to the paired t-test | Did ranked scores shift from pre to post? | Matched-pairs r |
| Kruskal–Wallis | Nonparametric alternative to one-way ANOVA (3+ groups) | Do three schools differ in ranked engagement? | Epsilon-squared |
| Spearman's rho | Monotonic association between two ordinal / non-normal variables | Is class rank related to attendance rank? | ρ (rho) |
A p-value below .05 is not a certificate of truth. The last decade of methodological reckoning — the "replication crisis" — has shown how ordinary analytic choices can manufacture false positives. Reading results well means knowing where the numbers can mislead.
Each claim below hides a common methodological flaw. Pick the one that best names it.
1 · Observational study: "Students who eat breakfast score higher on state tests, so schools should serve breakfast to raise scores."
2 · "We tested 20 outcomes and found one significant at p < .05, which we report as our key result."
3 · "With 5,000 students, a correlation of r = .04 between an app's use and grades was significant (p < .01), so we recommend the app."
4 · "The treatment–control difference was not significant (p = .20), proving the program has no effect."
A quantitative claim is only as trustworthy as its measures and its transparency. Check the practices your study uses to gauge how defensible it is.
Does the instrument yield stable, repeatable scores? Via internal consistency (Cronbach's α), test–retest, inter-rater agreement.
Content, construct, criterion (concurrent & predictive) validity. A measure can be reliable yet invalid.
Independent replication is the ultimate test of a finding — the antidote to false positives and the "replication crisis."
Pre-registration, open data and code, and reporting all analyses combat p-hacking and HARKing.