What can go wrong with a testing decision?
You will be able to: Identify errors in context and connect β with power.
What can go wrong with a testing decision?
A team tests whether a new reminder increases participation above .50. It could claim an increase that is not real, or miss an increase that is real.
A useful starting point: How do you finish a one-proportion test in context? →
Words and symbols before equations
- Type I error
- Rejecting a true null hypothesis.
- Type II error
- Failing to reject a false null hypothesis.
- β
- Probability of a Type II error at a specified alternative value.
- Power
- Probability of rejecting H₀ at a specified true alternative; 1−β.
What this picture assumes
Greater-than normal z-test of p₀=.50. Exact binomial enumeration evaluates the finite-sample rejection rate of that approximate test rule under the null and selected alternative. Its actual false-alarm rate need not equal nominal α because counts are discrete.
Read the picture in three steps
- Read the axes and labels first. Identify what each symbol and line represents. Read the units and fixed conditions before comparing quantities.
- Nominal α=0.05; actual false-alarm probability=0.04431. At true p=0.6, power=0.62253 and β=0.37747. Exact binomial evaluation of a greater-than normal z decision rule.
- Check what the picture assumes below. Use the Explore task to predict one change before moving a control.
Connect the picture to the mathematics
If participation truly remains .50 but the test rejects H₀, that is a Type I error: a false alarm. If participation truly increases but the test fails to reject, that is a Type II error: a missed increase.
α controls the intended Type I error rate under the null, approximately for a normal test. β and power depend on which alternative value is actually true, as well as n, α and the procedure.
Consequences guide planning. A false claim could waste resources; a missed improvement could delay a helpful program. Neither error can be identified with certainty from one observed p-value when the population truth is unknown.
A worked example, step by step
A test has power .80 when the true share is .65. What is β there, and what would a Type I error mean for H₀:p=.50?
- Power .80 is the chance of rejecting when p=.65.
- β=1−.80=.20 at that alternative.
- A Type I error means rejecting even though p=.50 is true.
- The two probabilities condition on different population truths, so they are not complements of each other.
α and β do not add to one. Power and β do, at the same specified alternative.
Does a rejected null tell us whether a Type I error occurred?
Compare with an explanation
Not by itself; the true population state is generally unknown.
Predict. Change one thing. Explain.
Read the null and alternative rejection probabilities separately. Explain which one measures false alarms and which one measures successful detection.
On narrow screens, swipe or scroll diagrams sideways to read all labels.
Nominal α=0.05; actual false-alarm probability=0.04431. At true p=0.6, power=0.62253 and β=0.37747. Exact binomial evaluation of a greater-than normal z decision rule.
| Quantity | Probability / count |
|---|---|
| Nominal α | 0.05 |
| First rejecting success count | 59 |
| Actual null rejection probability | 0.044313 |
| Power at true p=0.6 | 0.622533 |
| β at that alternative | 0.377467 |
Greater-than normal z-test of p₀=.50. Exact binomial enumeration evaluates the finite-sample rejection rate of that approximate test rule under the null and selected alternative. Its actual false-alarm rate need not equal nominal α because counts are discrete.
Explain what you noticed: Answer the investigation prompt above. State one observation and explain it using the probability values, reference groups, graph scales or model assumptions. Identify what the representation cannot tell you.
Apply the idea to a fresh problem Practice →Show what you understand.
Two original questions are a starting check, not proof of mastery. Explain your choice before revealing the answer.
Original written challenge
4 points · self-check · not an official AP questionA test of whether a return rate exceeds .10 fails to reject when the true rate is .18. Name the error and explain power .75 at .18.
This response is not submitted or saved. Copy it before leaving.
Compare with the answer and four-point rubric
- 1 point: The null is false at .18 for the greater-than alternative.
- 1 point: Failing to reject is a Type II error: missing the elevated return rate.
- 1 point: Power .75 means a 75% chance of detecting that specified increase with the procedure.
- 1 point: β=.25 at .18; it need not be the same at another true return rate.
Accept equivalent correct methods and explanations. This is a Refresh Kid teaching rubric, not an official AP scoring guideline.
Retrieve it before you reveal it.
RECALL 1When does a Type I error occur?
When a true null is rejected.
RECALL 2Why must power specify a true alternative?
Detection probability depends on effect size.
RECALL 3Can α and β be interpreted under the same truth?
No; α is under the null, β under a specified false-null alternative.
Revisit these tomorrow and a week later. Try a fresh problem and explain why the method applies.
What can go wrong with a testing decision?
- Type I: reject true H₀.
- Type II: fail to reject false H₀.
- Power=1−β at a specified alternative.
Remember: α and β do not add to one. Power and β do, at the same specified alternative.
Conditions: Greater-than normal z-test of p₀=.50. Exact binomial enumeration evaluates the finite-sample rejection rate of that approximate test rule under the null and selected alternative. Its actual false-alarm rate need not equal nominal α because counts are discrete.
Refresh Kid · AP Statistics Unit 3 · Objectives 3.8.A, 3.8.B, 3.8.D · Review edition
Framework, scope and review status
Mapped to College Board, AP Statistics CED, Topic 3.8, objectives 3.8.A, 3.8.B, 3.8.D. Framework effective Fall 2026, checked September 17, 2026. Unit 3 includes inference for one and two population proportions, errors and power, and chi-square homogeneity/independence tests; it is part of the revised five-unit course.
Examples and datasets are synthetic, independently authored teaching material. Inference requires a justified design and appropriate counts. Intervals use observed proportions; null tests use their reference-model proportions. The revised CED states chi-square expected counts should be greater than 5; this unit follows that wording even though some companion texts use at least 5. Normal and chi-square inference are approximate. The chi-square goodness-of-fit test is not included in this unit’s official scope.
The Organic Chemistry Tutor companion title and destination were located; the full video was not reviewed. Khan Academy’s destination was checked, but its lesson content was not fully readable by the research tool. OpenStax provides optional reference reading. No provider scripts, questions or graphics were copied. Refresh Kid is not affiliated with these providers.
GitHub’s 3D website collection informed optional spatial inspection. Our original categorical count grid uses self-hosted Three.js with its MIT license. Two rows and three columns organize synthetic people into categorical cells. Each stacked block represents one person. Camera rotation changes only the view; exact counts, expectations and contributions are always available in the 2D table. Inference curves and intervals remain 2D; use the exact table rather than apparent 3D size for comparisons. Complete labeled diagrams, count tables and explanations remain available without 3D.
Independent teacher review and observation of students remain pending. Technical checks do not certify statistical accuracy, accessibility or learning effectiveness. This is a review edition.
Released AP Statistics questions and scoring guides are optional. Older exams use the earlier framework, so check alignment before selecting parts. All practice on this page is original, not official AP material.
Learn → Explore → Practice → Review is informed by the IES learning guide. This implementation has not yet been evaluated with learners.
Want to work through this with a tutor?
Bring your question about What can go wrong with a testing decision? Your explanation and answers remain free to access.