The short answer
Write the drafting decision and comparison plan before inspecting the outcome. Specify the unit, conditions, window, observation age and missing-data rule. Record how conditions were assigned. Different public posts usually reach different audiences, so an ordinary before/after comparison cannot by itself prove that one writing change caused a gain.
What you’ll take away
- Name the decision and metric before seeing the result.
- A collection of public posts is not automatically a randomized A/B test.
- Missing records and changing audiences limit what the comparison can establish.
Choose a decision small enough for the evidence
Start with the change you might make in the next draft: a clearer explanation, a different demonstration order or a shorter introduction. Define what would count as useful evidence and what would leave the question unresolved. Do not start with a desired lift and then select whichever metric looks favorable.
Keep the task separate from a product-wide revenue or audience claim. You may decide that a version is easier to maintain without establishing that it produces more engagement. A learning decision can be useful even when the available observations cannot support a causal performance statement.
Record assignment and competing explanations
NIST’s design guidance treats objectives and experimental design as choices made before the experiment, and randomized designs assign conditions to experimental units randomly. Merely labeling two drafts A and B does not perform that assignment. Posting A on Monday and B on Friday leaves timing, audience and other differences in the comparison.
Write down which factors actually stayed comparable and which did not. Topic, distribution, format, account, observation age and collection rules can change together. Even a balanced number of posts does not make those differences disappear. Do not promise a clean isolated effect when the real design cannot deliver one.
Inspect the complete six-record exercise
The constructed plan uses entered impression events at a declared seven-day post age. A has 80, 100 and 120; B has 85, 105 and 900. Both medians are easy to reproduce: 100 and 105. B’s mean is about 363.3, dominated by the 900-event record.
The exercise is deliberately unrandomized and does not establish a causal gain. The summaries answer different descriptive questions, so choose the summary before seeing the appealing result. Six records here are a teaching example, not a recommended universal sample size, a power calculation or evidence about real creators.
| Constructed condition | Median events | Mean events |
|---|---|---|
| A: 80 / 100 / 120 | 100 | 100 |
| B: 85 / 105 / 900 | 105 | About 363.3 |
Keep missing outcomes and stopping decisions visible
The separate missing example leaves B2 unavailable. It does not replace it with zero, the previous value or the value that would make the conditions equal. Record whether the declared comparison must wait, or whether a clearly labeled available-record description answers a narrower question.
Also record exclusions and the stopping rule before inspecting outcomes where practical. Ending the review as soon as one version looks better changes the evidence you are describing. A screenshot of a favorable moment is not a completed comparison under the original observation plan.
Write the narrow conclusion first
A supported exercise conclusion is that the entered B median is five events higher, while the mean is heavily influenced by one record. It does not say B caused better distribution, reached more different people or produced customers. The same counts can fit different explanations.
State the uncertainty and your next drafting decision separately. If the source cannot provide the needed unit or matched observation age, leave the comparison unresolved. Do not join private viewer identities or bypass collection preferences to make incompatible systems appear comparable.
Keep a short plan and a complete observation record
The bilingual worked file includes every example row, the prior plan and bounded interpretation. The blank record prompts for conditions, assignment status, metric/window, missing/excluded records and allowed inference. Complete it privately before running your own authorized comparison; this page performs no post or participant action.
The optional local checker verifies explicit medians, the mean, the unavailable marker and original file identities. It is not a general statistical analysis service or a certification of causal design. Actual randomized reader tasks, real preferences and payment outcomes need their own appropriate protocol and evidence.
Sources & product references
Product details reflect the publication date. External features and interfaces may change.
- NIST: choose objectives and design before the experiment
- NIST: randomized assignment of experimental units
- Original complete constructed cases (JSON)
- Complete readable worked examples (English)
- Vollständige lesbare Beispiele (Deutsch)
- Original bilingual examples, private records and optional local checker (ZIP)
