# Review a prediction claim on outcomes it did not already know

Constructed teaching cases with full context. No actual people, accounts, payments, research participants or forecasts.

Freeze the prediction question, information available at that time, target horizon and evaluation rule. Separate training/tuning from later held-out outcomes and compare with a simple prior-only baseline. Future measurements, duplicate objects and test-informed preprocessing can inflate performance. A small backtest is not proof of reliable future creator results.

## Define what is predicted and when

An analytics claim might predict a post’s impressions at 24 hours before publication. Record the target definition, source, prediction timestamp and features available then. A24-hour reaction count is future information for that question, even if it exists in today’s exported worksheet.

If the real tool cannot provide predictions, timestamps or eligible held-out outcomes, record that limitation. Do not call a ranking of already successful posts a forecast. A retrospective explanatory model and a deployable pre-publication prediction answer different questions.

## Keep later outcomes outside training and tuning

For a future-facing question, choose an appropriate time-based evaluation and avoid training on later outcomes to test earlier ones. Keep duplicate originals or dependent groups together where the deployment question requires it. The intended generalization—future posts by the same author or entirely new authors—changes the suitable split.

scikit-learn warns that preprocessing or feature selection fitted on test data leaks information. Fit learned transformations on training data, apply them to held-out data and preserve the exact recipe. A pipeline can help enforce that separation; it does not automatically solve an inappropriate split or hidden future feature.

## Choose a prior-only baseline and meaningful error

Use a simple comparator available at prediction time, such as a declared training-history median. Choose an error measure before reviewing the held-out result. For nonnegative impression counts, mean absolute error expresses average absolute error in the same count unit; it is not a percentage of people or proof of business usefulness.

Retain every eligible test observation and prediction, including poor outcomes. If you repeatedly tune on the test set, it becomes part of selection and a fresh appropriate evaluation is needed. Do not publish only the most favorable time period or author group as the overall performance.

## Resolve three constructed prediction reviews

H1 — Constructed history 80,100,120 gives a frozen baseline 100. Later toy targets 110,90,130 have baseline absolute errors 10,10,30: mean 16.67. Supplied illustrative predictions 115,95,125 have mean error 5. Decision: retain the declared prediction-time rule and all three results. Outcome: a calculation on supplied fixtures, not a real trained model, forecast receipt or validated future advantage.

H2 — Constructed leakage: a model predicts 24-hour impressions using the same post’s24-hour reactions, although the promised prediction is before publication. Decision: reject that feature for the stated timing and re-evaluate an actual prior-only method if available. Outcome: not an acceptable forecast merely because its retrospective score is high.

H3 — Constructed split mismatch: irregularly timed posts are treated as equal-duration rows, and repost copies enter both training and test. Decision: choose an appropriate calendar/group separation and preserve originals. scikit-learn’s TimeSeriesSplit documentation requires equally spaced samples for comparable fold durations. Outcome: no automatic proof of validity from selecting a named cross-validator.

## Keep uncertainty, scope and actual use distinct

Three invented test values cannot establish robust performance across changing accounts, topics or distribution. Report sample size, source gaps, time horizon and error distribution rather than promising the same score for the next post. Inspect whether the prediction would support a useful real decision at all.

Accurate outcome prediction would still not prove that changing a feature causes the outcome. A feature associated with high impressions may be a proxy for author, topic or exposure. Keep predictive evaluation, causal effect and reader/customer value as separate claims.

## Retain a complete evaluation contract

The worksheet keeps target/unit, prediction time, feature availability, train/tune/test dates, object/group rules, preprocessing, frozen baseline, all predictions/errors and limits. The offline packet reproduces the tiny arithmetic only; it trains no model, requests no account and installs no statistical library.

Current primary leakage and TimeSeriesSplit documentation was read on 1 November 2026. All H-cases are constructed. No real predictive service, private analytics or future content performance was tested or promised.

## Primary sources

- [scikit-learn: preprocessing and data leakage](https://scikit-learn.org/stable/common_pitfalls.html)
- [scikit-learn: time-ordered evaluation and equal-spacing condition](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html)