The one idea
Before you repeat a number, ask one question:
Roughly what number did I expect, and is this within a factor of a few of it?
Most bad numbers are not subtly wrong. They are wrong by a factor of ten, a thousand, or a sign — a units error, a double count, a filter that did not apply. Those are catchable in your head in about fifteen seconds, and catching them is most of what people mean when they call someone good with numbers.
Before you believe a number, guess roughly what it should be. If the real one is wildly different, something is wrong somewhere.
Estimate the answer independently from a different direction, check the base of every percentage, and check the sample size behind any comparison.
Validate by order-of-magnitude reasoning, distinguish absolute from relative change, check the distribution before quoting a mean, and treat any observed association as unexplained until a mechanism and confounders are addressed.
Check one: order of magnitude
You do not need the right answer. You need to know whether it is in the right neighbourhood, and you get that by building the number a different way.
A company with 50 people reports annual travel spend of ₹4 crore. Divide: that is ₹8 lakh per person per year — roughly a trip every fortnight for everyone, including people who never travel. So the figure includes something else, covers more than a year, or is wrong.
You have not audited anything. One dividing question, and now you know what to ask the finance team.
A dashboard says the website had 2.3 million visits last month. The company has maybe 4,000 customers and does no advertising.
2.3 million visits is roughly 77,000 a day. Nothing in the business produces that. The realistic explanations are bot traffic, a tracking script firing more than once per page, or the metric counting page views and calling them visits.
Reporting 2.3 million without asking is how a false number enters a board deck and stays there for a year.
Check two: percentage of what base
Every percentage has a base, and the base is usually left out.
- 40% improvement from 5% to 7% is two percentage points. A relative change dressed as a big one.
- Doubled from 3 users to 6 is not a trend.
- "Costs rose 15%" — of total cost, or of one line item?
- "30% of respondents" — out of how many respondents? 30% of 10 is three people.
The distinction to keep straight: a percentage point is the arithmetic difference (5% to 7% is 2 percentage points). A percent change is relative (5% to 7% is a 40% increase). Both are true. One of them is chosen to impress.
Whenever you see a percentage, ask for the two raw numbers underneath. If nobody can produce them, that is your finding.
Check three: average versus median
The average (mean) is the total divided by the count. The median is the middle value when you line them all up. When they differ a lot, the average is being pulled by extremes, and quoting it alone misleads.
Nine employees earn ₹5 lakh. One founder earns ₹95 lakh.
- Average: ₹14 lakh.
- Median: ₹5 lakh.
"Average salary is ₹14 lakh" is arithmetically correct and describes nobody in the company. This is the standard trap in salary figures, house prices, response times, order values and time-on-page — all of them skewed by a long tail.
Recommendation Use the median for anything that can have a long tail — money, durations, counts
of things people do. Use the average when the values cluster around a centre and
you need to combine it with a total. When in doubt, report both; if they are
close, the average is safe, and if they are far apart, you have just found the
story.
Check four: correlation is not causation
Two things moving together does not mean one caused the other. Three other explanations need ruling out first, and in workplace data one of them is usually the real answer.
Support says: teams on our premium plan raise 30% fewer tickets, so premium reduces problems — upgrade everyone. Alternatives that fit the same data:
- Reverse direction. Teams with fewer problems have the budget and calm to upgrade. The low ticket count caused the upgrade, not the other way round.
- A third cause. Larger, better-resourced companies buy premium and have in-house IT that solves problems before a ticket is raised. Company size explains both.
- Selection. Premium customers get a dedicated contact and raise issues by phone, so their problems never become tickets. The problems did not fall; the counting method changed.
The useful move is not to dismiss the finding. It is to ask: what would I expect to see if the causal story were true, that I would not see otherwise? Here, tickets should drop for a team after it upgrades. Look at the same teams before and after, and you have a real test.
Check five: small samples are noisy
A conversion rate of 40% from 5 visitors means two people. Run it again tomorrow and it could be 20% or 60% with nothing having changed.
Rough working guides, not rules:
- Under about 30 observations, treat any difference as a hint, not a finding.
- A rate built on fewer than roughly 10 events in either group is noise.
- The most extreme values in any ranking are almost always the smallest samples. The "best performing" and "worst performing" branches on a list are frequently just the two smallest ones.
Always report the denominator. "Top region, 62% satisfaction" hides everything. "Top region, 62% satisfaction, from 13 responses" tells the truth.
You are told your team's bug-fix rate improved 25% this month. Ask two questions: how many bugs last month, and how many this month.
The answers are 8 and 10. That is two bugs, in a month with a holiday in it. The correct response is not to dispute the improvement — it is to say the number is too small to read either way, and to suggest a three-month view.
Saying that in a meeting, politely, is one of the fastest ways to be taken seriously early in a career.
The sixty-second check on any number you are handed
1 of 6Say what you expected before you look properly. An order of magnitude is enough: hundreds, thousands, lakhs. If you cannot form an expectation, you do not yet understand the metric, and that is the real problem.
Try this
A report states: "Customers who use our mobile app spend 3x more than those who do not. We should push everyone to install the app."
Give three reasons the conclusion might not follow.
Spotting a misleading chart
Three things to check on any chart before you accept its message.
Where does the y-axis start? A bar chart starting at 95 instead of 0 turns a 2% difference into a bar three times taller than its neighbour. Bars encode length, so a truncated bar axis is a distortion. Lines may reasonably not start at zero — but check the range, because a narrow one exaggerates every wobble.
What is the interval on the x-axis? Evenly spaced points that are really one month, one month, then two years produce whatever slope the author wanted.
What was left out? A chart starting in 2023 when the data goes back to 2015 has had a beginning chosen for it.
Your challenge
Level 3 · IndependentFind a real statistic in a news article, an ad, or a company report — one with a percentage in it.
Produce a short note that: states the claim, works out or asks for the raw numbers underneath, gives your own order-of-magnitude estimate from a different direction, and states one alternative explanation for any causal claim.
Success criteria: someone reading your note can tell whether the original claim is solid, exaggerated, or unfalsifiable as stated — without going to the source.
What people usually get wrong
- Repeating a number because it was in a slide. Once you say it aloud, it is your number.
- Quoting a percentage without the base. It is the most common way true numbers mislead.
- Using the average on money. Income, order value, house price and donation size all have long tails. Report the median.
- Treating a ranking as a finding. The top and bottom of any list are dominated by the smallest samples.
- Saying "caused" when you mean "associated with". One word, and it changes what people will do next.
- Comparing a partial period to a full one. Eleven days of this month against thirty of last month always shows a collapse.
How someone experienced does it
Experienced people carry a handful of reference numbers for their own domain — typical order value, headcount, average deal size. Not to quote, but as a ruler. A new number gets divided by one of these and its plausibility is immediate. Building that set takes a few weeks of attention and pays back for years.
They also state uncertainty in the number itself. "Roughly 4,000, and I would not bet on the second digit" is more honest than "4,127". False precision is a claim you did not mean to make, and someone will hold you to it.
And they distinguish a number that is wrong from a number that is surprising. The instinct on seeing an unexpected figure is to hunt for the bug, which is right most of the time. But sometimes the data is fine and the mental model is wrong, and that is where the valuable findings live. Check the data path first; if it holds up, take the surprise seriously rather than quietly dropping it.
When not to use this
Do not apply this to every number in casual conversation. Interrogating a colleague's rough figure in a hallway makes you tiring, not rigorous.
The check earns its place when a number will drive a decision, appear in something you sign, or be repeated by someone else on your authority. For those, sixty seconds is cheap.
Prove it
Take a number you produced yourself in a previous lesson of this module. Write half a page: what you expected before you ran it, how you rebuilt it from a different direction, the denominator of any percentage in it, and how confident you are with a reason.
The value is in doing this on your own work, where it is uncomfortable.
Keep learning this
Paste this into any AI assistant. It turns the assistant into a tutor that tests you instead of just answering you.
Act as an experienced practitioner who is good at teaching. I have just learned checking whether a statistic is plausible and what its base is. Assume I am intelligent but relatively new to this — treat me as beginner level. Work through this in order, and wait for my reply at each step: 1. Ask me 5 questions that test whether I actually understood checking whether a statistic is plausible and what its base is. Do not reveal the answers yet. 2. After I answer, tell me which parts I got right, which I got wrong, and which I only half-understand. Explain only what I misunderstood — do not re-teach what I already know. 3. Give me one practical challenge based on something I could genuinely encounter at work or in daily life. Do not solve it for me. 4. Evaluate my solution the way an experienced person would judge it, including what a professional would have done differently. 5. Tell me what to learn next, and why that comes next. 6. Give me trustworthy sources for deeper study — prefer official documentation, primary research or standards bodies over blogs and videos. Rules for you: no buzzwords. No motivational filler. Say "I'm not certain" when you are not certain, and tell me which parts of your answer I should verify myself. Clearly separate facts from your recommendations and your opinions.
Become independent at this
Use this when you want a path from where you are to actually good, with checkpoints you can test yourself against.
I want to become independently capable at sanity checking numbers and spotting misleading statistics — not permanently dependent on AI, tutorials or step-by-step guides. Design a progression for me with five stages: Beginner, Guided practice, Independent practice, Real-world application, Professional level. For each stage tell me: - what I must know - what I must be able to do without help - the mistakes people make at this stage - one practical challenge - one real project that would prove I reached this stage - one way I can test myself honestly Then tell me the signals that I am ready to move to the next stage, and the signals that I have skipped ahead too early. Keep the theory to the minimum I actually need. Focus on ability I can transfer to situations you and I have not discussed.