Can ChatGPT Do Statistical Analysis? An Honest Assessment of AI in Statistics

Credit first: what can AI genuinely do in statistics?

Large language models like ChatGPT, Claude and Gemini have quietly revolutionized statistics education. Questions like "what does a p-value actually mean?", "what's the difference between ANOVA and a t-test?" or "why do effect sizes matter?" get patient, level-adjusted explanations, repeated as many times as you need. A tool that can explain — at 2 a.m., with examples from your own data — a concept your textbook dispatched in one paragraph is genuinely valuable for learning.

They are also strong at code: ask for an analysis script in R or Python, or even SPSS syntax, and you'll usually get clean, working code. Paste in analysis output and ask "how do I interpret this?" and you'll get a reasonable draft interpretation. Versions that accept file uploads can produce simple descriptive statistics and charts too.

So the question isn't "is AI useful?" — it clearly is. The real question is: can you hand it the statistics of your thesis or paper, the analysis you'll put your name on? That's where the picture changes.

Task by task: where AI is good, where an expert is still needed

The table summarizes where mainstream AI tools (ChatGPT, Claude, Gemini) stand across the statistics workflow as of 2026. Models improve fast, so the strengths may grow; the limits are structural, so they change more slowly.

Perhaps the biggest danger: AI chooses the scope by itself

When you hand your data to a statistician, the work starts inside a frame: your research questions, hypotheses, the measurement levels of your variables and your field's reporting expectations define the scope of the analysis up front. When you tell an AI "run statistical analysis on this data," no such frame exists. There are dozens of technically possible analyses, and the model — without knowing which ones your study actually requires — picks among them on its own. The result is often arbitrary: the same dataset can get one set of analyses in one session and a completely different set in another.

In practice this most often shows up as underdoing: the model produces a few means, percentages and charts — descriptive statistics only — and presents it as a finished analysis, while the hypothesis tests, group comparisons or association analyses your thesis actually needs were never run. Worse, it frequently does this without even examining variable types: taking the mean of a categorical variable, or treating an ordinal scale as continuous, are common failures of exactly this kind. The opposite happens too: it overdoes, stacking dozens of tests whether needed or not, bloating the report — and running many tests without correction undermines the credibility of whatever comes out significant.

What makes this so dangerous is that in both cases the output looks tidy and complete. Spotting what's missing is harder than spotting what's wrong — there is no signal telling you "something is off here." This is precisely the first step of a statistician's job: before any analysis, asking "what does this research question require, and what does this data allow?" and setting the scope accordingly. AI cannot give itself that directive; someone has to give it.

Where AI still struggles in statistics

Most of these limits aren't "not good enough yet" issues; they're structural consequences of how large language models work. That's why caution remains necessary even as model versions change.

• Confidently wrong answers: when AI is wrong, it says so with the same confidence as when it's right. For someone without statistical training, telling the two apart is nearly impossible.

• Number hallucination: especially in long sessions and output interpretation, it can produce p-values, means or confidence intervals that never existed in the actual analysis — often in a convincing-looking table.

• Assumption blindness: unless you ask, it won't raise normality, homogeneity of variance or multicollinearity checks — precisely the first things committees and reviewers look at.

• Missing context: it doesn't know your research design, sampling structure or your field's reporting conventions; its suggestions can be technically valid yet wrong for your study.

• Reproducibility: it can answer the same question differently across sessions. Reproducibility — a basic requirement of scientific analysis — is not guaranteed in a chat interface.

• Data privacy: uploading raw data — especially patient data or personal information — to a public chatbot can conflict with data-protection law and the commitments in your ethics application.

• The accountability gap: when the analysis turns out wrong, AI doesn't fix it, defend it, or bear the consequences. With an expert, there is a named person responsible for the error and its solution.

AI in expert hands: a different tool entirely

Here's the interesting part: the people who get the most out of AI in statistics are the ones who need it least. An experienced analyst spots a wrong test recommendation in two lines, catches a hallucinated p-value by checking it against the real output, asks for the missing assumption checks, and puts the tool to work where it truly shines — speed, drafts, trying alternative approaches. The same tool, in the hands of someone who can't audit its output, can become a machine for accelerating mistakes.

That's why "AI or a statistician?" is the wrong question. The right question is: "who guarantees this analysis is correct?" AI is a tool; the party who carries responsibility, evaluates your data in its context, and produces a rationale you can defend before a committee is always human. The strongest combination isn't pitting one against the other — it's expertise using the tools under supervision.

This is exactly where GetBayes stands: our job is making sure that whatever tools are involved, what comes out the other end is a defensible analysis. We regularly hear from researchers who got partway with AI: "I got this far, but I'm not sure it's right." We take the analysis over from there — verify the numbers against real outputs, complete the missing assumption checks, and make the report stand up to committees and reviewers.

5 rules for using AI safely in your thesis

You don't need to shun AI entirely; you need to use it within the right boundaries. These five rules remove most of the risk:

1. 01

   Use it to learn, not to decide

   It's excellent for explaining concepts, drafting code and sparking interpretation ideas. But don't leave thesis-defining decisions — test selection, sample size, the results you'll report — to AI alone.

2. 02

   Think twice before uploading raw data

   Don't upload files containing personal information or patient data to public chatbots; at minimum, anonymize first. Remember that your ethics application includes commitments about how the data will be handled.

3. 03

   Independently verify every number

   Confirm every p-value, mean and table AI gives you by running the same analysis in SPSS, JASP or R. Never put a number you couldn't verify into a thesis or paper.

4. 04

   Read your university's and journal's AI policy

   Many universities and journals now require disclosure of AI use; some restrict it for critical steps like data analysis. Learn the policy up front to avoid surprises at your defense or in peer review.

5. 05

   Consult an expert at the critical thresholds

   At hard-to-reverse points — test selection, power analysis, unexpected results, the final report — showing the work to an expert is always cheaper than losing weeks on the wrong path.

And the future? Honestly: we don't know — but we know today

These tools have improved at a pace nobody predicted, and they keep improving. Nobody can honestly say today how much of statistical analysis will be safely automated in five years — and we won't pretend to. Be a little skeptical of anyone who speaks with certainty here.

But we do know today: in 2026, for work you will defend before a committee, reviewers or an ethics board, statistical expertise is still indispensable. And as AI spreads, the role of expertise isn't shrinking — it's shifting: experts no longer just run analyses, they increasingly serve as the referee for the question "is this output actually correct?" Auditing the tool is itself a form of expertise.

In short: use AI on your learning journey and don't fear it — but before you sign off on your thesis statistics, make sure there is expertise in the process that can stand behind that signature. If that expertise is yours, wonderful; if not, we're here.