“I’ve been in the advice business for 30 years. The most dangerous thing isn’t that AI gets it wrong — it’s that every other source gets it wrong too, and nobody wants to say that out loud.”
A Boston University economics professor just published a warning that should worry anyone within 10 years of retirement: AI chatbots are giving dangerously wrong Social Security claiming advice. On the same day, CNBC reported potential changes to how the earnings test affects retirees who work, and disabled Americans reported new barriers to accessing their benefits.
But here’s what nobody is saying out loud: the government’s own agents have been giving wrong answers for decades. And at least the chatbot doesn’t pretend its answers are official.
Table of Contents
Each weekday I send one short email: one money story that caught my attention, why it might matter to you, and one manageable thing you can do with it. A couple of minutes, and a dad joke.
In the latest issue (Oct 9): Does Financial Literacy Make Financial Sense?
Free. I sell nothing. Unsubscribe any time.
The Problem Is Bigger Than AI
Yes, ChatGPT and Copilot hallucinate. They make up Social Security rules that don’t exist and miss exceptions that do. Professor Laurence Kotlikoff documented specific examples where AI chatbots gave advice that would cost retirees $50,000 to $100,000 over a lifetime.
But let me tell you something that might surprise you: the IRS literally tells you not to trust its own phone agents.
What the IRS Actually Says About Its Own Advice
The IRS Internal Revenue Manual (Section 32.3.1.9) states that oral advice from IRS agents is “not binding on the Service” and “cannot be relied upon as a basis for obtaining retroactive relief.”
Federal tax law (IRC Section 6404(f)) only provides penalty relief for erroneous advice that was given in writing, in response to a specific written request. Phone calls don’t count.
Translation: if an IRS agent tells you something wrong on the phone and you follow that advice, you’re still on the hook.
And this isn’t a theoretical problem. The Government Accountability Office has been testing IRS phone agents for decades:
By 2010, accuracy had improved to about 95%. Then funding cuts gutted the workforce. By 2022, the IRS was answering fewer than 2 out of every 10 calls. This year, after a 27% workforce reduction, wait times are up 70% — and here’s the part that should concern you most: the National Taxpayer Advocate’s 2025 report found that the IRS doesn’t even measure whether its agents give correct answers anymore. They track whether calls get answered. Not whether the answers are right.
Social Security Is the Same Story — Arguably Worse
The Social Security Administration has 2,700-plus rules governing benefits. When to claim. How spousal benefits work. What happens if you work while collecting. How disability interacts with retirement. Getting any one of these decisions wrong by a single year can cost you tens of thousands of dollars over your lifetime.
A 2016 GAO study observed 30 in-person claims at SSA field offices. In 8 of 26 cases where waiting would have produced higher benefits, the agent never mentioned that waiting was an option. Only 7 of 18 claimants who were affected by the earnings test received complete information about how it worked.
Here’s the Part That Should Make You Angry
When the Social Security Administration makes a mistake — overpays you based on their own wrong calculation — you have to pay it back. The SSA Inspector General found that 87% of sampled overpayments were caused by the agency’s own incorrect computations. The current clawback rate is up to 50% of your future benefits until the overpayment is recovered. There is no federal law equivalent to the IRS’s written-advice protection. If SSA tells you the wrong amount and pays you too much, that’s your problem.
And the situation is getting worse. The SSA just went through the largest staffing cut in its history — 7,000 workers. Wait times have been reported at up to 5 hours. Performance metrics have been removed from the public website. If you couldn’t get reliable answers before the cuts, what do you think is happening now?
But What About the “Experts”?
This is where the story gets uncomfortable. The standard advice is “hire a fee-only financial planner.” And I’ve given that advice myself. But let’s be honest about what that really means.
That financial planner is working from the same government documents that the government’s own agents can’t interpret correctly. They’re reading the same IRS code sections, the same SSA rule book with 2,700-plus provisions. And increasingly, they’re using AI tools to help them analyze your situation — the same AI that Professor Kotlikoff just proved gives $100,000 mistakes with a straight face.
I’m not saying don’t hire a professional. I’m saying stop pretending there’s a simple answer to this.
Think About What “Getting the Right Answer” Actually Requires
To know whether you should claim Social Security at 62, 67, or 70, someone needs to correctly interpret how dozens of interacting rules apply to your specific situation — your earnings history, your spouse’s record, your health, your other income, how the earnings test works, how taxation of benefits works, how Medicare premiums interact with income. A single wrong assumption in any of these areas can cost tens of thousands of dollars. Getting the RIGHT answer requires hours of skilled analysis by someone who genuinely understands the system. And even then — how do you verify that answer is correct? Who checks the checker?
It’s hard to be outraged that AI gets this wrong when every other cog in the machine gets it wrong too. The government agents get it wrong. The IRS won’t stand behind its own answers. The SSA claws back its own mistakes from your check. And now we’re told the solution is to pay a professional who’s working from those same documents with those same AI tools.
I’ve personally experienced this. Years ago, dealing with the Department of Education on student loan issues, I pointed out that their process didn’t match what their own manual said. The response I got back? “That’s how we do it.” Not “you’re right, we’ll fix it.” Not “here’s why the manual doesn’t apply.” Just: that’s how we do it. The manual says one thing. The agency does another. And you’re supposed to somehow navigate the gap.
Where does “right” actually exist? And how would you prove you had the right answer without a significant investment in repairing a bad situation after the fact?
What You Can Actually Do
I don’t have a neat answer. Anyone who tells you they do is selling something. But after 30 years, here’s what I know works better than the alternatives:
- Get everything in writing. From everyone. If the IRS tells you something on the phone, it’s legally worthless — their own manual (IRM Section 32.3.1.9) says so. If a financial planner tells you something, get the recommendation in writing with their reasoning. If the SSA tells you your benefit amount, get the computation sheet. Written records are the only thing that gives you any recourse when someone is wrong.
- Never trust a single source. Run your Social Security question through the SSA’s official calculators AND a professional AND the IRS Interactive Tax Assistant for the tax implications. If two of three disagree, you’ve found a problem before it costs you money.
- For disability benefits: Don’t navigate appeals based on what any single source told you. Contact a disability rights legal aid organization through LawHelp.org — they deal with the actual law, not interpretations of it.
- Protect what’s already protected. Before you make any move that touches protected retirement accounts, understand that your 401(k) and IRA are shielded in bankruptcy. Don’t let anyone — not an AI, not a phone agent, not a debt settlement company — talk you into cashing out retirement to pay dischargeable debt. That’s one area where the law IS clear.
- When the stakes are high, invest in verification. For a Social Security claiming decision worth $50,000+ over your lifetime, paying $200–$500 for a fee-only financial planner is still your best option — not because they’re guaranteed to be right, but because they’re the only source who will put their analysis in writing and explain their reasoning. Ask them how they arrived at their recommendation. Ask what assumptions they made. Ask what happens if those assumptions are wrong. A good professional won’t be offended by those questions. A bad one will.
What people believe: “There’s a reliable place to get the right answer about my Social Security / tax / benefits situation.”
What’s actually true: There has never been a reliable, free, no-consequences source of personalized financial guidance. The IRS won’t stand behind phone advice. The SSA claws back its own errors. AI hallucinates. Paid advisors work from the same flawed sources — and increasingly use the same AI. The best protection isn’t finding the “right” answer — it’s getting everything in writing, cross-checking multiple sources, and understanding what you’re risking before you act.
If AI and advisors are already getting Social Security wrong today, imagine the confusion when COLA caps and trust fund depletion change the math entirely. Know your real numbers — not what a chatbot tells you.
The Bottom Line
And if you’re wondering when to actually claim Social Security, I did the breakeven math everyone quotes about Social Security claiming age — and the answer isn’t what you’d expect.
The outrage over AI giving wrong Social Security advice misses the larger, uglier truth: the entire system for helping regular people make $100,000 decisions about their retirement, taxes, and benefits has always been broken. Government agents give wrong answers and face no consequences. The agencies claw back their own mistakes from your benefits. And the professionals who are supposed to help are working from the same flawed materials, increasingly assisted by the same flawed AI. There is no magical source of “right.” The best you can do is get everything in writing, never trust a single source, and — when the stakes are high enough — pay for a professional who will show their work. Then verify it yourself. That’s not a satisfying answer. But it’s an honest one.
I wish I could end this by pointing you to the one reliable source that gets it right every time. I can’t, because it doesn’t exist — and after 30 years in this space, I think it’s more dangerous to pretend it does than to say so out loud. The system was never built to give regular people dependable answers on the decisions that matter most. That’s not new. But with AI making wrong answers more accessible than ever, and government agencies less staffed than ever, the gap between what people need and what they can actually get is wider right now than at any point in my career. Use everything I’ve shared here as context — not instruction. Cross-check it. Question it. And don’t let anyone — including me, and especially not a chatbot — make these decisions for you.
If this opened your eyes, send it to someone who’s about to make a big Social Security or retirement decision. They deserve to know what they’re really up against.
Dealing With Debt? Understanding your options is the first step. See how all your debt relief options compare — including ones most sites won’t tell you about. The Find Your Path quiz gives a recommendation based on your actual numbers, and the Scam-O-Meter checks any company’s complaint history before you sign. Federal Reserve research shows bankruptcy filers recover faster than those who don’t file.
