5 Reasons to Think Twice Before Using ChatGPT—or Any Chatbot—for Financial Advice

I’ve used ChatGPT to help me build a budget before, and it was genuinely helpful. After I input my monthly salary as well as my standard utilities and recurring expenses, the chatbot drafted a few solid options, and I tweaked them into penny-pinching perfection. I’m admittedly part of the growing number of people turning to chatbots, like Anthropic’s Claude, Google’s Gemini, and OpenAI’s ChatGPT, for financial advice. “Millions of people turn to ChatGPT with money-related questions, from understanding debt to building budgets and learning financial concepts,” says Niko Felix, an OpenAI spokesperson, when reached for comment. “ChatGPT can be a helpful tool for exploring options, preparing questions, and making financial topics easier to understand, but it is not a substitute for licensed financial professionals.” OpenAI’s Terms of Use state that the AI tool is not meant to replace professional financial advice. While you may consider chatbots to be practical financial assistants, it's always worth keeping the limitations of these AI tools in mind. Beyond miscalculations, here are five additional reasons to approach them with skepticism when it comes to money tips. AI Still Confidently Outputs Incorrect Answers When I ask ChatGPT for help managing my money smarter, the bot is confident in its responses, often laying out what seems like solid reasoning behind each bullet point of advice. But always keep in mind that chatbots can weave convincing errors into outputs. OpenAI has reduced the rate of hallucination in more recent model releases, but chatbot tools still output errors. “There seems to be this sense emerging, at least among casual users, that the hallucination problem has been fixed,” says Srikanth Jagabathula, a professor of technology, operations, and statistics at NYU. “But that's definitely not the case, because they're fundamentally statistical machines. They don't have a notion of a ground truth, or what is true.” Even if an answer seems correct at first, one easy way to stress test the output is simply to ask a chatbot to double-check everything it just said. While this approach won’t confirm whether the output is correct, this method has highlighted plenty of issues in AI responses and leaves me feeling increasingly skeptical about turning to bots for advice on any topic, beyond just money. Yes-Bot May Affirm Preexisting Beliefs When you turn to a human financial advisor for money tips, they will likely be cordial and professional and push back on any preconceptions you may have about saving, investing, and spending money. On the other hand, chatbots are known for being overly agreeable, often taking the user’s side. “AI sycophancy is not merely a stylistic issue or a niche risk, but a prevalent behavior with broad downstream consequences,” reads part of a study about AI’s conversational flattery published earlier this year in the journal Science. “Although affirmation may feel supportive, sycophancy can undermine users’ capacity for self-correction and responsible decision-making.” The study looked at how AI will take a user’s side during interpersonal conflicts, but concerns about sycophancy are relevant to financial questions as…