AI gives small business owners confident marketing advice that is frequently wrong — and a new Harvard Business School study confirms it. Researchers found that when people relied on AI predictions without knowing whether the data was reliable, their decisions were up to 176% worse.
The fix? Simple alerts that told users when to trust the AI and when to question it reduced errors by 49%.
I had a client last month who asked ChatGPT to rewrite her email welcome sequence. The AI delivered polished, professional copy in about 90 seconds.
She sent it to her list of 400 subscribers — mostly local chiropractors she’d met at BNI meetings. Open rates dropped from 34% to 11%.
The AI wrote copy that sounded like a SaaS company talking to enterprise buyers. It had zero context about her audience, her voice, or the relationship she’d already built with these people.
The AI was confident. The AI was wrong. And the Harvard study explains exactly why.
What Did the Harvard Study on AI Decision-Making Find?
The study, led by Professor Kris Johnson Ferreira and published in Manufacturing & Service Operations Management, tested how people performed when making predictions using AI-generated forecasts.
When AI worked with familiar data — patterns it had seen before — its recommendations were solid. When the data was unfamiliar or unusual, the AI still sounded confident.
It didn’t say “I’m not sure about this.” It delivered its answer with the same authority either way.
The worst results came when people dealt with both familiar and unfamiliar data at the same time. They overcorrected for the patterns they recognized and under-corrected for the ones they didn’t.
Their predictions were 176% worse than people who only dealt with unfamiliar data. And 143% worse than those working with familiar data alone.
Think about what this means for your marketing.
Every time you ask AI to write an email, suggest a social media strategy, or analyze your website traffic — the AI responds with equal confidence whether it has good data or garbage data. It doesn’t flag its own blind spots.
Where AI Marketing Mistakes Hit Small Businesses Hardest
The Harvard study focused on demand forecasting, but the principle applies to every AI marketing task you’re doing right now. Here’s where I see small business owners getting burned most often.
Email marketing copy. AI writes emails based on patterns from millions of emails. Your list of 200 contractors in Phoenix is not “millions of emails.”
The AI doesn’t know your subscribers read emails on their phones between job sites at 6:30 AM. It doesn’t know they respond to direct, short messages — not the three-paragraph newsletters it keeps suggesting.
This is exactly why treating AI like a new employee with boundaries matters so much.
Social media strategy. Ask AI what to post on LinkedIn and it’ll give you a content calendar full of “thought leadership” posts with engagement hooks.
For a financial planner with 340 connections who gets most clients through referral dinners? That advice is disconnected from reality. The AI doesn’t know your sales process.
Ad spend recommendations. AI tools will tell you to allocate budget based on platform benchmarks. But platform benchmarks come from companies spending $10,000 a month or more.
If your marketing budget is tight — and for most of you, it is — those benchmarks will lead you straight into wasted money.
SEO content suggestions. AI will generate keyword-optimized content all day long.
But if your business serves a hyper-local market and the AI is pulling national search data, you’re optimizing for people who will never walk through your door.
🛑 DON’T COPY BLINDLY
The 2024 Keap survey found that 29% of small business owners don’t know if their marketing is working, and 30% don’t have a marketing plan. Adding AI on top of that uncertainty doesn’t create clarity — it creates faster, more confident confusion.
How to Build Your Own “Warning System” for AI Marketing
The Harvard researchers proved that warnings and endorsements — simple signals about when to trust AI and when to question it — reduced errors by 49%.
Most AI tools don’t include these signals. So you have to build your own.
Here’s a framework I use with clients. Before you accept any AI marketing recommendation, run it through three filters.
Filter 1: Does the AI have YOUR data?
If you asked ChatGPT to improve your email subject lines but didn’t give it your past open rates, your audience description, or examples of what’s worked before — the AI is guessing.
Guessing with good grammar isn’t strategy. Give it your specific context or treat the output as a rough draft, not a final answer.
Filter 2: Is the recommendation based on YOUR business size?
Most AI training data comes from mid-market and enterprise companies. When it suggests “A/B test your landing page with 1,000 visitors,” and you get 200 visitors a month, that advice doesn’t apply to you.
Watch for recommendations that assume resources you don’t have — bigger lists, bigger budgets, bigger teams.
Filter 3: Does it match what you know about your customers?
You’ve had hundreds of conversations with your customers. You know what makes them buy, what makes them hesitate, and what language they respond to.
If the AI suggests something that contradicts your direct experience — trust your experience. The Harvard study found that people who trusted their own knowledge in unfamiliar-data situations performed dramatically better.
💡 STRATEGY ALERT
The researchers called it “naive adjusting behavior” — when people accept AI’s answer and only make small tweaks instead of questioning the whole premise. It’s the marketing equivalent of taking AI-generated copy and changing two words instead of asking “Should I be sending this email at all?”
The “Endorsement” Side of AI Marketing
The study wasn’t all bad news. When AI worked with data it had been trained on — familiar patterns, representative information — its predictions were strong.
For small business marketing, this means AI is worth trusting when the task has clear patterns and your data is solid.
Writing variations of a proven email subject line. Reformatting content you’ve already written for a different platform. Summarizing customer feedback you’ve already collected. Cleaning up grammar and structure in copy you drafted yourself.
The common thread: you’re giving AI good inputs and asking it to do pattern-based work. That’s where AI earns its keep.
Understanding the difference between AI agents and basic automation helps you assign the right tasks to the right tools.
Where it falls apart is when you ask AI to make judgment calls about your specific market, your specific customers, or your specific business situation.
That’s your job. The core principle of DIY marketing — owning your process and decisions — has never been more relevant.
A Simple Diagnostic Table for AI Marketing Decisions
| If You See This… | It Means… | Your Next Move |
|---|---|---|
| AI gives you a content calendar without asking about your audience | It’s working from generic data, not your data | Feed it your customer persona and past performance data first |
| AI recommends a marketing channel you’ve never tested | It’s pulling from industry averages, not your results | Test with a small budget before committing — or skip it |
| AI-generated copy sounds polished but generic | It doesn’t know your voice or your customer relationships | Use it as a first draft, then rewrite in your voice |
| AI suggests “scaling” a campaign getting 12 clicks a month | It doesn’t understand your traffic volume | Fix conversion on what you have before spending more |
| AI confidently recommends a pricing strategy | HIGH RISK — pricing depends on context AI doesn’t have | Never let AI set your prices without your direct review |
What This Means for Your Marketing Process
The Harvard study confirmed something I’ve been saying for years: process comes before tools.
It doesn’t matter how sophisticated your AI tool is if you don’t have a process for evaluating its recommendations.
The researchers recommended designing “human-AI collaboration” with built-in checkpoints. For a solopreneur, that means creating a simple habit.
Every time AI gives you a marketing recommendation, pause and ask — does this match what I know about my customers and my business?
That two-second pause is your warning system. And according to Harvard, it’s worth a 49% reduction in mistakes.
The U.S. Chamber of Commerce reports that 40% of small businesses have experimented with AI tools. The 2024 Keap survey showed 48% say lack of time is the top reason they don’t market more.
AI promises to solve the time problem. But faster decisions aren’t better decisions unless you have a way to check the work.
⚠️ REALITY CHECK
The study’s co-author Matthew DosSantos DiSorbo put it this way: training data that becomes “outdated, corrupted, or flawed” breaks AI predictions. For small businesses, your market shifts constantly — new competitors, seasonal changes, economic pressure. AI trained on last year’s patterns doesn’t know about this year’s reality. Your direct customer knowledge does.
FAQ
What are the most common AI marketing mistakes small businesses make?
The biggest mistake is accepting AI recommendations without questioning whether the AI had relevant data.
Small business owners often let AI write copy, suggest strategies, or allocate budgets without feeding it specific information about their audience, business size, or past performance.
The Harvard study showed this leads to decisions that are up to 176% worse than decisions made with proper context. The second most common mistake is treating AI output as final instead of as a first draft that needs your expertise layered on top.
How do I know when to trust AI marketing advice?
Trust AI when you’ve given it your own data and the task involves recognizable patterns — like rewriting existing copy, reformatting content, or analyzing data you’ve already collected.
Question AI when it makes recommendations about your specific market, pricing, or strategy without access to your customer information.
The Harvard study found that “endorsements” (signals the data is familiar) and “warnings” (signals the data is unfamiliar) cut errors by 49%. If your AI tool doesn’t give you those signals, build your own by asking: did I give it the data it needs to get this right?
Does AI replace the need for a marketing strategy?
No. AI amplifies whatever marketing process you already have — good or bad.
If you don’t have a defined marketing strategy, AI will generate more content, more ideas, and more recommendations faster. But none of them will be connected to a clear business goal.
Process must come before tools. Get clear on who your customer is, what you’re selling them, and how you’re reaching them. Then use AI to speed up execution of that plan — not to replace the plan itself.
What did the Harvard Business School study on AI decision-making prove?
The study by Kris Johnson Ferreira and colleagues proved that users make significantly worse decisions when AI handles a mix of familiar and unfamiliar data without any signals about data quality.
Participants who received no alerts about data reliability made predictions that were 176% worse than those who worked with only unfamiliar data.
Adding simple warnings and endorsements — alerts about when to trust or question the AI — reduced user errors by 49%. The takeaway for small business owners: you need a system for checking AI’s work, not more AI tools.
How should solopreneurs use AI without making expensive mistakes?
Start by using AI for tasks where you provide the inputs and check the outputs — drafting, reformatting, summarizing. These are low-risk tasks where AI performs well.
Never accept AI recommendations on pricing, audience targeting, or strategy without running them through your own knowledge of your customers. Those are high-stakes decisions that depend on context AI doesn’t have.
Build a simple three-filter habit: Does AI have my data? Is this sized for my business? Does it match what I know about my customers? If the answer to any of those is no, override the recommendation.
Keep Reading
🔧 Stop Guessing. Get a Fix-It Session.
Not sure if your AI-generated marketing is helping or hurting? In a Fix-It Session, I’ll review your marketing in 24 hours and tell you exactly what’s working, what’s broken, and what to do next. No meetings. No fluff. A video walkthrough, a 1-page action plan, and clarity you didn’t have yesterday.
