Passive-aggressive people often use subtle phrases that seem harmless but are designed to chip away at your confidence—here’s how to spot them and take your power back.
We’ve all felt it—that little wobble in your stomach after someone says something that sounds fine on the surface but somehow leaves you second-guessing yourself.
Psychology has a word for this pattern.
The APA describes passive-aggressive behavior as conduct that looks neutral but indirectly expresses an aggressive motive.
In plain English: it’s hostility with a friendly mask.
“Seemingly innocuous… but indirectly displays an aggressive motive,” as the APA puts it.
Let’s break down eight common phrases that can chip away at your confidence—and how to respond without losing your center.
1. “I’m just joking”
Sarcasm can be playful, but it can also be a delivery system for contempt.
In research on disparagement humor, psychologists Thomas Ford and Mark Ferguson show how jokes that “put down” a target can normalize disrespect and silence pushback because “it was just a joke.”
In other words, the laugh track becomes a shield for aggression.
If you’ve ever heard “Relax, it’s a joke” right after a jab at your intelligence or appearance, you’ve felt this.
A simple, steady reply works: “If it’s a joke, it still landed as a dig. Let’s keep it respectful.”
You’re naming the effect without debating their intent.
When I was freelancing in a busy newsroom, a colleague used to “joke” that my deadlines were “aspirational.”
I stopped laughing along and said, “I’m open to feedback—keep the jokes out of it.”
The jokes stopped.
The work got better.
2. “No offense, but…”
This is the linguistic equivalent of throwing a punch and handing you an ice pack at the same time.
Communication scholars have found that backhanded compliments don’t just sting the receiver; they also make the speaker look worse.
As one Harvard summary bluntly put it, “backhanded compliments usually undercut the speakers.”
Try this line: “If there’s feedback, give it to me straight—no disclaimer needed.”
You remove their shield and invite clarity.
3. “You’re too sensitive”
This one questions your reality.
Clinicians describe gaslighting as a specific form of emotional manipulation that makes you doubt your perceptions.
The Cleveland Clinic puts it clearly: true gaslighting is “a specific form of emotional abuse and mental manipulation.”
When someone says “You’re too sensitive” after they’ve crossed a line, it’s a small step in that direction—shifting the problem from the behavior to your feelings about it.
Respond with boundaries, not a debate about whether you “are” sensitive: “I’m allowed to feel what I feel. I’m asking that you don’t talk to me that way.”
4. “If you say so”
On paper, it’s agreement.
In tone, it’s a shrug that questions your competence.https://link.springer.com/rwe/10.1007/978-3-319-16999-6_193-1
Psychologically, this sits in the family of indirect aggression—behavior intended to harm someone’s social standing or self-esteem without open confrontation.
Think eye-rolls, sighs, and vague compliance that erodes trust.
Researchers define indirect aggression as harming others “through circuitous means,” which is exactly what this phrase does.
One way to neutralize it: “Yes, I do say so—and here’s why.”
Then state your reasoning or decision criteria.
You convert subtext into text, which passive-aggression hates.
5. “I thought you knew”
When a deadline is missed or a message goes unanswered, this sounds innocent.
But repeated “Oops—thought you knew” responses can be a form of withholding or strategic ambiguity that leaves you feeling incompetent.
The APA notes that a long-standing pattern of “forgetting” or “misplacing” things can be a passive way to express resistance and negativism.
If the pattern keeps you off balance, your confidence takes the hit.
I’ve mentioned this before but one of the best fixes is process clarity, not more emotion.
“Next time, let’s put it in writing: Who’s doing what by when.”
You’re trading fuzziness for agreements.
6. “Calm down”
Ever notice how no one calms down after hearing “Calm down”?
It’s a classic emotional minimizer.
The message isn’t “I care about your nervous system”; it’s “Your reaction is the problem.”
Over time, this can train you to question your cues—Was I really upset? Should I be quieter?—instead of staying connected to what matters.
Try flipping the script: “I’ll calm down when I feel heard. Here’s the point I’m making.”
That way, you regulate and re-center the issue at hand.
Quick travel lesson here.
Years ago on a crowded train in Osaka, a guy shoved my camera bag while insisting “Relax.”
A stranger next to me said, “Listen first.”
Five seconds of listening later, the shove turned into an apology and a seat swap.
Presence beats policing tone almost every time.
7. “I’m just trying to help”
Help is great.
Unsolicited help that sneaks in a hierarchy—“Here’s how you should’ve done it”—isn’t help.
It’s control in a friendly outfit.
In humor research, as mentioned above, this often pairs with aggressive humor—teasing framed as guidance—which correlates with poorer relationship outcomes because it enhances the self at another’s expense.
You can recognize it by the aftertaste: you feel smaller, not supported.
A clean reply: “If you’re offering help, I’ll ask when I need it. If you have feedback, give it directly.”
You honor both possibilities while taking back agency.
8. “Everyone thinks…”
This is a confidence eroder because it recruits imaginary witnesses.
Psychologically, it’s a blend of appeal to consensus and indirect aggression: instead of owning the critique, the person hides behind “people.”
Research on relational/indirect aggression shows the tactic often aims to damage someone’s standing in a group without confronting them outright.
Pin it down with specifics: “Name one person and the exact concern so we can address it.”
If they can’t—or won’t—the tactic falls apart.
Why these phrases work (and how to make them stop)
A useful theme runs through all eight: plausible deniability.
The speaker can claim innocence (“It was a joke,” “I meant well”) while you carry the confusion.
That confusion is the point.
It moves attention away from the behavior and onto your reaction.
Academic work on disparagement humor even shows how “it’s just a joke” reduces social costs for the aggressor by making the target look uptight if they object.
Here’s what helps:
-
Name the effect. You don’t have to prove intent. “That came across as a dig.” Or, “That makes me doubt myself, and I don’t want that dynamic.”
-
Ask for the direct version. “Say that plainly.” Most passive-aggressive phrases disintegrate in direct light.
-
Replace hedges with agreements. “Let’s write down who owns what.” Structure beats subtext.
-
Use short, calm scripts. The goal isn’t to win a debate; it’s to stop the erosion. Think: “Not helpful. Try again,” or “I’ll continue when this is respectful.”
Final thought
Confidence isn’t bravado.
It’s the quiet agreement you keep with yourself: I’ll trust my read and speak up when words are used to shrink me.
The next time someone slips one of these phrases into the conversation, don’t over-explain or over-apologize.
Name it, ask for clarity, and move forward.
Your calm, direct language is the antidote.
If You Were a Healing Herb, Which Would You Be?
Each herb holds a unique kind of magic — soothing, awakening, grounding, or clarifying.
This 9-question quiz reveals the healing plant that mirrors your energy right now and what it says about your natural rhythm.
✨ Instant results. Deeply insightful.
