Phish Setlist for Security: Building Your Attack Playlist

Your Organization Needs a Phish Setlist — Not Just One Test

In 2023, the FBI's IC3 received over 298,000 phishing complaints — making it the most reported cybercrime category for the fifth consecutive year. Yet most organizations I work with still run the same single phishing simulation once a quarter and call it done. That's like a band playing one song at every concert and wondering why the audience stops paying attention.

A phish setlist is a concept I've been teaching for years: a curated, rotating playlist of phishing simulation scenarios that you deploy against your own employees over time. Just like a band rotates songs to keep the crowd engaged, you rotate attack types to keep your workforce sharp against evolving threat actors. If you only test with one template, your people learn to spot that template — not actual phishing attacks.

This post breaks down exactly how to build a phish setlist that covers real-world attack vectors, keeps employees on their toes, and measurably reduces your organization's risk of a data breach. Whether you're a security team of one or running awareness programs for thousands, this framework works.

What Is a Phish Setlist, Exactly?

A phish setlist is a planned rotation of phishing simulation scenarios — different lure types, pretexts, sender spoofs, and payloads — organized into a sequence you deploy over weeks or months. Think of it as your testing repertoire.

Each "song" in your setlist is a distinct phishing scenario. One might mimic a credential theft attempt disguised as a Microsoft 365 password reset. Another could be a spear-phishing email from a spoofed CEO requesting a wire transfer. A third might drop a simulated ransomware payload through a fake invoice attachment.

The goal isn't to trick employees for sport. It's to expose them to the full spectrum of social engineering tactics so they build real pattern recognition — not just familiarity with one type of lure.

Why a Single Simulation Fails

I've audited security awareness programs where the same "Your package couldn't be delivered" template ran four quarters in a row. Click rates dropped to near zero — not because employees got smarter, but because they memorized that specific email. When a novel Business Email Compromise hit the same company, the CFO's assistant wired $87,000 to a threat actor without blinking.

One scenario teaches employees to recognize one scenario. A phish setlist teaches them to think critically about every unexpected email.

The 8 Tracks Every Phish Setlist Needs

Based on the Verizon Data Breach Investigations Report, which consistently identifies phishing and pretexting as top attack vectors, here are the eight scenario categories your setlist should cover.

1. Credential Harvesting

The classic. A fake login page for Microsoft 365, Google Workspace, or your company VPN. According to the Verizon DBIR, stolen credentials are involved in roughly 50% of breaches. This track should appear in every setlist rotation, but vary the brand and pretext each time.

2. Business Email Compromise (BEC)

Spoofed emails from the CEO, CFO, or a vendor requesting urgent wire transfers, gift card purchases, or sensitive data. The FBI IC3's 2023 report showed BEC accounted for over $2.9 billion in adjusted losses. Test your finance and HR teams with these specifically.

3. Malicious Attachment

Fake invoices, shipping notices, or HR documents with simulated malicious payloads — Word docs with macros, PDFs with embedded links, or ZIP files. This tests whether employees open unexpected attachments without verification.

4. Smishing (SMS Phishing)

If your organization issues mobile devices, you need SMS-based scenarios. Package delivery alerts, MFA verification codes, or fake IT helpdesk texts. Smishing is surging because people trust their phones more than their inboxes.

5. Vishing Pretexts

Not every phish setlist entry has to be email. Include voice-based social engineering scenarios where someone calls claiming to be IT support and asks for credentials or remote access. Even a scripted tabletop exercise around vishing adds value.

6. QR Code Phishing (Quishing)

A newer addition that's earned its spot. Threat actors embed malicious QR codes in emails or even physical printouts posted in offices. Your setlist should include at least one QR-based lure — a fake parking validation, Wi-Fi login, or benefits enrollment page.

7. Spear Phishing with OSINT

These are highly targeted scenarios using real details scraped from LinkedIn, company websites, or social media. "Hey [Name], great presentation at [Conference] last week — here are the slides you asked for." This tests whether employees question emails that feel personal and relevant.

8. Vendor/Supply Chain Impersonation

Emails appearing to come from real vendors your company uses — DocuSign, FedEx, your payroll provider, your cloud hosting company. This is how many ransomware campaigns begin, and it's a blind spot for organizations that only test with internal impersonation.

How to Structure Your Phish Setlist Rotation

Don't fire all eight categories in a single month. Here's the rotation framework I recommend for most organizations.

Monthly Cadence, Quarterly Themes

Run one simulation per month. Group three months into a themed quarter:

• Q1: Credential and Access Threats — Credential harvesting, MFA bypass lure, VPN phishing
• Q2: Financial and BEC Attacks — Wire transfer request, invoice fraud, gift card scam
• Q3: Payload Delivery — Malicious attachment, ransomware dropper, QR code phish
• Q4: Advanced Social Engineering — Spear phishing with OSINT, vendor impersonation, vishing scenario

This gives you 12 distinct scenarios per year. No repeats. Progressive difficulty. Each quarter builds a different muscle.

Vary the Difficulty

Start each quarter with a moderate-difficulty lure, then escalate. Your January credential harvesting email might have obvious red flags — a misspelled domain, generic greeting. By March, the lure should use a convincing lookalike domain and reference a real internal system. This approach trains progressive pattern recognition rather than just testing pass/fail.

Measuring What Your Phish Setlist Reveals

Running simulations without measuring outcomes is like performing a concert with no audience feedback. Track these metrics for every setlist entry:

• Click rate: Percentage of recipients who clicked the malicious link or opened the attachment
• Credential submission rate: Percentage who entered actual credentials on the fake page
• Report rate: Percentage who reported the email through your phishing report button
• Time to report: How quickly the first report came in after deployment

The report rate matters more than the click rate. You want a culture where employees flag suspicious emails fast — that's your human detection layer. If your report rate stays below 20%, your cybersecurity awareness training program needs reinforcement.

Segment by Department and Role

Finance teams should face more BEC scenarios. Developers should see more supply chain and code repository phishing. Executives need spear phishing with OSINT. A phish setlist that treats every employee identically misses the point. Threat actors don't send the same email to your intern and your CFO.

What Happens After Someone Clicks

This is where most programs fall apart. Someone fails a simulation, gets a pop-up saying "this was a test," and nothing changes. Here's what actually works.

Immediate, Contextual Training

The moment someone clicks, serve them a 60-second training module that explains exactly what red flags they missed — in that specific email. Generic "don't click suspicious links" training is useless. Contextual, scenario-specific feedback changes behavior.

Repeat Offender Escalation

Employees who fail three or more simulations within a year need additional intervention. Not punishment — structured training. Enroll them in focused phishing awareness training for organizations that walks through real-world attack patterns with interactive exercises. I've seen repeat-click rates drop by over 60% when organizations pair simulations with targeted follow-up training.

No Public Shaming

I've watched organizations post "wall of shame" leaderboards showing who clicked. This kills your reporting culture overnight. Nobody will voluntarily report a suspicious email if they fear being ridiculed. Protect the data. Use it for coaching, not punishment.

Does a Phish Setlist Actually Reduce Breach Risk?

Yes — when combined with technical controls. A phish setlist is your human-layer defense. It works alongside multi-factor authentication, email filtering, endpoint detection, and zero trust architecture. None of these alone is sufficient.

CISA's Shields Up guidance explicitly recommends phishing simulations as part of organizational readiness. The NIST Cybersecurity Framework includes awareness and training as a core protective measure under the Protect function.

Organizations that run regular, varied phishing simulations see measurable improvements. The key word is varied. A single repeated test builds complacency. A structured phish setlist builds resilience.

Building Your First Phish Setlist: A Quick-Start Checklist

If you're starting from scratch, here's your action plan:

• Audit your threat landscape: What phishing attacks has your industry seen in the past 12 months? Start there.
• Pick your first four scenarios: One from each category — credential theft, BEC, malicious attachment, and vendor impersonation.
• Set your cadence: Monthly is ideal. Bi-monthly is the minimum.
• Configure your reporting mechanism: Deploy a phishing report button in your email client. Make it one click.
• Baseline your metrics: Run your first simulation without prior announcement. That's your honest baseline.
• Layer in training: Every simulation should connect to educational content. No test without a teaching moment.
• Review and rotate quarterly: Retire scenarios that everyone passes. Introduce new lures that reflect current threat intelligence.
• Brief leadership: Share aggregate data with executives monthly. Click rates, report rates, trends. Keep security awareness visible at the top.

The Setlist Never Ends

Threat actors don't stop innovating, and your phish setlist shouldn't either. Every major breach, every new social engineering tactic, every emerging AI-generated lure is a new track to add to your rotation.

I update my recommended setlists every quarter based on what I'm seeing in the wild. Right now, AI-generated voice cloning and deepfake video calls are entering the social engineering toolkit. Your 2026 phish setlist should include at least one scenario that tests employee response to synthetic media or AI-crafted messages that lack traditional red flags like typos and awkward phrasing.

The organizations that treat phishing simulation as an ongoing, evolving program — not an annual checkbox — are the ones that catch real attacks before they become real breaches. Build your phish setlist. Rotate it relentlessly. And make sure every employee in your organization has faced enough variety that the next real phishing email triggers suspicion, not a click.