Phish Setlist for Security: Build Your Attack Plan

Why Every Security Team Needs a Phish Setlist

In March 2022, Okta confirmed that the Lapsus$ threat actor group breached a third-party support engineer's account — and a big part of that attack chain started with social engineering. A single compromised credential. One phishing message that worked. That's all it took.

If you run security awareness programs, you already know that sending one random phishing simulation per quarter isn't cutting it. What you need is a phish setlist — a deliberate, structured rotation of phishing simulation scenarios designed to test every attack vector your employees will actually face.

Think of it like a musician's setlist. Every song is chosen for a reason: pacing, audience, energy. A phish setlist works the same way. Each simulated attack is sequenced to build difficulty, cover diverse tactics, and expose the specific gaps in your organization's human defenses. I've built these for organizations ranging from 50-person startups to enterprises with thousands of endpoints, and the difference between random simulations and a planned setlist is the difference between noise and actionable intelligence.

What Exactly Is a Phish Setlist?

A phish setlist is a pre-planned sequence of phishing simulation scenarios used in security awareness training programs. Each entry on the list specifies the attack type (credential theft, malware delivery, business email compromise), the pretext (fake invoice, IT password reset, CEO wire transfer request), difficulty level, and target employee group. Organizations use phish setlists to systematically test and train employees against realistic social engineering attacks over weeks or months.

Without a setlist, most phishing simulation programs fall into one of two traps. Either they repeat the same basic "click this link to reset your password" scenario until everyone recognizes it, or they jump straight to advanced spear-phishing that frustrates employees without teaching them anything. A structured phish setlist avoids both problems.

The $4.88M Reason You Can't Wing It

According to IBM's 2022 Cost of a Data Breach Report, the average cost of a data breach hit $4.35 million globally — and breaches that started with phishing were among the most expensive. The Verizon 2022 Data Breach Investigations Report found that 82% of breaches involved a human element, including social engineering, errors, and misuse.

Those aren't abstract numbers. They represent real organizations that thought their training was "good enough." In my experience, the ones that get breached almost always lack a systematic approach to phishing simulation. They test once, check a compliance box, and move on. A proper phish setlist turns a checkbox exercise into a measurable security control.

The FBI's Internet Crime Complaint Center (IC3) reported over $2.4 billion in losses from business email compromise (BEC) in 2021 alone. BEC is a phishing subtype that most basic simulations never even test for. If it's not on your setlist, your people aren't ready for it.

How to Build a Phish Setlist That Actually Works

Step 1: Inventory Your Threat Landscape

Before you write a single simulation, look at what's actually hitting your mail gateway. Pull 90 days of quarantined phishing attempts. Categorize them: credential theft, malware attachments, BEC/impersonation, smishing (SMS-based phishing), and pretexting. Your phish setlist should mirror your real threat profile, not some generic template.

If 60% of real attacks targeting your org are credential harvesting pages disguised as Microsoft 365 login portals, then 60% of your early simulations should test that exact scenario. Match reality.

Step 2: Define Difficulty Tiers

I use a three-tier system:

Tier 1 — Obvious: Generic greeting, mismatched sender domain, spelling errors, urgent language. These establish a baseline. If employees fail these, you have a foundational problem.
Tier 2 — Moderate: Correct branding, plausible sender name (but spoofed domain), contextually relevant pretext like a benefits enrollment or IT maintenance window. These test whether employees look beyond surface-level cues.
Tier 3 — Advanced: Spear-phishing with employee-specific details scraped from LinkedIn, spoofed internal domains, thread hijacking pretexts, or multi-channel attacks (email + phone call follow-up). These test your most security-aware staff and your incident reporting process.

Your phish setlist should progress through these tiers over time. Start most employees at Tier 1. Move them up as they demonstrate competence. Repeat offenders stay at lower tiers with additional training — which is exactly the kind of structured program you can build using phishing awareness training designed for organizations.

Step 3: Vary the Attack Vectors

A setlist that only includes email-based credential phishing is incomplete. Real threat actors use multiple channels. Your setlist should include:

Credential harvesting: Fake login pages for Microsoft 365, Google Workspace, VPN portals, or internal apps.
Malware delivery: Simulated malicious attachments (Word docs with "macros," PDFs, ZIP files).
BEC / CEO fraud: Impersonation of executives requesting urgent wire transfers or sensitive data.
Vishing pretexts: Phone-based social engineering paired with email lures.
Consent phishing: OAuth app permission requests — an increasingly common vector in 2022.

Each vector tests a different defensive reflex. Employees who spot a fake login page might still authorize a malicious OAuth app because they've never been trained on that scenario.

Step 4: Schedule for Cadence, Not Surprise

Monthly simulations are the minimum. I've seen the best results with bi-weekly sends, staggered across departments so no two groups get the same simulation on the same day. This prevents the "Hey, did you get that weird email?" effect that contaminates your data.

Map your phish setlist to a 6-month calendar. Twelve to twenty-four unique scenarios, progressing in difficulty, covering all major vectors. Review results monthly. Adjust the setlist based on click rates, report rates, and which pretexts are fooling the most people.

Step 5: Pair Every Simulation with Training

A simulation without follow-up training is just a gotcha. Every failed simulation should immediately redirect the employee to a brief, specific training module that explains what they missed and why it mattered. This is where programs like cybersecurity awareness training courses become essential — they provide the structured learning that turns a mistake into a teachable moment.

The best phish setlists build in "training recovery" time. After a particularly difficult Tier 3 simulation, the next send should be a Tier 1 scenario. This lets employees apply what they just learned and rebuild confidence. Demoralized employees stop reporting suspicious emails entirely — and that's worse than clicking.

Metrics That Matter: Measuring Your Phish Setlist's Impact

Click rate gets all the attention, but it's the least useful metric in isolation. Here's what I track:

Report rate: What percentage of employees forwarded the simulation to your security team? This is the metric that actually saves you during a real attack. CISA's guidance on Shields Up emphasizes that rapid reporting is a critical defense layer.
Time to report: How quickly did the first report come in? Under 5 minutes is excellent. Over 30 minutes means your reporting process has friction.
Repeat offender rate: What percentage of employees have failed two or more simulations in a row? These individuals need targeted intervention, not just another simulation.
Tier progression: Are employees advancing from Tier 1 to Tier 2 scenarios over time? If click rates on Tier 1 aren't dropping after three months, your training content needs work.
Department-level trends: Finance, HR, and executive assistants face disproportionate BEC risk. Track their performance separately.

Common Phish Setlist Mistakes I See Constantly

Sending the Same Scenario to Everyone at Once

When 500 people get the same phishing email at 9:00 AM on a Tuesday, Slack lights up within minutes. Your click rate data becomes meaningless because peer-to-peer warnings — not training — drove the results. Stagger your sends across time windows and vary the pretext slightly per group.

Never Testing Executive Leadership

Executives are the number one target for spear-phishing and BEC, yet many organizations exclude the C-suite from simulations because "they're too busy" or "they'll complain." In my experience, executives fail Tier 2 simulations at higher rates than average employees because they process email quickly and rely on assistants to filter. Your phish setlist must include them.

Ignoring Multi-Factor Authentication Context

If your organization has deployed multi-factor authentication (MFA), your simulations need to reflect that reality. Test whether employees enter credentials on fake login pages even when MFA should make them pause. Adversary-in-the-middle (AiTM) phishing kits that steal session tokens post-MFA were a major threat in 2022. Your setlist should include scenarios that teach employees MFA isn't a silver bullet.

Treating the Setlist as Static

Threat actors evolve constantly. Your phish setlist should be a living document, updated quarterly based on emerging tactics. When a new pretext trends — like the fake "Microsoft Teams" notifications that surged in 2022 — add it to the rotation within weeks, not months.

Connecting Your Phish Setlist to a Zero Trust Strategy

A phish setlist doesn't exist in a vacuum. It should feed directly into your broader zero trust architecture. Every simulation result tells you something about where your human perimeter is weakest. Use that data to:

Tighten conditional access policies for departments with high click rates.
Prioritize phishing-resistant MFA (FIDO2 keys) for repeat offenders and high-risk roles.
Adjust email gateway rules based on which pretexts bypass your technical controls.
Inform your incident response playbooks with real data on employee reporting behavior.

The NIST Cybersecurity Framework emphasizes that "Protect" and "Detect" functions depend on trained personnel. Your phish setlist is how you operationalize that requirement with measurable outcomes.

A Sample 6-Month Phish Setlist

Here's a simplified setlist template I've used successfully:

Month 1: Tier 1 credential harvest (generic Microsoft 365 reset) + Tier 1 malicious attachment (fake invoice PDF).
Month 2: Tier 1 BEC (CEO requests gift cards) + Tier 2 credential harvest (branded IT helpdesk with correct logo).
Month 3: Tier 2 malware delivery (fake DocuSign with macro-enabled doc) + Tier 2 consent phishing (OAuth app permission).
Month 4: Tier 2 BEC targeting finance (vendor payment redirect) + Tier 1 recovery scenario (obvious phish to rebuild confidence).
Month 5: Tier 3 spear-phish (LinkedIn-sourced details, spoofed internal domain) + Tier 2 smishing via SMS.
Month 6: Tier 3 thread hijack pretext + Tier 3 multi-channel (email lure + vishing follow-up call).

Adjust based on your data. If Month 2 shows a 40% click rate on the branded helpdesk scenario, insert additional Tier 2 credential harvesting scenarios before advancing to Tier 3.

Start Building Your Phish Setlist Today

The organizations that survive real phishing attacks in 2022 aren't the ones with the fanciest email gateways. They're the ones whose employees have been tested against a realistic, structured rotation of social engineering scenarios — and trained on what to do when something looks wrong.

Your phish setlist is the operational backbone of that program. Build it deliberately. Measure it rigorously. Update it constantly. And pair it with comprehensive training — both foundational cybersecurity awareness training and targeted phishing awareness training for your teams — to turn every simulated attack into lasting behavioral change.

Because the next real phishing email hitting your inbox won't follow a script. But if your employees have trained against a well-built setlist, they won't need one either.