Lenny's Podcast · Jun 19AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff
Its first event gathered 600,000 attack techniques, a dataset every major AI lab now uses, and won Best Theme Paper at EMNLP out of about 20,000 submissions. Crowdsourced players with prizes beat hourly red teams, and its 2023 run showed prompt-based defenses fail.
Tips
- Do not rely on prompt instructions to block prompt injection▶ 1:10:20These defenses were tested in the HackAPrompt 1.0 challenge in May 2023 and did not work then or now.
- Run crowdsourced red teaming with incentives to find attacks▶ 58:20The paper won Best Theme Paper at EMNLP, out of about 20,000 submissions.
