You might be surprised to learn that I don’t believe red teaming always works. You might be even more surprised that I believe red teaming can sometimes do more harm than good. Here are seven red flags that might indicate that you need to review and perhaps reconsider how your red team goes about its business.
- The red team or the red team’s client ignores the presence and effects of uncertainty. Red teams can help reduce uncertainty, but they can never eliminate it entirely. The superior red team always identifies and discusses residual sources of uncertainty (example below). If you’re hiring a red team, ask them how they handle and address issues of uncertainty. You might be surprised at the “ums” and “ahs” you get back!
- The red team or the red team’s client plays fast and loose with the red team’s scope or rules of engagement. This is something you want to watch very carefully. It involves both oversight of the red team and discretion regarding the red team’s findings. Prudence requires the red team to agree to and abide by explicit rules of engagement. When establishing these rules, management should carefully consider the potential effect on ongoing operations as well as any relevant legal risks the red team’s activities might raise. Further, the person responsible for overseeing the red team should monitor the read team’s activities and performance throughout the engagement. Issues of scope can also emerge when clients, wanting to trim costs by limiting the scale of the effort, then want to apply the red team’s limited findings to the department or enterprise as a whole. This can skew risk analysis and lead to poor follow-on decisions.
- The red team knows much more or much less than the real-world adversary knows. If the red team knows much more, its findings are likely to skew your risk perspective. For example, you might end up spending a lot of money patching holes that your likely real-world adversaries would never have discovered. If the red team knows much less, it’s unlikely to uncover the vulnerabilities your real-world adversary might very well exploit. Either way, you’re asking to be surprised. Since you will never be able to match your red team’s capabilities and knowledge perfectly with the capabilities and knowledge of your real-world adversaries and competitors, mark this as a consistent source of residual uncertainty.
- The red team confuses spectacle with risk. Inferior red teams often go straight for the “wow” attacks, the ones security expert Bruce Schneier calls “movie-plot threats.” In most cases, you’re asking your red team to identify sources of risk, and the highly surprising, over-the-top attacks are likely to be high-consequence but low-likelihood. This isn’t to say that “movie-plot threats” are without risk, it just means that you need to put them in the context of overall, relative risk.
- The red team overlooks the value of time and persistence. This one can be tricky. Most red team engagements explore and weigh short-term possibilities. Often, however, the most dangerous adversaries are those who pursue long-term strategies. This mismatch can lead the red team to undervalue an adversary’s will and persistence. Clients and red teams should discuss this possibility when designing a red team engagement.
- The red team doesn’t know how to practice discretion. This issue is probably obvious, but no one wants a red team boasting at the water cooler about what it just uncovered. Red teamers sometimes like to tell stories about their exploits, and not every red teamer’s ego is bound by his or her wisdom. You can often get a sense of this when interviewing a red team or red teamer. Ask them about their experience and listen to what they are willing to share. Do they share too much detail about their previous engagements? If so, they are likely to share to much about their engagement with you.
- The red team is insufficiently trained or staffed. This one is also probably obvious, but I need to mention it nonetheless, in large part because an insufficiently trained or staffed red team can lead to all the other issues mentioned here. Sometimes a little red teaming is more dangerous than no red teaming at all.
Red teaming is hard work, and the work doesn’t begin when you set your red team loose; it begins the moment you consider hiring or running a red team. In fact, much of the most important work you will do must be done prior to the actual “red teaming.” In a very real sense this up-front work involves red teaming the red team—or, in other words, anticipating what could go wrong and accounting for it in advance. And while the list above is not necessarily complete, it can help you avoid some of the more common pitfalls.