Once Upon a Time in Cyber…

Twenty years ago now, I was fortunate enough to work for the InfoWorld Test Center. I would receive all the cool new cybersecurity tools and I would build and execute a test plan for the participating technologies. I would provide the objective findings for Victor R. Garza (Bob) to write into a review. We would present the findings with the vendors, do a fact check and give them a chance to present clarity. Bob and I would balance all the subjective feedback the vendors told us about themselves (and their competitors) against what the objective test findings. Ultimately we were solely concerned on presenting the InfoWorld subscriber with accurate, actionable information.

“Night at the Roxbury”

In the years since, InfoWorld has stopped performing product testing, NSS Labs folded and the analysts (Gartner, Forrester, etc.) have become echo chambers for the messaging the vendors create. As both a vendor and a customer of cybersecurity and other IT products, I often feel like I’m being assaulted by relentless vendor advances ala Chris Kattan and Will Ferrell in “A Night at the Roxbury.”

Using AI to Research Vendors

Part of my role at WitFoo is making sure my team and partners understand the similarities and differences between our products and our competitors. Twice a year, I block a week and read hundreds of marketing, press and analyst reports so I can synthesize a new competitive analysis. This time around the horn, I used Bing Chat (powered by ChatGPT) to do the first pass for me. I cut my work from days to a few hours. I realized I could use this to help me as a purchaser as well. That’s what I’ll share here.

Configuring Bing Chat for Product Research

I do my research on Bing Chat via Edge browser but you can interface with ChatGPT with native apps or use Bard via Google. I’ll limit this article to Bing Chat. Here is what you need to do to set up:

• Install the Edge browser. It’s native on Windows machines and available in Apple and Android stores. If you had a bad experience with Edge in the past, as I did, it’s important to note that in 2020, Microsoft released an overhauled Edge built on Chromium.
• Click on the Bing Chat icon.
• Select “More Creative.” I know we want to use “More Precise” but lacking objective product testing, we need to give a green light to the chat bot to “read between the lines” of the data it has available. Many (if not most) vendors in cybersecurity see obscurity as leverage in how they transact business. The chat bot will have to use some guile in processing the subterfuge.

Asking the Questions

The only thing left to do is ask the questions. Let’s use “endpoint detection and response” as an example in our exercise.

1) Let’s start with a definition to ensure we’re on the same page. “What is EDR?”

2) What are the major EDR solutions?

3) How are the solutions priced?

4) Which solutions do customers prefer?

More Questions

A you continue your chat, you can try some of these questions, as applicable:

• Which solutions are cloud native?
• Are any of these solutions available through the AWS Marketplace?
• Which solutions are compatible with Redhat Linux?
• Does {my partner} resell any of these solutions?
• Which of these solutions is supported by Splunk or WitFoo?
• Which solutions are FEDRAMP certified?
• Compare productA and productB

Trust but Verify

At the bottom of each response, Chat provides references. Before you submit the budget or acquisition report, be sure to ask for verification from your reseller, integration partner and the vendor. The bot bases its responses on publicly available information that is often a product of dubious marketing campaigns.

The information you collect in these queries can reduce the effort in disqualifying tools and prepare you to ask the hard question when the sales teams show up with sandwiches.

Love the One We’re With

Until journalistic reviews return to cybersecurity, Bing Chat and other LLM powered chat bots may be our best tool in sorting through the multi-billion dollar marketing machines that spew out specs of useful information wrapped in fluff an F.U.D.