What’s Behind Whistic’s Proprietary CrowdConfidence Scoring Algorithm?

January 31, 2018

Part of the magic of Whistic’s vendor assessment platform is the algorithm that works behind the scenes to interpret and evaluate third party risks. While most of our clients have heard us use the term “CrowdConfidence” to explain the score they see in the platform, few likely have visibility into what the score is comprised of or why the propriety process of developing the score is so valuable not just to Whistic, but to the industry as a whole.

In this article, we’ll share a behind-the-scenes look into Whistic directly from the source who has helped Whistic develop the CrowdConfidence Score from the ground up: a Quantitative Marketing Professor at Brigham Young University, and Data Science / Analytical Scoring Expert Advisor for Whistic, Jeff Dotson. Before Jeff began as a professor at BYU, he taught at Vanderbilt and prior to that, received his PhD from Ohio State University in Quantitative Marketing, which is the application of quantitative research techniques to the field of marketing, usually involving the construction of questionnaires and scales. Quantitative Marketers like Jeff use information obtained to understand needs of individuals in the marketplace, and to create strategies and marketing plans based off of findings. It’s this type of intense research application into the Whistic platform that makes it so powerful.

As Jeff has been involved with Whistic since its inception, there’s no one better person to help our customers understand the score’s importance than Jeff himself.

What Is The CrowdConfidence Score and What Does It Demonstrate?

In the simplest sense, the CrowdConfidence Score helps Whistic’s users quickly understand the specific areas in which a vendor requires additional attention and helps them gain visibility by benchmarking vendor risk.

Jeff explained that this proprietary scoring algorithm applies a single number to the risk potential of any company with the idea being to emulate something like a credit score, which demonstrates safety as it pertains to finances. The CrowdConfidence Score is a representation of various factors that help a company quickly determine if they want to do business with a vendor or not.

Vendors work through a Whistic-supported vendor assessment questionnaire and answer binary questions, such as “Does your company do [X] or not?” The score is computed as a function of those responses in addition to the collective wisdom of IT security professionals with respect to which security controls are most important and least important, along with a magnitude of other characteristics.

Jeff went on to explain that the CrowdConfidence Score can and should change over time, and it can change in two ways. First, a company can improve its security profile by turning “on” features that are relevant to the score or by making improvements to the security posture of the platform or the company itself. Second, as Whistic gathers more data through the platform over time, it will continue to learn and iterate. The platform was seeded with collective wisdom of many security professionals, but the algorithm will eventually change to reflect industry-specific needs as well as other external factors, making the score even more comprehensive.

What Was the Origin of the Score?

When beginning to build the Whistic platform, the team had a big challenge. Jeff explained it like this:

“If I wanted to build a system that would allow me to map a company’s security characteristics to a negative security event (such as a breach), I would need to know the security state of every single company around the globe and would need to see exactly when data breaches occur. Data like this simply doesn’t exist because we observe very few breaches in real-time. We don’t know when they’ll occur, otherwise they could be prevented.”

Jeff explained that when Whistic began thinking about how we would collect meaningful data that would help our platform predict security issues and help companies understand risks so they can make better decisions both internally and as it pertains to third party vendors, the origin of the score emerged.

As the team built the initial research study, we presented scenarios to IT security professionals by providing them with several questions, each with different characteristics. We would then ask them which company they would be most likely to trust (based on the differing responses). We repeated the process many, many times, each time presenting a different set of characteristics. Over time, we began to learn which features matter most and which matter least, and began to assign weighted value to each ‘security feature’ that make up commonly recognized security controls.

Once we knew we had a solid algorithm in place — taking into account binary data from companies, characteristics from similar industries or companies, and combined it with deep IT expertise — we filed a provisional patent on the idea and have since filed for a full patent.

How Can It Help Whistic’s Users, and What Can They Expect In the Future?

Our customers know firsthand that it’s very difficult for InfoSec teams to process 300+ questions on each vendor response and then determine with authority whether they want to move forward with a relationship or not. The benefit of going from this abstract place to a single number (ie, the CrowdConfidence Score) is that security teams can use it to speed up their decision-making process and learn how their own company or their vendors are performing in terms of security measures. Ultimately, this saves InfoSec teams significant time resources that they can dedicate to more strategic priorities.

Users can also see categorical breakdowns of security controls to identify specific problem areas in which they can drill into and determine how much value they put onto a certain category, such as Audit Assurance and Compliance and Application and Interface Security. In addition, if a company chooses to conduct an on-site audit or a deep-dive assessment for a high-value or high-risk vendor, they can use their time to drill into specific areas called out by the score.

And in the future? Jeff explained that long-term, the aim of the score is to become a definitive ranking by which to determine vendor risk. IT security professionals will be able to ask “What is their CrowdConfidence Score?” It’s Whistic’s goal for the score to be part of every new vendor assessment conversation, which will ultimately help all security professionals — not just Whistic’s users — as the score becomes more accessible even outside of the platform.

Ready to Learn More?

Check out our resources below for more third party vendor best practices and insights on how your organization can effectively respond to security assessments.


Why Third Party Security is Critically Important

Request a Live Demo with a Whistic Product Specialist

risk assessment vendor security security vendor assessment risk scoring

About the author


The latest insights and updates on information security and third party risk management.

Hate security reviews?
Want FREE AirPods?*

Offer valid for any decision-maker/influencer in relation to your company’s third-party risk management strategy. Company size must exceed 100 employees. Exclusions apply. Limit 1 pair per company.