Real-World Data: Automated Assessment Outcomes and Comparisons Using AI

According to our 2025 third-party risk management (TPRM) impact report, 59% of risk and InfoSec leaders view AI as the single biggest factor in the future of the industry. 

The reason is simple: AI is an amplifier. It takes great, disciplined TPRM and makes it faster, more insightful, and easier for those doing the assessment and their vendors. That means your existing resources go farther, allowing you to: 

• Perform all the assessments you need (94% of companies say they’d assess more vendors if they had the capacity).
• Perform the depth of assessment you require (97% of companies say they don’t have time to go deep on vendor risk).
• Ease the burden on vendors to build stronger relationships.
• Get through the TPRM assessment process faster, so you can start to benefit from the tools and partnerships your business needs.  

AI can deliver these outcomes by automating manual steps in the vendor assessment process, so you can get the decision-making info you need in a fraction of the time. 

In this article, we’re sharing data from the Whistic Platform’s suite of AI capabilities—called Assessment Copilot—that demonstrate these time savings using some of the most commonly available vendor security documentation. 

TPRM Automation Using Vendor Summary Tool

Before we dive into our findings, it’s important to understand how this information was generated in the platform and how Whistic AI is designed for automation. Whistic’s AI suite has two core capabilities:

1. Vendor Summary 

Vendor Summary AI performs an automated assessment based on the vendor security documentation you’ve collected and made available to the platform. Using this documentation, Vendor Summary performs a risk assessment against your chosen framework; this can be a regulatory framework, standard questionnaire, or customized questionnaire. 

The completed AI assessment includes:

• Designation as to the compliance of the vendor based on responses
• Complete rationale for every response
• Full citations for every response and direct links to the source
• Confidence scores based on the certainty of the response
• Designation as to the compliance of the vendor based on responses
• Full audit capabilities for deeper dives or to add notes to the completed report

2. SOC 2 Summary
This second core AI feature takes lengthy SOC 2 audit reports and distills them to a five-page executive summary based on your specific controls. This allows you to focus on meaningful exceptions without poring over hundreds of pages manually. It’s also digestible, making it perfect for sharing with the executive team or other stakeholders. 

Methodology: Comparing Assessment Completion Percentages Using Common Frameworks

For the purposes of this article, we’ll be focusing on the automation results that can be achieved using the Vendor Summary tool. Specifically, we looked at the percentage of questions that AI was able to answer based on the following initial circumstances:

Assessment Framework

The assessment framework is used to determine if the vendor meets acceptable risk criteria; it’s the standard against which vendor risk is measured. For this report, we used Vendor Summary to measure compliance against four common security frameworks that cover a broad range of risk. These frameworks are:

• CAIQ LIte
• SIG Lite
• HECVAT (used for regulatory compliance in higher education)
• Whistic Control Framework (WCF; this framework is based on the NIST framework, modified to make the 98-question set more AI-ready)

Data Sources for the Assessment

AI automates vendor assessments based on the security data provided. But one of the most time-consuming parts of the TPRM process is collecting answers to custom questionnaires manually from a vendor. And 79% of all companies report that their process is largely questionnaire based. 

But huge time savings are possible if you are able to begin an automated assessment with data that you either already have access to or is easier to source. This may include:

• Security profiles in a shared exchange like Whistic’s Trust Center Exchange
• Publicly available trust centers
• Risk rating services (utilized by 88% of companies we surveyed)
• Completed standards or audit reports like SIG, SOC 2, or CAIQ; many vendors have already collected such responses and are happy to share quickly (with the help of an NDA)
• Common security policies; these are also easier for the vendor to send than a questionnaire, and AI can analyze them so you don’t have to do it manually. This makes policies a much more resource-effective part of the process when you leverage AI. 

For this comparison, we organized the available data types into tiers to illustrate how much time you can save based on the amount of data you have available:

• Tier 1: A completed SOC 2 audit report
• Tier 2: SOC 2 + collected policies*
• Tier 3: SOC 2 + policies + completed standard (CAIQ Lite, SIG Lite, and HECVAT)

We assessed each Tier of data against the WCF to measure:

• The percentage of questions AI could answer completely
• The percentage of questions AI could not answer, but could identify supporting sources
• The percentage of questions AI could not answer

Additionally, we assessed Tier 3 data sources against CAIQ Lite, SIG Lite, and HECVAT standards, respectively, to illustrate the time savings you can realize using other common frameworks. 

Let’s take a look at the results.  

Automated Assessment Responses Using the Whistic Control Framework

In the following chart, you can see the results achieved for each Tier of security data when Vendor Summary assessed them against the WCF:

Analysis

Here are some important takeaways from the Tier-based approach:

• Massive time savings. For each Tier of data, Vendor Summary was able to provide complete answers to a huge majority of the questions in minutes. While no data Tier was able to answer 100% of all questions in the WCF, the reduction in time represented is clear. 
• Clear source citations increase time savings. Even when AI was unable to confidently answer a question based on the data sources, it was able to close the time gap by providing direct citations for supporting information within the sources. That means that instead of manually searching through documentation, your team can now go directly to the relevant section of the source, making it faster and easier to locate an answer. 
• Easy-to-share data is impactful. Each data Tier used in this study is more readily available and easier to collect from a vendor than a manual questionnaire. By simply collecting a SOC 2 (Tier 1) from a vendor, you can complete more than 70% of your vendor assessment in just a few minutes. Another way to think of this: if your typical questionnaire is 100 questions long, Vendor Summary and a SOC 2 can reduce that questionnaire to 12 questions. If you’re able to collect more data easily, that number can be reduced to a single question. 

Automated Responses Using Other Common Frameworks

As we mentioned previously, the Whistic Control Framework is based on an AI-optimized version of NIST. Setting up the WCF to be “AI ready” helps to increase the completion percentages, automating more of the vendor assessment. 

But we understand that not every company uses the WCF/NIST framework as their assessment standard. Some use other common frameworks, while other companies use a compliance-specific framework required by regulation. AI can still automate huge portions of the assessment process against other frameworks and generate time savings.

In the following two charts, we show a comparison of response rates using a few of these common frameworks and one regulatory framework (HECVAT, used in higher education). We’ve also included the WCF for comparison. 

In the first chart, we demonstrate response rates against these frameworks using a SOC 2, policies, and a completed CAIQ Lite questionnaire: 

And in the second chart, we demonstrate response rates for the same frameworks and data, substituting a completed SIG Lite questionnaire: 

Overall Takeaways

Our survey data shows that the majority of businesses do not have the time and resources to conduct all the vendor assessments they’d like at the depth they feel is necessary. That simply means companies are accepting too much risk. 

Closing that gap with headcount is expensive, and it probably isn’t enough—the growing size of vendor inventories is outpacing the growth of TPRM teams. That’s an expensive arm race to be losing. 

As this study shows, Whistic AI makes it possible to:

• Begin an assessment faster with more readily-available vendor security data.
• Automate a vast majority of assessment responses, even if you can only collect a small amount of information from your vendor.
• Run an automated assessment against the framework of your choice—and still see time savings in minutes. 
• Eliminate manual, administrative overhead—so you can assess more vendors in greater depth, in less time and without adding headcount. 

Whistic’s Assessment Copilot suite of AI capabilities isn’t just about speed, though. Our automated assessments also include confidence scores, rationale for responses, specific citations for every AI response, and the ability to automatically create a short custom questionnaire based solely on the few questions AI couldn’t answer. 

In short, it’s an automated assessment engine built on trust, transparency, and insight. It’s closing the value gap between TPRM costs and TPRM outcomes, making business more effective and helping them take on more risk. 

To learn more about how we conducted this study and to see Whistic’s Assessment Copilot in action, please schedule some time to meet with our team for a brief consultation. 

*Because we are a vendor and respond to many assessment requests, we used Whistic’s own policy documentation to conduct the AI assessments; policies used include: Acceptable Use Policy; Whistic Security Overview; System Access Control Policy; Vulnerability Management Policy; Incident Management Policy; Password Policy; Vendor Management Policy; Asset Management Policy; Backup Policy; Data Protection Policy; Whistic Secure Development Lifecycle Policy; Regulatory Compliance Policy; Disaster Recovery Plan; Risk Management Policy; Encryption Policy; Data Retention Policy; Whistic Internal Audit Policy; Physical Security Policy; Data Classification Matrix; Whistic Business Continuity Plan