Don’t Take The Vendor’s Word For It: The Importance of Third-Party Testing

Correct Check Box Digital Concept On The Circuit Board

The Buyer’s Dilemma

It’s no secret that cybersecurity has become a crowded marketplace, particularly in the last several years. The endpoint protection market, for instance, now has more than 140 vendors, with more new players on the horizon. This can be confusing for organizations trying to evaluate solutions and can lead to a very real condition known as choice overload, or “overchoice.” As described by Alvin Toffler in Future Shock, overchoice occurs when “advantages of diversity and individualization are cancelled by the complexity of the decision-making process.” It can render decision-making overwhelming due to the sheer volume of potential outcomes and risks that can result from making the wrong choice. This concept can apply to many decisions in our lives and definitely extends to endpoint security.

In today’s endpoint market, it is clear that change is needed and wanted. As a recent study by CrowdStrike® and ESG indicates, over the next year, 76 percent of buyers have changed or plan to change their endpoint protection provider. In this process, they are likely to be faced with overchoice.

With so many vendors in the endpoint protection market, it is not surprising that it is rife with buzzwords and hype, making it difficult for even the most diligent decision-makers to choose the best solution for their organizations. To combat this phenomenon, third-party testing has become an increasingly valuable tool for validating the capabilities and claims of vendors —  effectively separating the “wheat from the chaff” and clarifying the options for organizations that may lack the time and staff to test competing claims in their own environments.

Why Third-Party Testing?

Third-party testing streamlines the decision-making process by addressing time and expertise gaps and giving actionable information that technical and business decision-makers can use. This public, independent and objective validation ensures you as a decision-maker are not “drinking a vendor’s KoolAid.” Organizations increasingly rely on third-party testing for a variety of reasons, including the following:

Limited Resources

Organizations in every sector struggle with wanting to evaluate products when making purchasing decisions but often lack the time and resources necessary to make a timely decision. The SANS Institute notes that “smaller organizations may not have the luxury of larger organizations…[and may] evaluate a product strictly based on tests conducted by a third party and/or limited test.” Without a full testing lab and security team, it’s almost impossible to completely assess a solution’s feasibility. Even for those companies that have the resources to perform in-house testing, it’s helpful to verify that your results are valid by comparing them with third-party tests.

Subject Matter Expertise

There seems to be consensus in the industry that independent validation of security products is no longer simply a “nice to have.” In its 2018 Magic Quadrant for Endpoint Protection Platforms, the respected analyst firm Gartner notes, “As with previous Magic Quadrants, this year’s inclusion criteria mandate that vendors must have participated in public, independent testing during 2017.” According to Gartner, “It’s worth noting that many vendors, from traditional to the new wave, are embracing the shift to a more open community. Solutions from vendors without a long-term commitment to engagement and transparency should be approached with caution 1.” In our view, this indicates a vendor’s openness to being critiqued, as well as its commitment to transparency and iterative solution improvement.

Removing Bias

The Anti-Malware Testing Standards Organization (AMTSO) exists to help customers understand which tests are fairly conducted, and has standardized this approach after more than two years of development. They have (literally) set the standard for third-party testing. As a result, choose an organization that is an AMTSO member.

Often a vendor will pay for an independent party to demonstrate its solution’s capabilities in a “doctored” environment with tailored parameters designed for optimal results. Paid test results add little value and simply add to the overchoice “hype” that decision-makers must contend with. Testing should be public, consistent and independent. If these conditions are not met, the results of the test may be skewed — negating the value of the third-party assessment.

When evaluating an endpoint security product for possible use in your environment, following best practices for independent, unpaid and unbiased tests that occur on a regular basis is essential to decision-making. This testing gives organizations the ability to leverage objective, external expertise, because experienced solution testers effectively imitate the threats you’ll face and push a solution’s capabilities to the limit. Valid independent testing provides verification of a product’s true capabilities.

CrowdStrike’s Commitment to Public Testing

CrowdStrike is committed to public testing, and regularly and openly submits the CrowdStrike Falcon® platform to third-party tests that are independent, unpaid and performed on a regular basis. They validate CrowdStrike technology capabilities and provide an opportunity to work with current and prospective customers to ensure they are receiving the best protection possible.

The following is a list of independent tests in which CrowdStrike participates, with links to the results:

  • AV Comparatives: Since being certified by AV Comparatives to replace legacy AV in 2016, CrowdStrike participates in the organization’s highly regarded real-world protection test each month. This pits Falcon against the best endpoint protection solutions on the market and verifies Falcon’s prevention and detection efficacies.
    • Offline Protection: Critical to protecting your endpoints is the ability to do so both on and offline. Validated by AV Comparatives, Falcon can protect endpoints whether or not they are connected to the cloud – ensuring complete endpoint protection.
    • macOS Protection: CrowdStrike is dedicated to protecting across multiple operating system environments, detecting and preventing 100 percent of the malware samples run against it.
  • MITRE Nation-State Emulation Test: In 2017, CrowdStrike was approached by MITRE, a federally funded research and development center, with a request to evaluate the Falcon platform’s ability to detect an advanced persistent threat. In the test, MITRE imitated Gothic Panda, a Chinese nation-state adversary. The test and subsequent report validated the Falcon platform’s success based on MITRE’s widely accepted ATT&CK Framework and revealed a small visibility gap in the product that was immediately corrected.
  • SE Labs: SE Labs validated CrowdStrike Falcon’s capabilities, as the only next-generation endpoint security firm to receive a “AAA” rating in their Q1 2018 Endpoint Protection Test. Based on current cybercriminal tools, techniques and procedures, Falcon was proven to prevent malicious activity while simultaneously classifying legitimate applications.  

This ongoing participation in independent, third-party testing demonstrates CrowdStrike’s commitment to continuously evolving, expanding and consistently improving the Falcon platform  with as much transparency as possible. Our goal is to clarify and simplify your security decision-making process by focusing on the critical information you need to make an informed decision.

Test for Yourself: CrowdStrike Falcon Prevent Free Trial

Third-party testing, although a great tool, should be one of several criteria you use to overcome overchoice when you evaluate endpoint protection solutions for your organization. Before you begin, it’s important to define your goals so that you can accurately match them with each vendor’s claims. Once goals are determined, use multiple evaluation methods, including independent testing results, to narrow your field of options. Finally, CrowdStrike recommends taking the opportunity to try a solution in your own environment, so you can ensure its ability to meet your requirements with certainty.

In order to prove CrowdStrike’s next-generation antivirus (NGAV) efficacy as an AV replacement, CrowdStrike now offers a free trial of its powerful Falcon Prevent™ solution. The trial gives you all the tools you need to successfully test Falcon Prevent by subjecting it to worst-case scenarios and observing the results for yourself. This is a great opportunity to undertake the testing that can give you the most accurate type of independent evaluation — one conducted by you, in your own environment.

If you’re ready to evaluate a more effective endpoint security solution with a minimum of overchoice hype, CrowdStrike invites you to make use of the following resources:


To learn more about replacing your antivirus, CrowdStrike’s detailed “Guide to AV Replacement” provides a comprehensive list of evaluation criteria to streamline the selection process and help diminish overchoice.

Experience the power of Falcon Prevent in your environment: Try Falcon Prevent for Free.

1 Gartner Magic Quadrant for Endpoint Protection Platforms, Ian McShane, Avivah Litan, Eric Ouellet, Prateek Bhajanka, January 24, 2018

CrowdStrike Falcon Free Trial

Try CrowdStrike Free for 15 Days Get Started with A Free Trial