UPDATE: NSS Labs Report Confirms Testing of CrowdStrike Falcon was Incomplete and Wrong

February 15, 2017

  • NSS’ report plainly states that testing of CrowdStrike Falcon was incomplete, and therefore, the results are invalid.
  • Including Falcon in the report based on an incomplete analysis is contrary to basic industry standards for testing.
  • All Falcon prevention capabilities were disabled during the testing, and therefore the report results are wrong.
  • CrowdStrike declined to participate in a public test after completing a private test with NSS, based on NSS’ flawed and improper testing execution.

On February 14, 2017, NSS Labs released a report available for purchase for approximately $12,000 claiming to analyze and address various advanced endpoint protection products in the security marketplace. After repeated requests, we were finally granted access to the report. Due to the extraordinary cost of the report, many readers of this blog (including people who have already commented publicly) have not reviewed the actual report. We want to set the record straight.

Here are the facts:

1: The testing of CrowdStrike Falcon was never completed

CrowdStrike is the only vendor in the report identified by a red dot, which — as NSS explicitly acknowledges — means it obtained only “partial data” for CrowdStrike. In fact, three out of the nine subcategories covered by the NSS AEP Group Test were labeled as “Test Not Complete” for CrowdStrike, meaning one third of the tests conducted on the CrowdStrike product were never completed. Including Falcon in the report based on an incomplete analysis is contrary to basic industry standards for testing.

2: CrowdStrike Falcon prevention capabilities were disabled during the incomplete public test

Based upon a review of the telemetry and full audit logs in the Falcon Platform, prevention settings were turned off during the entire test period. The report ignores this simple fact. As a result of the failure to turn on prevention, the report’s conclusions about the total cost of ownership, the blocking of malware, exploit mitigation, and blended threat prevention are simply false.

In addition, NSS states “over the course of the test, Falcon Host blocked 70.0% of malware delivered by HTTPS.” Since prevention was completely turned off during the entire time of the testing, no blocks ever triggered in the UI, where did the 70.0% block rate come from? And since this result, as well as results for all other preventions that NSS had given us credit for, are wrong, it calls into question all other results.

Based on the above, the inclusion of CrowdStrike in the report is wrong and inconsistent with industry testing standards, and presents an inaccurate and misleading picture of our Falcon product.

Below is some background to why we declined to participate in the NSS Labs AEP Group Test:

  1. Once we learned that NSS had a deeply flawed methodology and made basic errors such as naming Firefox, Skype and Java (which are digitally signed by legitimate vendors) as malware, we had little faith that any test performed by NSS would be accurate.
  2. We were also aware of other security providers such as FireEye and Palo Alto previously voicing public concerns surrounding NSS’ prior public testing activities.
  3. We directly engaged with NSS by telephone and through email to address problems with its testing methodology. NSS representatives admitted they had made mistakes.
  4. We initially sought to work collaboratively with NSS to address the testing issues. We told NSS’s CEO, Virkram Phatak, that CrowdStrike would not participate in the public test unless they fixed the underlying problems in the testing methodology. Furthermore, we also informed him of his contractual obligations by email and provided him a link to our terms of service, which expressly prohibits access to Falcon for unauthorized competitive testing. In addition, our counsel sent a letter to NSS’ external counsel restating these limitations.
  5. NSS ignored our concerns and our explicit direction to not go forward with the public testing. Instead, NSS colluded with a reseller, David Thomason, to create credentials from a customer account of a Fortune 1000 company in order to provide NSS access to the Falcon platform. This behavior was in violation of our contracts, multiple provisions of our license agreements, and various laws.
  6. Not to be deterred, NSS went ahead and published its admittedly incomplete report.
  7. As a company that cares deeply about the integrity of ethical product testing and the protection of our intellectual property and contractual rights, we recently filed suit in Federal court. This litigation remains ongoing, and we intend to pursue our claims against NSS and hold it accountable for its improper conduct.

Taken in total, NSS’ failure to conduct the most basic of fact checking during the private testing and the well publicized history of problems with NSS testing ultimately gave us no confidence that NSS Labs could conduct accurate testing of our security products. Therefore, we declined to participate in the public test.

For any questions regarding the NSS Report and CrowdStrike’s response, please email: FactsAboutTesting@crowdstrike.com

Don’t take our word for it, review these reputable industry organizations and what they have to say about our technology. CrowdStrike was recently named Visionary in the Gartner Magic Quadrant for Endpoint Protection Platforms and CrowdStrike Falcon has been independently tested and certified as an effective AV replacement by AV-Comparatives and SE Labs.


 

On Friday, February 10, 2017, CrowdStrike filed suit in U.S. Federal District Court against NSS Labs to hold it accountable for unlawfully accessing our software, breaching our contract, pirating our software, and improper security testing. Regardless of test results (which we have not seen), CrowdStrike is making a stand against what we believe to be unlawful conduct.

CrowdStrike values independent testing and we initially engaged NSS to conduct a private test of our software. We soon learned their methodologies were deeply flawed. For example, they made basic errors including labeling legitimate software such as Firefox, Skype, and Java, digitally signed by vendors, as malicious – leaving us with no confidence in their testing methodology or ability. As a result, we decided not to participate in a public test and expressly declined NSS’ later request to conduct public testing. After explicitly telling NSS on multiple occasions that they were prohibited from using our software for public testing, they colluded with a reseller and engaged in a sham transaction to access our software to conduct the testing. In doing so, NSS breached their contract with CrowdStrike, violated our end user licensing agreement (EULA), misappropriated our intellectual property, and improperly used credentials. Once we became aware that an unauthorized user account associated with a reseller was used for the tests, we suspended access immediately. Any test results that NSS did obtain are incomplete and materially flawed.

CrowdStrike is Committed to Qualified Independent Testing and Validation

CrowdStrike supports independent and ethical validation—including public testing—for our products and for the industry. In fact, we underwent public testing with two reputable independent testing houses: AV-Comparatives and SE Labs in the past six months. We believe the actions of NSS are detrimental to the security industry and they need to be held accountable. We reject the unethical, illicit, and subversive way that NSS does business and the harm it brings to our industry, security research, and most of all, the users of security technologies. We hope that other leaders in the security industry will join us in speaking out and taking action against those who seek to harm our industry and security for their own gain.

To be crystal clear, the results of the report are unknown to us at this time and irrelevant, we are suing NSS because of their illicit activity, breach of contract and misappropriation of our intellectual property.

Please follow our blog as we continue to share more updates on this matter.

 

Stop Breaches with CrowdStrike Falcon request a live demo