RMM Software Is Critical U.S. Infrastructure
At first glance, the United States dodged a bullet amid the REvil Ransomware attack against Kaseya VSA and MSPs that run the RMM (remote monitoring and management) software.
President Biden believes the attack caused minimal damage to U.S. businesses. In many ways he has a point: Electric grids didn’t go dark. Airports didn’t close. Energy pipelines continued to flow. The casual American may have seen the Kaseya cyberattack story on the evening news before quickly moving on to other news of the day.
Still, there was fallout. The attack extended from Kaseya to roughly 50 MSPs. From there, the automated attack stretched out to roughly 1,500 downstream customers. Some reports suggest a million endpoints were impacted — but that figure may involve the REvil gang seeking to pump up the story.
MSPs Flying Blind Without RMM Software
The bigger issue — largely missed by the mainstream media — involves perhaps 10,000 or more MSPs that went without RMM software for more than a week, while Kaseya engineers worked around the clock to patch SaaS- and on-premises versions of the software. We also don’t know how many small businesses permanently lost data in the attack.
For those who missed the cyberattack timeline:
- Friday, July 2: The attack starts. Kaseya shuts down the SaaS version of VSA software, and urges MSPs to turn off on-premises servers that run the software. Many U.S.-based MSPs and cybersecurity pros were heading out the door for the July 4 holiday weekend… only to discover they’d be working instead of celebrating Independence Day.
- Tuesday, July 6: Kaseya attempts to restart the SaaS version of its VSA software, only to halt the process and double down on more security enhancements. CEO Fred Voccola apologizes for the delay, and vows the company will have VSA up and running again by Sunday, July 11.
- Sunday, July 11: Kaseya begins the SaaS version restore and also issues an on-premises VSA patch, along with specific guidance to help MSPs implement the patch. 60% of SaaS customers come back online, with the remaining 40% expected to be back online within hours.
- Monday, July 12: 100% of SaaS-based VSA customers are back online as of 3:30 a.m. ET. By midday or so, Kaseya takes down the SaaS service for unplanned maintenance. The overall SaaS service is fully up and running again by 3:30 p.m. ET or so.
What’s the big deal with that timeline?
- Imagine airplanes flying without radar and air traffic controllers;
- doctors working without X-Ray and imaging systems; or
- first responders working without GPS.
That’s the MSP market without RMM software for roughly 10 days.
Yes indeed, President Biden: Critical U.S. infrastructure was hit by the REvil Ransomware attack vs. Kaseya. In this case, the critical U.S. infrastructure is RMM software.
How Ransomware Threatens U.S. Small Business Economy: The Math
Let me be clear: I’m not knocking Kaseya. Instead, I’m reinforcing a larger issue that extends across Main Street U.S.A. The economic math looks something like this:
- There are at least 50,000 or so MSPs that run RMM software from Kaseya and rivals such as ConnectWise, Datto, N-able, NinjaRMM, Atera, SuperOps.ai, Syncro, Naverisk and others, ChannelE2E believes.
- Those 50,000 or so MSPs support anywhere from 500,000 to 5 million small and midsize businesses worldwide, using a conservative estimate of 10 to 100 end-customers per MSP.
- Those 500,000 to 5 million SMBs surely have tens of millions of PCs, servers, smartphones, cloud workloads, information technology (IT) and operational technology (OT) under MSP management.
Attack the RMM software market, and you’ve attacked the SMB heartbeat of America’s economy — along with regional economies worldwide.
Indeed, a cyberattack on a single MSP or MSSP could cause $80 billion in economic losses across hundreds of small businesses, a research report issued before the Kaseya attack asserted.
Perhaps that’s why President Biden’s executive order on cybersecurity — issued in May 2021 — specifically called out IT service provider security practices more than a dozen times.
MSP Judgment Day: Industry Insiders Saw It Coming
Rewind to June 2019. ChannelE2E warned readers that MSP Judgment Day was coming. While we love the MSP industry, we predicted the MSP sector could soon face a “crisis of credibility” if the market doesn’t take major steps to more effectively mitigate ransomware threats, attacks and fallout.
I do believe MSPs and their software providers take security more seriously in 2021 than they did in 2019. But our view on the market in 2021 remains unchanged from 2019, at which time we predicted:
- In a worst-case scenario, the MSP industry could be torn apart if ransomware-related lawsuits fly between end-customers, MSPs and their technology providers.
- In a best-cast scenario, MSPs and their technology providers emerge as Dark Knights that snuffed out ransomware long before attacks reached end-customer systems.
- Anywhere in-between leaves us with a crime-ridden Gotham that tarnishes the MSP industry as a whole.
The Case Is Made: Protecting RMM Software Is Critical to U.S. Economy
As I wrote in 2019: It’s time for MSPs and their software providers to rise to the occasion. And as for the federal government, it’s time to realize that RMM software is critical infrastructure. I suspect that means it’s time for the federal government and the MSP industry to sit down with one another. And yes, it’s time to discuss — at a federal level — how to regulate and protect MSPs, and the powerful software tools they run.
I agree Joe, something needs to happen to secure both the future of the managed services market and the SMB segment it serves. The question is, how is this solved? Regulation may call for compliance certifications, but in the case of Kaseya they have SOC 2 Type 2, ISO 27001, and more which would suggest they have acceptable policies and controls in effective operation. What’s the next step up, and which emerging or smaller technology companies will be able to digest the cost associated with meeting such prerequisites from an ongoing regulatory audit standpoint beyond the implementation and operational burden? While something is needed, it will need to be carefully considered or it could lead to monopolization of the industry or even worse, stifle innovation.
Sorry, Joe, I do not believe the software providers care more about security in 2021 than they did in 2019. They are motivated to generate financial results in the same way they have previously, which does not factor in changes in security posture. I’ve made two cases in editorial pieces about the broken financial incentives of software executives.
– SolarWinds elected to pay their executives their bonus for 2020, despite over seeing the largest espionage breach of the US government. One executive elected to blame the intern in testimony to Congress. (Editorial, including quote https://www.youtube.com/watch?v=o6nXj2-Mokg)
– Kaseya outlined that their security plan worked exactly as intended, and this “is the world we live in.”. Kaseya’s position “All of a sudden cyber crime and ransomware become the topic the day, and we’re caught in the middle of it” (Editorial including quote, https://youtu.be/r8r97GonMc4). This stance despite your coverage, and ProPublica’s piece of the risk via MSP in 2019, or the warnings from CISA in 2018.
In the Miami Herald, Voccola notes “We might go from a 28% annual growth rate to 26% or 25%”, and that “The part of our business that got hit is an important one but a small one.”
The larger the company, the more capable they are of simply writing off a security breach as a cost of doing business. As I note in my first editorial cited above, Marriott, Equifax, and Target all show that from a market valuation perspective, those companies saw little material damage over any extended period of time. Even SolarWinds is well back on track to recovering their market value, as measured by the stock price.
Small companies are bearing the brunt of the ransomware. A large organization can simply write off the breach. A small company does not have the margin or valuation to absorb the breach, despite now becoming a target.
For those MSPs flying without instrumentation…. now is the time to examine that dependency. Those vendors supplying software do not have the same small margin of error that the IT services company does, and without actual financial consequences, will continue to act exactly as they are being incentivized to do.
Colin: Great to hear from you. I’m not an expert on regulations, but generally speaking sometimes regulations can spark more information & opportunity — rather than less. Tracking HIPAA and Sarbanes-Oxley over the years, I was always amazed by the resulting upside for IT service providers that helped customers address those compliance mandates. Still, I know regulations can also introduce bureaucratic & expensive red tape that’s difficult for small businesses to navigate. We’ll be watching and reporting on how this all plays out.
Dave: Thanks for the healthy debate. I’ll push back a bit on your views anecdotally — though I need to take some time to gather data so that I can potentially share more thoughts in a more detailed way.
So, let’s start with the anecdotes: Generally speaking, I think it’s safe to say that multiple MSP software companies have ramped up their cyber hiring to safeguard their own businesses. Examples include the growing cyber team at Datto, led by CISO Ryan Weeks. Also, N-able hired a CSO. ConnectWise just hired a CIO and launched a bug bounty program earlier this year. NinjaRMM has quietly built a cyber team. And the list goes on. Still, I realize these are anecdotes rather than data-driven information about cyber budgets, exact staffing figures, etc. We’ll strive to gather that info for a follow-up report.
Still, you raise some really important points. The MSP software market, overall, still needs to raise its cyber game. Amid all the cross-company/ISV integrations, all it takes is one weak cyber link in the chain to rock this industry.
And I agree: Small businesses (i.e., MSPs in the SMB sector) don’t have the margin, scale or valuation to absorb and/or write off a breach — while the giants can often recover their valuation (and more) once a breach is mitigated.
Thanks again for your perspectives.
-jp
I have a healthy dose of skepticism here. I think some vendors (Hi Datto) are getting this right, some vendors (Hi CW) are trying and somewhat failing and the rest are either totally getting it wrong or I don’t know about it because they don’t have anything published.
I’ve been pretty vocal that the vendors here – especially the big four – need to do more and get this correct. Yet, we continue to see breaches month after month in these tools.
As an MSP myself, the crappy security posture of my toolset is what I consider the biggest threat to the business. We can do EVERYTHING right, and because one of our vendors didn’t do their thing right I can be put in a bad spot or out of business.
The Kaseya exploit was embarrassing. Looking over that codebase – it’s classic ASP, and not even well written. I’ve seen second-year college students produce better code than what I looked over there. Connectwise Automate isn’t much better, with overly complex and outdated code paths that are almost physically impossible to enforce proper SDLC practices such as unit testing around. I personally have a full takeover chain on Automate I submitted last year, and in my research have several other areas of interest I’m currently looking at.
In the end, the vendors aren’t going to take this seriously until it hurts them financially to do so. A month ago Kaseya’s Vulnerability Disclosure Program said the following:
Bug Bounty
Kaseya does NOT offer compensation for vulnerabilities that are disclosed. We will, from time to time, say thank you for new and interesting reports in our thanks section of this page. Please note however that providing a report does not guarantee a credit.
It now says:
Bug Bounty
We’re appreciative of the research community and welcome productive cooperation in eliminating threats, vulnerabilities or exploits. At this time Kaseya does not offer compensation for vulnerabilities that are disclosed. We will, from time to time, say thank you for new and interesting reports in our thanks section of this page. Please note however that providing a report does not guarantee a credit.
So they made it a LITTLE less hostile towards the research community, but it’s still garbage.
Other vendors (Hello Connectwise) hide their bounty programs behind an NDA – if you get paid you can’t disclose.
We as the MSP community need to demand better from our vendors. Kaseya will pay more in PR for this ONE incident than they would in the entire life of a bug bounty program. A program that may have incentivized up-and-coming security researchers to look over this code and submit this before they got whacked. DIVD did that, but it was too little too late. If Kaseya doesn’t write a HUGE check to the volunteer organization that is DIVD, then a great travesty will have happened because this could have been way worse.
Only by encouraging – no that’s not the right word – DEMANDING that our vendors take security seriously and be more transparent will things begin to change. If they don’t change then your article is completely wrong – MSP’s will learn to run without their RMM tools, because the risk of using one will outway the automation benefits they provide.
Jason: Thanks for taking the time to share your detailed thoughts. You’ve given me — and the industry — plenty of food for thought. We’ll also take a closer look at the bug bounty programs and the associated fine print within and across the industry.
Best,
-jp