Subscribe To Our Daily Enewsletter:

Surviving the DDoS Attack Against Linode

linodeLinode, a global data center provider, has been battling a distributed denial of service (DDoS) attack since Christmas Day. The DDoS attack is a painful reminder that hundreds — perhaps thousands — of businesses can be thrown offline when key Internet connection points go dark. And it begs the question: Is the U.S. government doing enough to find and prosecute those who attack U.S. infrastructure?

For example: WP Engine, a leading provider of WordPress hosting systems, is among those that suffered outages during the extended DDoS attack against Linode. Some of WP-Engine’s customers — including ChannelE2E — were impacted by the outages.

Updated January 5, 5:35 p.m. ET: Federal agents now investigating DDoS attacks and unauthorized logins targeting Linode.

DDoS Attack: Massive and Sophisticated

Alex Forster

Alex Forster

The DDoS attack against Linode sounds like it was massive and sophisticated.  In a December 31, 2015, status update, Linode Network Engineer Alex Forster wrote, “Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change.”

Each time Linode took corrective action, the attackers seemed to change their approach. Things got particularly bad on January 1 and January 2 — when the DDoS attack undermined Linode’s Atlanta, Ga., data center.

After troubleshooting and countering the issue in multiple ways for 36 hours, Linode on January 2 acquired a dedicated transit link that is now directly connected to the Linode network. The transit provider, in turn, applied DDoS mitigation hardening. But the challenges didn’t end there.

DDoS Attack Hits Close to Home

ChannelE2E’s own website performance took a hit on January 1. Readers like Internet & Telephone CTO MJ Shoer were kind enough to say our site apparently had an outage. We replied explaining the DDoS attack at Linode, and thankfully we were back online shortly thereafter.

Things for us improved a bit on January 2 but we began to ask a key question: What if the DDoS attacks never stop — and what if Linode, in turn, never fully mitigates the attack? With that worst-case concern in mind we dialed WP Engine phone support mid-day on January 2 and asked them to move us to another data center.

WP Engine’s response? Although there was 30 minutes of hold time, the support engineers were professional and responsive during a high-stress period for the company. At a time when most employees had hoped to be on holiday, it was clear that WP Engine was striving to get on top of the service calls.

  • The good news: Both Linode and WP Engine have managed to keep us online since overcoming the bulk of the attacks on January 2.
  • The bad news: WP Engine, swamped with service calls, has yet to move us to another data center as of this writing. But we do have an open support ticket documenting the rest…

Overall, it sounds like the DDoS attacks against Linode’s Atlanta-based data center have ended. But the company continues to battle additional inbound attacks elsewhere. By January 3, the DDoS attackers were targeting Linode’s DNS infrastructure. At the time the company noted:

“Our engineers have taken steps to further harden our DNS systems against further attacks. It should be noted, however, that AXFR functionality has been disabled for the moment due to these measures. We will resolve this once we are able to implement additional levels of mitigation. We are continuing to monitor for any further disruptions. Thank you for your patience.”

DDoS Attacks and U.S. Cyber Security Policy?

At first glance, it’s easy for customers to blame Linode for the DDoS-related outages. Some folks were especially harsh on Twitter, flaming both Linode and WP Engine for lack of communication and poor support during the extended DDoS attacks.

Yes, both companies could have communicated more clearly. But this is becoming a familiar story during DDoS attacks — whether they involve Linode or NetFlix or Sony or Microsoft. Instead of blaming the victim, shouldn’t we blame the attackers? And shouldn’t we also call on the U.S. government to accelerate and strengthen this country’s cyber security plans and policies?

Think of it this way: If a foreign organization continually carpet bombed a U.S. power utility and knocked power offline, the U.S. government would likely declare war against the offenders. Anything short of that and U.S. citizens would openly wonder if the government was serious about homeland security — and protecting our infrastructure.

Roughly one year ago, The White House announced plans to fight DDoS attacks. More recently, presidential candidate Donald Trump has made wild statements about shutting down the Web in ISIS-controlled Syria and Iraq — an idea that would be a logistical nightmare.

I sense that the Obama administration is working behind the scenes to sort through the latest cyber security policy issues. But it’s time for more aggressive action. Linode and WP Engine — and their customers — have felt serious DDoS pain in recent days.

And the pain for Linode’s engineers isn’t likely to reside anytime soon. As I was wrapping up this blog on January 4, I received this update from Linode’s incident response team: “The denial of service attacks against our DNS infrastructure have resumed at this time.”

And that brings me back to my original thesis: What if the DDoS attacks never end? It’s time for the U.S. government and private industry to assume they won’t… and then plan accordingly.

Return Home

4 Comments

Comments

    Alex Williams:

    Very well put. This is clearly cyber warfare and it hit home this holiday season for a lot of innocent customers. Thanks for running this story.

      Joe Panettieri:

      Hey Alex: Thanks for the note and for reading ChannelE2E. I have to confess: I originally thought WP Engine had their own data centers. I didn’t realize they were hosted in the Linode data centers until the DDoS attacks started. Always good to ask your SaaS providers where they’re housed… and why… and what’s the policy to change that location if needed?

      Overall, I’ve been pretty impressed with WP Engine’s responses. And I do feel for Linode — the target of these attacks. Once the dust settles I’ll be curious to read the additional info Linode shares (they’ve promised to offer a deeper dive at some point). And, I wonder if/when our government will stand up against these attacks.
      -jp

    Nando:

    Moving WP Engine to another data center, like Digital Ocean for instance, isn’t a real solution. The DDoS attackers could turn on DO next. Then what do you do? Move back to Linode? Try a small, obscure provider that would be completely overwhelmed by such an attack and forced out of business? At least Linode has the resources to fight it off.

      Joe Panettieri:

      Hi Nando: You’ve raised a key point that we’ve been considering in recent days. And that’s why we’ve been leaning heavily on WP Engine for their thoughts/guidance in recent days. Running from one data center ‘provider’ to the next isn’t much of a practical option. But sticking with your partners — especially if they’re responsive — and moving around in their infrastructure is a potential solution (though certainly not fool-proof). Hence the need for the U.S., as a whole, to have a far more aggressive stance against these attacks…
      -jp

Leave a Reply

Your email address will not be published. Required fields are marked *