The Gotham Bus Company

  • About
  • How Can We Help?
    • IT/Tech Consulting
    • Managed Cloud
    • Managed WordPress Hosting
    • Colocation/Datacenter
    • Shared Hosting
    • Spam Filtering
    • Denial of Service Attacks
  • Blog
  • Contact Us
  • Get Help

September 22, 2015 by Drew Linsalata

Root Cause Analysis – Datacenter Connectivity Issues Sept 15-16 2015

On the morning of September 15, 2015 connectivity to our Long Island datacenter was interrupted by a distributed denial of service (DDoS) attack against a neighboring subnet in the datacenter.  Analysis shows that this was a RIPv1 reflector attack, wherein old equipment still on the Internet is used to create malicious packet streams designed specifically to cause problems with routing and switching equipment.  While the attack stream was not overly large, the specific profile of the attack was such that the existing DDoS edge protection on the network was not sufficient for mitigation. This caused problems in the routing core of the network, effectively breaking connectivity for customers with equipment located in the datacenter.

Connectivity was restored by isolating the attack target using blackhole routing, by completely removing the target from the network, by establishing dedicated and isolated ingress and egress paths for the target, and also by adding additional traffic filters both on the local network and on the networks of several upstream providers.  The initial impairment lasted for approximately 3 hours on the morning of the 15th.  There were additional partial outages on the afternoon of the 15th, then again in the early morning hours of September 16 when additional filtering was added that permanently mitigated the attack.

A root cause analysis of this issue shows us that the edge DDoS protection on the network was simply not sufficient to quickly and automatically handle this type of attack. Instead, manual intervention and engineering was required, which is naturally slower to implement.  To prevent this from happening again, edge DD0S protection has been upgraded and augmented with on-demand BGP-based mitigation from Arbor Networks, one of the industry’s largest protection providers. This solution was implemented late last week and has been tested thoroughly with no impact on service.  In the event of a similar or larger attack, all traffic is diverted in real-time to the Arbor network, where it is “scrubbed” before being passed through to us. In most cases the only noticeable impact could be a few extra milliseconds of transit time, with little to impact on real-world performance from an end-user perspective.

While we cannot stop future attack attempts, we feel confident that the upgraded protection now in place will prevent any future attack from causing the disruption experienced last week.

 

 

 

Filed Under: Network Status Tagged With: networkstatus

Follow Us

Contact Us

FREE TECH NEWS AND TIPS

Get helpful tech news and tips via email. We’ll never spam you or share your information with anyone. Un-subscribe at any time for any reason.
  • This field is for validation purposes and should be left unchanged.

See Posts About ..

aws business cloud ddos dns domain name registration domain names email funny hosting internet legal monitoring networkstatus outage security spam sysadmin virtualization webdesign windows wordpress wordpresswisdom

Navigation

  • About
  • How Can We Help?
    • IT/Tech Consulting
    • Managed Cloud
    • Managed WordPress Hosting
    • Colocation/Datacenter
    • Shared Hosting
    • Spam Filtering
    • Denial of Service Attacks
  • Blog
  • Contact Us
  • Get Help

Gotham Bus Policies

  • Acceptable Use Policy
  • Privacy Policy
  • Copyright/DMCA

Recent Blog Posts

The Cloud For Non Technical People in About 90 Seconds (Audio)

WordPress Management for Non-Techies: The Weak Link Is People

LinkedIn Marketing – Really A Thing?

WordPress Management For Non-Techies: Proactive vs Reactive

Copyright © 2023 · The Gotham Bus Company

Copyright © 2023 · Executive Pro Theme on Genesis Framework · WordPress · Log in