Going Viral – What it means at the network level

In todays world, we constantly hear about posts “going viral”, however as engineers, we don’t really have visibility into what impact going viral can have on your infrastructure until it’s too late.

Thankfully at WordPress.com, we have quite a good history of weathering the storm of viral posts. Most recently, we had a post go quite viral, and I wanted to share the impact in terms of traffic it has:

Screen Shot 2014-02-14 at 09.44.58

 

At about 17h30 UTC, we saw one post go viral, bringing in about 5Gbit extra, however a real storm hit when a popular site published a really popular post. You can see almost 10Gbit extra come in across a 15 minute window.

Graphs curtesy of the absolutely excellent Observium, which we use to monitor all our network devices.

 

Filtering NTP to your control plane

Lately, there have an large increase in using NTP amplification to generate traffic for DDoS attacks. This is due to CVE-2013-5211. All JunOS platforms are affected as per JSA10613 .

As it stands already, all your control plane traffic should be filtered, and unneeded traffic should be dropped. Juniper themselves suggest a fix, however I think this is a perfect opportunity to introduce a nifty JunOS features called apply-path which allows you to create dynamic prefix-list based on some part of the configuration.

First, we will need two prefix-lists to filter – one to match all configured interface addresses, and one to match any NTP configuration.

!! Setup ACL's to match configured NTP peers and all addresses owned by the router 
set policy-options prefix-list router-ipv4 apply-path "interfaces <*> unit <*> family inet address <*>"
set policy-options prefix-list ntp-servers apply-path "system ntp peer <*>"

And now, verify!

> show configuration policy-options prefix-list router-ipv4 | display inheritance 
##
## apply-path was expanded to:
##     199.47.90.144/30; 
##     72.51.48.84/30; 
##     64.125.75.136/30; 
##     62.115.37.56/30; 
##     192.0.127.0/31; 
##     192.0.127.6/31; 
##     192.0.127.8/31; 
##     192.0.127.253/32; 
##
apply-path "interfaces <*> unit <*> family inet address <*>";

And now, we just configure our NTP peer:

!! configure your NTP peer. This will populate the ntp-servers prefix list. 
set system ntp peer <snip>

Finally, tie it all together to match traffic sourced from the local router, or our configured NTP peers:

!! Match traffic to/form NTP server to/from locally configured interfaces    
set firewall family inet filter CoPP term ACCEPT_NTP from source-prefix-list ntp-servers
set firewall family inet filter CoPP term ACCEPT_NTP from source-prefix-list router-ipv4
set firewall family inet filter CoPP term ACCEPT_NTP from destination-prefix-list ntp-servers
set firewall family inet filter CoPP term ACCEPT_NTP from destination-prefix-list router-ipv4
set firewall family inet filter CoPP term ACCEPT_NTP from protocol udp
set firewall family inet filter CoPP term ACCEPT_NTP from port ntp
set firewall family inet filter CoPP term ACCEPT_NTP then count ACCEPT_NTP
set firewall family inet filter CoPP term ACCEPT_NTP then accept

Just apply the policy to your loopback interface, and you’re done:

set interfaces lo0 unit 0 family inet filter input CoPP

Just dont forget to also allow management traffic to your router

A blog about Network Engineering

Follow

Get every new post delivered to your Inbox.