The Glider

In the second round of what may possibly be a protracted Internet skirmish, a denial of service attack briefly blocked access to the cablegate.wikileaks.org web site this morning around 8:00 am EST. On twitter, Wikileaks pegged the DDoS as exceeding 10 Gbps (significantly larger than my 2-4 Gbps estimate for the first round of attacks on Sunday).

As compared with this Sunday’s initial attack (blog analysis available here), ATLAS data from 110 providers around the world suggest today’s DDoS was both larger and more sophisticated. Specifically, today’s attack involved several different components, including a low bandwidth application level DDoS and a 2-3 Gbps Syn attack against the primary “cablegate” IP addresses (the hosted web site is load balanced across data center locations in Europe and the US West Coast).

An example of one of the anonymous alerts ATLAS collected yesterday is shown below. This alert is for a modest TCP Syn attack against cablegate.wikileaks.org targeting high number ports. The source address blocks are anonymized with XX replacing the high number bits.


<attack start="2010-11-30 18:10:01 GMT" stop="2010-11-30 18:56:27 GMT">
    <rate bps="70312432" pps="220847"/>
      <protocols>TCP</protocols>
      <tcpflags>Syn</tcpflags>
      <source>
          <ips>xx.xx.25.0/27</ips>
          <ports>1024-2047</ports>
      </source>
      <dst>
          <ips>204.236.131.131/32</ips>
          <ports>16384-32767,32768-65535</ports>
      </dst>
</attack>


In the below chart, I graph traffic from 110 ATLAS carriers around the world to address blocks (BGP prefixes) used by Wikileaks. Note these address blocks may also include traffic to other customer using the same hosting provider. The attack began around 7am EST though a smaller traffic spike occurs around 2am. All times are EST. At the time of this blog posting, the DDoS is still ongoing though not significantly impacting Wikileaks access.

Based on Netcraft and other reports, the outage was brief though cablegate web site performance was moderately impacted throughout the day.

Interestingly, the attack appears to originate from a relatively small number of source IPs, including machines in Russia, eastern Europe and Thailand.

- Craig

Yesterday morning, a DDoS attack temporarily disrupted traffic to Wikileaks hours ahead of the “Cablegate” release of leaked US documents. Wikileaks announced the outage on a Facebook update and Twitter post around 11:00am EST while simultaneously derogating the attack and insisting “El Pais, Le Monde, Speigel, Guardian & NYT will publish many US embassy cables tonight, even if WikiLeaks goes down”.

In the below graph, I show traffic to one of Wikileak’s primary hosting provider on November 28 through 100 ATLAS providers around the world. At approximately 10:05am EST, traffic abruptly jumps by 2-4 Gbps as the attack begins.

Shortly after the attack started, Wikileaks redirected DNS from its AS8473 Swedish hosting provider to use mirror sites hosted by a large cloud provider in Ireland (and later the US as well). While the DDOS attack generated an outpouring of blog posts, news articles and tweets, it appears to have had little impact on the Wikileaks “Cablegate” disbursement of documents.

Overall, at 2-4 Gbps the Wikileaks DDoS attack was modest in the relative scheme of recent attacks against large web sites. Though, TCP and application level attacks generally require far lower bps and pps rates to be effective (more discussion of recent DDoS trends is available here). Engineering mailing list discussion also suggests the hosting provider and upstreams decided to blackhole all Wikileaks traffic rather than transit the DDoS.

At the time of this writing, all Wikileaks domains are reachable from servers in the US, Europe and Asia. The New York Times and most other major media outlets also have since published extensive synopses of the leaked documents.

While the source of the attack is unknown, blogs and social networking sites have alternatively blamed governments and vigilante hacker groups. At least one twitter account with a history of past attacks ( th3j35t3r">“the Jester”) has claimed responsibility. In earlier tweets, the Jester boasted of using low bandwidth application layer attacks instead of relying on large botnets (all of which is consistent with the data ATLAS observed for this Wikileaks attack).

Wikileaks also came under fire in 2008 with a 500 Mpbs DDoS attack shortly before the release of leaked Swiss bank documents.

Update: A follow-on blog post analyzing the second day of Wikileaks DDoS attacks is now available here.

 
- Craig
 

My blog post last week on the April 8th China BGP hijack incident generated significant discussion and raised additional questions in both the media and research / engineering community.

In particular, I agree with Dmitri Alperovitch’s recent McAfee blog post that “This topic is highly technical and very difficult to explain to people not fully immersed into the BGP routing jargon… this incident underscores the very serious problems that exist on the Internet due to the system of trust…” It is also clear from my reading of articles and Dmitri’s blog that much of the press mischaracterized his initial estimate of the hijack impacting “15% of the Internet” — Dmitri was referring to routes and not traffic.

In his blog, Dmitri goes on to argue that any analysis of the incident should focus on the upstream ISP (AS23724) responsible for the hijack.

Both at the time of the incident in April and prior to my posting of this China hijack blog, I had private conversations with operations staff at several of AS23724′s upstreams, network operators around the world, collaborators in other security companies, and Arbor’s own resident engineers in the region. All of these private discussions reflect the sentiment espoused in public engineering forums that the China hijack had modest to minimal impact on Internet traffic volumes, including this RIPE statement, NANOG discussion thread and even the BGPMon blog at the heart of the controversy.

In the below graph, I chart traffic from 80 ATLAS providers around the world that terminates or transits AS4134 (the primary upstream to the Chinese company responsible for the BGP hijack). Traffic is shown as a weighted average percentage of all inter-domain traffic using the peak five minute daily value for the month of April 2010. As in my

The main take-away from the above graph is that ATLAS data shows no statistically significant increase for either AS4134 or AS23724. While we did observe modest changes in traffic volumes for carriers within China, the BGP hijack had limited impact on traffic volumes to or from the rest of the world.

As a couple readers of my blog observed (link to comments), traffic volumes provide an awkward measure of the security implications of a BGP hijack. In particular, the volume of hijacked traffic change depends on:

1. The scope of the hijack. Specifically, how far and well the routes propagated. In this case, the additional ASPath length meant few peers and upstreams preferred the AS23724 routes. In other words, the leak had limited scope.
2. Termination of the traffic. Did the China ISP (AS23724) drop hijacked packets or complete the connections? For example, in the former drop scenario, my laptop might just send 40 byte TCP Syn packets to an unresponsive destination. Since the TCP connection does not complete, my laptop will never send any significant volume of data traffic — China would only get lots and lots of Syns (and the rest of the world ICMP unreachables in exchange). UDP and ICMP, of course, are slightly different stories. On the other hand, if the traffic transits China or Chinese computers / VMs otherwise respond to the TCP requests, than significantly larger volumes of hijacked data traffic would flow from the rest of the world to China.
3. Objective of the hijack. Though some of the media have drawn cyber-war conclusions, we may likely never know if this was a misconfiguration, practice run, or intentional hijack. In any case, traffic volumes do not map well to the different possible security threats. For example, if the goal was to disrupt Internet communication and “blackhole” hijacked traffic, then we would expect to see a global decrease in Internet traffic and a large volume of Syn directed at China. However, the technical particulars of the April hijack were not particularly well-suited for this type of large-scale Internet disruption (see this article or an earlier blog post for examples on how to do this correctly).

   Alternatively, the intent could be a trial run exploring worm-like attacks against the global routing infrastructure. In this scenario, a small set of well-crafted malformed routing messages (hidden in a hijack of thousands of other routes) quickly propagates across the Internet crashing core routers and switches. Or something a little like this event in August (as a side note, Xiaowei did absolutely nothing wrong in her August experiment and is a really nice person to boot). I also note that man-in-the-middle attack, relatively low volumes of diverted traffic, and thousands of bogus routes announced as a smokescreen (credit for this scenario to my colleague Danny McPherson in a NYTimes interview). In other words, basically close to what we observed on April 15th.

   Or maybe, of course, this was just a typo in a configuration file.

As I observed in my earlier blog, inadvertent BGP route leaks and intentional hijacks have been part and parcel of Internet routing for the last twenty years. BGP hijacks happen all the time. The research and operations community have written hundreds of papers on the topic (including my own small contributions).

If I have not been clear up to this point, we have a problem. We need to address BGP security (as well as DNSSec, botnets, DDoS and other critical infrastructure threats) as quickly as possible. The Internet’s future may depend on it.

 
- Craig
 
 

While Google’s YouTube outage today generated a steady stream of tweets and blog posts, a quick look at traffic across 50 or so small / mid-size ISPs around the world suggests this was more of a “blip” than a global outage.

Certainly the outage was nowhere as large nor prolonged as the great “GoogleLapse” last year.

Below is a graph of traffic originating in Google (AS 15169) over the last 24 hours using data from 50 ISPs around the world selected at random. All times are EDT. Looks like a small outage overnight preceded the larger traffic 8am EDT drop-off.

And a quick aside, my intent is not to pick on Google (unless, of course, they do not pick Ann Arbor) — all providers have outages. I just find Google an especially interesting case study given their size and overall impact on the Internet.

Google’s recent FTTH announcement generated a wave of media coverage and industry discussion. Responses ranged from exuberant local communities racing to sign up to anti-competitive howls from incumbent carriers.

Industry pundits wondered what is Google up to? What will the search giant do with 1Gbps to the home? And more ominously, is Google getting too big?

While this blog post won’t explore the politics / strategy behind Google’s FTTH initiative (except to suggest Google should choose Ann Arbor), we will share some data on Google’s relative size and growth from a global Internet perspective.

Google is big.

And by “big”, I mean really big. If Google were an ISP, it would be the fastest growing and third largest global carrier. Only two other providers (both of whom carry significant volumes of Google transit) contribute more inter-domain traffic. But unlike most global carriers (i.e. the “tier1s”), Google’s backbone does not deliver traffic on behalf of millions of subscribers nor thousands of regional networks and large enterprises. Google’s infrastructure supports, well, only Google.

Based on anonymous data from 110 ISPs around the world, we estimate Google contributes somewhere between 6-10% of all Internet traffic globally as of the of summer of 2009.

The below graph shows the weighted average percentage of all Internet traffic contributed by Google ASNs between June 2007 and July 2009. Most of Google’s rapid growth comes after the acquisition of YouTube in 2007.

Before getting much further, a few words about what we’re measuring. Traffic volumes provide only the most indirect measure of a network’s size or popularity (for example, it takes tens of thousands of Tweets to match the bandwidth of a single HD video). Our anonymous data also does not include internal provider services (e.g. IPTV or VPN) nor data served from co-located caches within provider data centers. Rather, we’re measuring inter-domain traffic, i.e. the traffic between providers (the “inter” in “Internet”).

With all of the above said, inter-domain traffic volumes provide a key metric for understanding Internet topology and the evolution of Internet traffic patterns.

But even traffic volumes tell only part of the story.

The competition between Google, Microsoft, Yahoo and other large content players has long since moved beyond just who has the better videos or search. The competition for Internet dominance is now as much about infrastructure — raw data center computing power and about how efficiently (i.e. quickly and cheaply) you can deliver content to the consumer.

And here again, Google is at the head of the pack.

In 2007, Google used transit providers for the majority of their Internet traffic (including Level(3)). But over the last three years, Google both built out their global data center and content distribution capability as well as aggressively pursued direct interconnection with most consumer networks.

The graph below shows an estimate of the average percentage of Google traffic per month using direct interconnection (i.e. not using a transit provider). As before, this estimate is based on anonymous statistics from 110 providers. In 2007, Google required transit for the majority of their traffic. Today, most Google traffic (more than 60%) flows directly between Google and consumer networks.

But even building out millions of square feet of global data center space, turning up hundreds of peering sessions and co-locating at more than 60 public exchanges is not the end of the story.

Over the last year, Google deployed large numbers of Google Global Cache (GGC) servers within consumer networks around the world. Anecdotal discussions with providers, suggests more than half of all large consumer networks in North America and Europe now have a rack or more of GGC servers.

So, after billions of dollars of data center construction, acquisitions, and creation of a global backbone to deliver content to consumer networks, what’s next for Google?

Well, I’m hoping for delivery of content directly to the consumer via a nice, fat 1 Gbps FTTH pipe.

Google, please choose Ann Arbor.