The Glider

Computer glitch screws with the signals on the TTC in Toronto this morning. Apparently the glitch was severe enough to bring the entire system to a grinding halt.

From The Globe and Mail:

“Rarely do we ever have to shut down the entire system,” Mr. Ross said. “But when we’re unable to move trains or see the trains, signals and switches that is a very significant safety concern.”

Mr. Ross said the transit agency is still trying to pinpoint what caused the computers at transit control to fail.

He said a similar computer malfunction prompted a five-minute shut-down on the Yonge-University-Spadina line earlier this week.

Knowing what little I do of their network there, I can only imagine.

Article Link

(Image used under CC from denmar)

Sometimes I just stare at a point in space while I try to wrap my head around something like this.

From Techworld:

Siemens confirmed Tuesday that one of its customers has been hit by a new worm designed to steal secrets from industrial control systems.

To date, the company has been notified of one attack, on a German manufacturer that Siemens declined to identify. “We were informed by one of our system integrators, who developed a project for a customer in process industries,” said Siemens Industry spokesman Wieland Simon in an email message. The company is trying to determine whether the attack caused damage, he said.

So how, might you ask, does the worm get access to the Siemens SCADA systems?

With a DEFAULT PASSWORD

To quote Denis Leary, “Make sure to get your whole head in front of the shotgun. Thanks for calling!”

Article Link

(Image used under CC from hans.gerwitz)

Since we’ve written about Google’s multiple past outages (e.g., the GoogleLapes of May 2009 and the more recent Google Blip), it seems only fair to quickly cover Facebook’s problems last Friday.

The below graph shows coarse grain Facebook (ASN 32934) traffic statistics from 60 randomly selected ISPs around the world. While most press / blog coverage (e.g. Gigaom’s “Facebook Sees Major Outage”) pegged the disruption at 5:30 pm ET, the traffic data suggest Facebook’s problems began much earlier in the day.

Normally, Facebook’s diurnal traffic follows the same pattern as other social media and interactive consumer sites. Generally, Facebook traffic reaches a low over night at 2am and then grows to its daily peak at 5pm EDT before declining briefly before a second smaller peak at 9pm ET (the peaks likely matching the North American end of work day and prime time across PDT and EDT).

But beginning Friday morning at 2am, Facebook saw dozens of modest traffic drops (each of a few Gigabits) until plumitting 30 Gbps at 5pm EDT for roughly twenty minutes.

What happened to Facebook?

While there is no shortage of speculation on Twitter and operations mailing lists, Facebook so far is not saying. I think a recent post to an engineering outage discussion list sums up the situation:

“Given Facebook’s complexity, who knows what the problem was. Load balancer or layer 7 filter/re-writer (think F5) issues? Back-end server problems? Software misconfiguration? … Some developer deciding to just roll something out in the middle of the day (as is quite common with social networking sites these days)? We’ll probably never know.”

Facebook has come a long way from a few hundred Harvard freshman looking for dates. As Facebook accelerates past 400 million users and pursues goals of nothing short of taking over the web, the social media giant has become critical infrastructure — at least from the perspective of millions of consumers and ISP support desks.

In an upcoming series of blogs, we’ll explore the growing Internet infrastructure footprint of Facebook, Google and other dominant Internet content companies.