The Bad Guys pwn your electricity

Even Insty's talking about this:

Cyberspies have penetrated the U.S. electrical grid and left behind software programs that could be used to disrupt the system, the Wall Street Journal reported on Wednesday.

The spies came from China, Russia and other countries, and were believed to be on a mission to navigate the U.S. electrical system and its controls, the newspaper said, citing current and former U.S. national security officials.

Both my long time readers (hi, mom and dad!) will remember me talking about this last September. It wasn't a surprise then, and shouldn't be now. There are some things to keep in mind:

1. There are friendly foreign countries; there are no friendly foreign Intelligence Agencies. If I were in charge of the Russian or Chinese Intelligence Agencies and my minions were not doing this, I'd fire them and get different minions. Or maybe shoot them. To anybody who thinks about the reliability of our power grid, this is where your planning starts from.

2. We have absolutely no idea what the power system control networks look like. The Russians and Chinese have a better idea than we do. The reason is that many different organizations own different parts of our grid, and they are motivated by cost, not security. It's cheaper for Joe Bloggs to plug a SCADA control server into Al Gore's Intarwebz so that he can manage it remotely. This reduces Joe's cost, so he gets a bonus for improving the company's bottom line. Unfortunately, the Russians and Chinese also manage it remotely. We will never win this battle with the respective motivations the way they are. To anybody who thinks about the reliability of our power grid, this is also where your planning starts from.

3. We have absolutely no idea what malicious software is running on servers controlling our grid. Once the Bad Guys pwn your system, the only way to recover is nuking the system from orbit. While we're flushing billions of dollars down the "stimulus" rathole, there won't be a penny to do this - and it would take billions more. Rebuilding secure SCADA systems to a known good state, and fixing the "let's just connect everything to the Internet" will take years, and will cause power outages all over the country while we do it. In other words, this will never happen. To anybody who thinks about the reliability of our power grid, this is what should really cause you to lose sleep at night.

4. The Power Grid is now no longer dependable in any meaningful sense. Instapundit makes a throwaway comment that isn't actually a throwaway: Maybe I should rethink buying that generator . . . . Actually, yes, this is precisely what you should do. Nobody can demonstrate that the grid is reliable to stated service levels. The secretary of the Department of Homeland Security should lose sleep over this.

Obligatory disclaimer: I work in the Internet Security field, for a company that makes products to demonstrate that security is working correctly. Before this, I worked at companies that made software to keep smart Bad Guys from taking over your computers. I am very, very pessimistic about this situation.

In the early days of the Internet (the 1980s) we told each other that we were trying to make a network that connected everything, everywhere. Lord, what have we done?

Mea culpa, mea maxima culpa.

UPDATE 8 April 2009 20:24: Slashdot has a good discussion, especially this:

Yes, fix the data security, but spend the money to make the needed improvements to physical security and redundant infrastructure. Our grid is routinely stretched to the breaking point. There's very little extra capacity. I think of people realized how vulnerable our electrical grid really is, they'd be terrified. The fact electricity is so reliable we take it for granted is testimony to the quality of the people working in the field.

Imagine living in L.A. or San Francisco with no electricity for a week.

I must say that it's touching to see so many technology geeks there who are so trusting of other country's Intelligence Agencies. RTWT, and despair. And this comment from an industry insider is worth reproducing in its entirety:

I am a control systems engineer, a member of ISA-99, and a contributor to several other standards on industrial control system cyber security.

The parent post is what SHOULD be done in a recently installed system. I can tell you from experience of dealing with other infrastructure (not the electric grid) that it isn't always that way. There were many systems installed around Y2k that are still in service. And most of you will remember that back then very few people took security seriously. Back then it was all about compatibility. Security wasn't even an issue. The big issue was SHARING the data.

Control systems and SCADA have long working lives ranging from ten to twenty years. The reason for this is because the field I/O validation cost is significant. It dwarfs the cost of the software, the control center, and all that lovely flashy stuff you're so used to seeing. Updating a configuration is very expensive, not just in validation costs, but also training costs, for miscellaneous costs such as review of operating procedures, control system narratives, and so forth. This is why many are forced to keep their systems isolated in the hope that by doing so, things will somehow stay secure.

But these days, that's no easy feat. Nearly every company has a contingent of data surfing desk jockeys with enough authority and enough dream-weaving synergy talk to push for interconnections. That's when things get very ugly.

The problem isn't that they want the data. The problem is that they want the data IN REAL TIME. Most of the time these idiots say the term though they do not understand the implications or even what it means. And that's how the exploits get started.

There are solutions. There are relatively secure methods for moving data in and out of a SCADA system. But they need careful review by people who know both the industrial side of things (to identify what is at risk) and the IT side of things (to know what the potential vectors could be). And the number of people with that kind of expertise is extremely small. We're talking about hundreds or maybe a thousand such people world-wide.

There simply aren't enough people to train the trainers who will train the trainers. And so, we're stuck with the status quo until we can build a community of cross trained people who understand industrial processes, control systems, and IT large enough to handle this situation.

I know many of you probably think you have it bad in the office IT business. And it is. Just know that there is far more truth in the Homer Simpson character than you'd ever dream of...

Mmmmm, exploits!