Monday, February 8, 2021

Connected to the Internet

 A lot of things are connected to the internet. It makes remote troubleshooting and maintenance possible. It also represents a security hole that can be patched and monitored, but never completely closed.

An unknown hacker entered the computer system that controls the water treatment plant in Oldsmar, Florida and raised the sodium hydroxide (lye) level from 100 ppm to 11,100 ppm. An operator on duty noticed the change and restored the correct setting. The FBI and the Secret Service are investigating. The system no longer has remote accessibility.

There are three priorities in computer systems and servers. They can conflict. 

1. Accessibility: The people that need access to the information and programs to perform their jobs need to be able to connect, access, and run programs with a minimum of difficulty. This is usually where admins are focused because when the analyst or the doctor can't get to what they want RIGHTNOWDAMMIT, the I.T. tech's boss get a call from his boss and things always roll downhill.

2. Redundancy: How is the data backed up? Where is the data backed up? How frequently is the data backed up? In the event of a failure, how do you restore the data? Have you real world tested it by doing a restore? Is there a second copy stored off site in real time?

3. Security: How hard is it to get into the system? How are users managed? How is access managed? What sort of firewalls are in place? What sort of intrusion detection? What sort of change tracking system? Is it automated? How often is there a full security review? (There's more questions, but if you can answer these, you're doing pretty good.)

It would not surprise me to find out that the person who did this either works for the city or is close to someone who is. The weak link in system security is the users.


Jonathan H said...

I wonder if anyone was fired recently... former employees have done stuff like this before...

Kurt said...

In other words, the venerable CIA triad


IoIT shouldn't be that hard. Fronting the sites/equipment with a good firewall and using good remote access technologies (tunnels, either IPSec or SSL/TLS) with MFA should take care of over 90% of this. The other 4.999% will be keeping patches up to date for the firewalls and good account/credential hygiene.

It is, of course, never 100%. The old bromide still applies - security is a process, not a state that can be achieved.


Beans said...

I can see internet accessibility for reading of monitors, so people off-site who are supposed to be able to access for monitoring can monitor.

Of course, better would be to not have access from off-site at all.

Richard said...

@jonathon Any competent organization disusers employees as they are going out the door.

Toirdhealbheach Beucail said...

Remote access is quite common in some industries - and quite scorned in others for the very reasons listed here.

It depends on how sophisticated an organization is in its IT thinking as well. Some have "hardened" their assets for IP or security reasons, but I suspect many are one step above in the 1990's.

Roy said...

"IoIT shouldn't be that hard. Fronting the sites/equipment with a good firewall and using good remote access technologies (tunnels, either IPSec or SSL/TLS) with MFA should take care of over 90% of this."

Well, there you go, Kurt. Having worked in the tech industry for many years, I understood every word of that. For my mother, my sisters, most of my friends... That might has well been in swahili.

Jonathan H said...

@Richard... You used a critical word there - Competent. As discussed by others, there are LOTS of organizations out there that are NOT competent, especially when we're talking the intersection of utilities and government.

I have read of cases where the company DID disable an employees account when he was fired, but it turns out he knew the passwords of other employees s it was a moot point...

Richard said...

@jonathan H
Yep. And the competence problem shows up everywhere. I was a community college that was smart enough to run the financial aid routine before they ran the billing cycle. My daughter was at Stanford to which this was an alien concept. She was there 12 quarters and every single bill I got was wrong.

Kurt said...


Then it's a good thing that your relatives and friends don't work in jobs where their ignorance can kill folks.

They're all nice people, I'm sure but may the gods keep them away from the controls that matter.


Ted said...

Retired from a Large Water Supply and treatment system ( 1/3 of the state are customers ) The Treatment systems control network is totally "Air Gapped" from the rest of the world. There are no connections. The distribution system, parts of which are 150+ years old and covers areas measured in 100's of square miles is connected to the IOT but reports on valve position and meter readings, etc and is purely passive. Active changes still require personnel to manually turn cranks on critical items.

But..... Just because a system is air gapped doesn't make it secure from a bad actor. If someone wanted to do harm, it is easy enough, given the skills and access, to leave a "Timebomb" in the code that would activate months after their departure, and delete it self after executing. And delete it self from any backups after the scheduled execution time if the backups were reinstalled.