Old Windows, New Risks: How End-of-Life Systems Threaten OT Security and What You Can Do

Aaron Crow (0:0.834): Hey everyone, thank you for joining me. Another episode of protected all podcast. Want to talk today, you know, topic that is not new to OT. It's it's definitely not the first time I've spoken about it, but really want to double down on it this week talking about, you know, end of life systems. Windows 10 being, you know, end of life and how that's going to impact. IT may not have as much impact.

Aaron Crow (0:31.022): caught replications of that. But OT is right. So every control room, every substation, every refinery has one right. That that old Windows seven box, that Windows XP box, that's when that Windows now Windows 10 box that's running a process that nobody touches, right? You don't want to touch it because it's not broken. It's sitting in the corner. It's doing its thing, right? But what happens when that if it ain't if it ain't broke, you know, don't fix it mindset becomes.

Aaron Crow (1:0.722): a vulnerability. The problem in OT when you have those spaces is, you know, it's one thing if you have them, but you have to be able to protect them other ways. We don't patch all the time in OT, but you know, we have to understand them and make sure that we can mitigate those controls and those risks in other ways. So let's dive into it. You know, what does end of life actually mean? Obviously the operating system is going to work.

Aaron Crow (1:27.094): And that's part of the problem in a lot of these spaces is we're not just depending upon the operating system. Again, in IT, it's really easy. You upgrade the system, you know, especially now with stuff being in the cloud, there's less implications of the operating system being changed. You can use Mac, you can use Windows, you can use Linux, you can use iOS. There's all these things that you can do in the compatibility of the products, the compatibility of the

Aaron Crow (1:56.116): of the capabilities are still there because they're cross platform. It doesn't as much matter many times unless you're running it locally. But in OT a lot of times it really does. But end of life really means, you know, when Microsoft in support on a product, what does that mean? It really means that, you know, no more patches, updates, security fixes. They're going to stop like it stopped time. The land that time stops. Windows XP will still run.

Aaron Crow (2:24.792): Today you can install it and it still works, but there's no additional security vulnerability. Sometimes occasionally they'll come out with an additional one if there's a really, really bad one. But for the most part, just, they're not supporting it anymore. Systems like Windows 7, Server 2012, know, older XP builds, they still exist in critical infrastructure, but they're not patched. not, there is no patch for them, right? So they're running in the corner.

Aaron Crow (2:53.574): you know, and, and, and one of those, it ain't broke, don't fix it. But that, that, that is a very big risk, especially if you are not aware of it and you don't have other mitigations around those things to protect them. you know, next is, know, why, why is it still everywhere in OT? I talk about this a lot, but you you get legacy hardware. you know, maybe you're working with a particular controller or.

Aaron Crow (3:21.358): particular software package and that vendor it's locked into support with that system. And it doesn't support going to newer systems. It doesn't support Windows 11. It doesn't support, you know, server, you know, XYZ, whatever that looks like. So, you know, to upgrade that capability, you have to completely replace it. And there's a cost to there. There's a risk there.

Aaron Crow (3:46.482): Sometimes it's not just a cost. Sometimes it's a risk to the organization because we know this works. And you know, all of the rest of my stuff, and it's the only thing maybe that works with my system, my hardware. The newer version of this thing doesn't support my hardware. So I'm not just upgrading software. I'm not just upgrading an operating system. I'm also having to upgrade my endpoint, the thing that is actually controlling something.

Aaron Crow (4:10.966): and that, that gets into bigger things, especially you, you thinking about in like the nuclear space. you know, a lot of that stuff is because if I change things, I have to relicense. so I have to, if I have to change out equipment, physical equipment, then I'm going to have to, know, there's a whole bunch of additional red tape that I have to do to be able to do that. So it's easier for me to keep what's there and protect it in other ways, data, diodes and other things. Right. So, I, you see that a lot in a, a,

Aaron Crow (4:39.784): in a nuclear space, whereas in wastewater, you're just going to have older systems and they may or may not be actually protected. There's also a lot of downtime fears. Shutting down may mean losing millions. I've been trained on this. I've got custom applications that run on this and they won't work with new stuff. The developer or the internal person that built that thing is no longer with the company or they've passed or whatever. And then there's software dependencies on the gate of the PLCs.

Aaron Crow (5:7.810): You know, not really built for modern OS compatibility. But you know. That leads to the next thing of, you know, the real risk that known vulnerabilities don't die and you know what? Once that support ends, once they're no longer patching, you know the attackers know they can exploit a permanent unpatched hole. If you if you have a Windows XP box, you know that is a vulnerability you can never get rid of. It's just there.

Aaron Crow (5:38.327): You know, if you have stringent controls and you have your monitoring for that stuff and you have other mitigated controls that are blocking those things, that's great. But that risk will always, you can't plug that hole. That is a hole that is always going to be there. And if a bad actor gets in or knows that's there or finds it in the wild somehow, that is a risk that is going to be there. Shadow IT, air gap myths. Yeah, the air gap myth.

Aaron Crow (6:7.218): No system. I won't say no system. There's been very few systems that I've ever actually walked into and they're saying, yeah, we're air gapped that were actually air gapped. They usually have control, remote control, remote access. They have a back door. have connection. Maybe it's temporary, but there are connections. so air gap is, is a pretty big myth. And again, that's, that's There are exceptions to that rule. Again, I mentioned nuclear, you know, a lot of,

Aaron Crow (6:35.714): critical government systems, I would argue, are actually air gapped. But most OT systems, even the ones that think they are, usually are not really actually air gapped. that and add onto that many OT networks being flat, a flat network, all those vulnerabilities are an impact to the other systems because I now own a system in that network. And now that is my pivot point.

Aaron Crow (7:6.178): which is a lot of times what you see, know, a chain is only as strong as its weakest link. So having those systems in your space is really making that risk. So, you know, really isolating if you're going to have to have a windows XP or any system that is unpatchable, or even if I'm, if it's a new system and I'm not going to patch it, this just goes to show why you need to have segmented environments, zero trust environments. So, you know, isolate that system so it can't impact anything else and nothing else can impact it.

Aaron Crow (7:33.231): I don't want anybody else to be able to get to it. I don't want it to be able to get to anything else. So you're really reducing that risk, that attack pack, a vector and the attack path to that device itself. WannaCry's success due to unpatched one is XP and server 2003 systems. It's an example of why that's a problem. You look at a lot of things from...

Aaron Crow (8:2.018): vulnerabilities that come out and, you know, you get, ransomware that comes and spreads in your environment, like wildfire, you know, and it's going to be because a lot of that, yeah, you don't patch your system, but if you segmented your environment, it would produce your risk to a particular area instead of spreading across, know, you need those firewalls, you need those, those boundaries that, that restrict those things. you know, how do, how do attackers find these systems? You know, obviously the normal paths, you know, showdown scans and you know,

Aaron Crow (8:30.996): spearfishing and infected USBs and, you know, remote access tunnels and reverse tunnels, like all that type of stuff. But many times it's, you know, I walk into these places and, and engineers are trying to do their job. These operators, these, these people that they're just doing their job. They need remote access. They're working with a vendor. They don't have screw mode access to these environments. So they plug them into the internet, right? And it was up, it was only plugged in for 10 minutes, 15 minutes, an hour, two hours a weekend.

Aaron Crow (8:58.828): because they needed to get access. It's not like it's always connected, but it was connected during that time. that sometimes that's all it takes. Bad timing. know, somebody happened to be looking in that, in that area of the internet at the wrong time. And you happen to plug in one of these systems and boom, they got you now, right? Because you, you don't even know what happened. And now you plug it back in, you go away. And it's too late. know, supply chain and maintenance vendors also are a problem.

Aaron Crow (9:29.192): Even in protected areas like in nuclear power, I've seen multiple vendors bring things in and that's going through all of the right things. It's not like they do anything outside the standard or the process. That's scanning systems, all the things, but as we know with, you know, antivirus and blacklisting, we only can scan for the things that we know about. So zero days by definition are not going to be there. So if there's something on there that it's not looking for,

Aaron Crow (9:56.265): it's not going to find it. You can scan a USB and it says all good and you can scan it with multiple, you know, agents and signatures and all the things. And it says all good, all good, all good. And you can still get something through because they tweaked it enough or it's an, it's a new, new vulnerability, whatever that looks like. Right. I'm going to pause for a second.

Aaron Crow (10:35.469): I just spilled a drink so cut this out.

Aaron Crow (10:45.283): All right, let me get back to this now. on just a second.

Aaron Crow (10:52.845): Alright, where was I?

Aaron Crow (10:56.628): supply chain. All right.

Aaron Crow (11:0.943): it.

Aaron Crow (11:3.663): So yeah, mean, you know, supply chain is a big issue. Vendors walking things in even, you know, transient devices being plugged in, coming into your environment, bringing them in the door. Those are going to be risk. Anytime you have unpatchable systems, those risks are always going to be there. You know, the reality check though is, you know, replace versus isolate. Replacement costs can be huge.

Aaron Crow (11:32.247): replacing an entire control system or replacing entire, you know, subsystem within an environment. Cause again, it's not just replacing a windows box and reinstalling, right? It's, it's replacing an entire system, sometimes all the way down to the hardware components. And that can be, that could be super costly and not just an upgrade cost, not just a, okay, the system costs me, you know, X amount of dollars, but it's also the downtime, the, the,

Aaron Crow (11:55.983): training, the updating process and procedures, you know, there's just a bigger cost that goes into that than just, you know, the windows cost or buying a new computer cost. Like it's, it's well beyond that in a lot of these spaces. Um, but isolation, like if you can isolate, then you're not having to necessarily upgrade. And again, I'm not saying you should wait, leave a windows XP machine running into perpetuity in these environments, but if you're going to do it, at least isolated.

Aaron Crow (12:22.615): It is so much reducing your risk by just isolating that system and putting it on its own environment, putting it on its own network so that you can, and then monitoring it. Monitoring is key. know, isolate and monitor. Network segmentation, you know, application white listing, you know, virtualization even can mitigate a lot of this. Again, monitor, monitor, monitor, monitor. If I know something is at risk, then I should be monitoring for any time anything happens there.

Aaron Crow (12:52.019): and that doesn't mean, know, Hey, I put in a, you know, an RDP or block RDP rule in my firewall, then right underneath that is an any, any, that doesn't do anything. because then that can, especially if it's only port and protocol, and it's not an application aware firewall, because that's not actually doing anything. if you don't understand that, then you should really actually do some research on how, some of these attacks happen, and bypass.

Aaron Crow (13:20.399): you know, firewall rules and firewalls themselves. Let's see. You can wrap. Yeah. So go into virtualization. You know, you can really wrap an old, an old operating system within a virtual container with strict access controls. There's a lot of times we've had older, older systems that we've had to, because of this application or the software that runs on it is the only one that runs. only runs in this, you know, let's say, you know, 16 bit or

Aaron Crow (13:50.003): 32-bit environment has to run on Windows XP or whatever the thing is. I'm dating myself here, but you can run that in a virtual environment and everything around it is better. And then you're protecting against those vulnerabilities and only this application is running in that space. And then I'm really locking everything down with strict ACLs and monitoring that access. So there's a lot of ways around it, even if you have to keep it because of a requirement like that, right?

Aaron Crow (14:19.843): when you get into leadership, you know, dilemma around these things, whether you're the CISO or the plant manager or, you know, the control system engineer, whatever the role is, you know, the question that you're really asking, right, is do I risk production downtime now to upgrade, replace, you know, ripping all these things out or potentially a catastrophic failure later? Unfortunately, sometimes the risk is not truly understood.

Aaron Crow (14:50.101): so I feel like we accept the risk way too often than way more often than we should. And I think that is mainly because a lot of times we don't truly understand the risk, and really understand how big of a risk it is and how much it can impact the environment. and, and also not necessarily understanding potential alternatives to just ripping and replacing knowing that, going to windows 11 isn't your option. What are my other options? What are ways that I can reduce my risk?

Aaron Crow (15:18.531): without having to rip and replace and go to something that's, you know, X million dollars. And I don't have that in my budget this year, next year, maybe even the year after. Does that mean I do nothing? The answer should be no, you don't have to do nothing. you shouldn't do nothing. There's, there's a lot of things that you can't do between, you know, zero and, and, you know, millions, balancing, you know, your business risk, you know, your, your, demands of cyber insurance, insurance demands, compliance,

Aaron Crow (15:48.591): You know, the, the fortunate thing, I guess, in, some of these spaces is there are compliant requirements for patching and things like that. And, um, but with everything, like there's always workarounds, there's exceptions. Um, so, you know, don't let, because I've checked the compliance box, uh, make you feel a false sense of security that I've done enough to make sure that I'm protected. Um, sometimes compliance is just the bare minimum. Um, and even though NERC SIP and 6443 and CMMC and all the things are great.

Aaron Crow (16:17.199): great standards and great compliance frameworks, there are still ways around that and that does not necessarily make you secure. So just understand that. You know what needs to change? I know I sound like a fricking broken record, but you know, asset inventory, you need to understand your assets and so many don't truly mean what that means. Well, I've got a spreadsheet that has, you know, how many devices I have.

Aaron Crow (16:45.795): and what their IP address is and their MAC address. Man, that's a great start, perfectly good. But I should include other data. I need to know what is its business function, what is its criticality. I should understand the OS lifecycle. I should understand what version, firmwares, bios, like all that other type of other pertinent information that really truly gets down to the level of understanding my risk. This is why a couple of years ago,

Aaron Crow (17:12.863): S bombs came out and everybody was so, you know, on the, on the S bomb train. you don't hear about them as often anymore, at least at the, the, the end user level, the asset owner level. but they're, they're, they're super important to understand the risks to your environment. And you can't truly understand that by just having a, a device, you know, I've got, you know, one device it's called ABC one, two, three. and it's this IP address. Like that's not enough to really understand your environment. Like you need to be able to really get down to the level of.

Aaron Crow (17:42.443): of context of what these devices are, how they can impact you, and where your risks are. You know, the C-suite, the management, you know, operations, they all need to have an understanding of end-to-life systems and that end-to-life systems aren't, you know, really just, they're not just technical debt. They are potentially active threats to your environment and you need to treat them as such.

Aaron Crow (18:9.011): And again, that doesn't mean you have to rip them out. That doesn't mean that you have to shoot them and buy a brand new one every time, you know, Microsoft decides they're not going to support something, but you do need to understand that that timer started. And it's not like it happens overnight. Like this has been coming. So organizations could have been planning for this and maybe they have, many have, but have all of them. And do we have an answer for all of them? Unfortunately, the answer is no. So, you know, so start modernizing through, you know, thinking about

Aaron Crow (18:37.741): Like again, I know I talk about this a lot, but I know Idaho National Labs, they're cyber informed engineering. Those principles really tie into this, right? And really understanding the risk and what is the risk to my overall process? Not just a cyber thing, not just a pie in the sky, know, nation state and bad actors are coming after me. It really is truly a risk in your spaces and you need to understand them, right? So.

Aaron Crow (19:4.899): You know, go back, think about it going into this week. Call the action for everyone listening to this is a short episode headed to ICS Atlanta this week. So if you're around, come see me there. This should get released on Monday morning. So I will be in Atlanta for the ICS Atlanta Conference. Definitely come by and see us. But yeah, call the action this week. Think about your environment and, you know, find all your unsupported Windows systems and document them.

Aaron Crow (19:33.863): You know, that's like the first step of really reducing your risk before it's too late. Hopefully a lot of you guys have already done that. Guys and gals have already done that. And you have a path out, you're planning, you're putting strong boundaries and barricades around those systems if I can't replace them, or if I can't replace them in the short term, maybe I've got to, they're on my timeline to be replaced in the next year or whatever that looks like. But definitely think about that. How am I going to monitor them? How am going to make sure nothing bad happens to those systems?

Aaron Crow (20:2.079): I'm going to definitely make sure that those things don't connect to the Internet. I'm going to definitely make sure that those things have no insecure remote access going to them. Nobody's plugging in USB devices to them. Like all of that type of stuff really, really have an understanding of where my risks are and how do I mitigate those things and reduce the risk. It's never going to be zero, but how do I reduce the risk on those systems knowing that I can't just upgrade them? So until then again, hope to see you guys out in Atlanta.

Aaron Crow (20:28.379): If not, definitely reach out and let me know what you're doing for your Windows systems, that end of life systems, how are you protecting them, how are you looking at them, are you concerned about them, do you have a lot, do you have a little, all the things. So until next time, have a good one.

Transcript lightly edited for readability.