Conficker Still Exists? The Hidden OT Malware Threat Putting Critical Infrastructure at Risk

Aaron Crow (0:1.454) Thank you everybody for taking me another time and joining me on the Protect It All podcast. I have a return guest, which I always love having those conversations. And and Leslie and I, we actually just hung out together at at DEF CON. well, I was at DEF CON in Singapore and you were there doing actual work. but we we were able to see each other on another continent for me at least. So thank you for your time here today and being patient with me, rescheduling and technical difficulties.

Aaron Crow (0:28.440) So Leslie, thank you. and why don't you introduce yourself for those who don't know you and a little bit about your background.

Lesley Carhart (0:34.934) Yeah, no worries, man. my hi everyone. My name is Leslie Carhartt. I am an OT or industrial incident responder. I've been doing that for about 18 years. I currently work for a company called Dragos. I am based in Melbourne, Australia, where I've been working for for some time now. And yeah, I'm I'm one of that small number of people who responds to like hacking and does forensics on power plants and trains and things when they get hacked into.

Aaron Crow (1:1.454) Which is all always fun. Obviously, it's been where I've spent a lot of my career as well. and and it's always, you know, before we started talking about this, we you talked about something we'll get into later. But it's funny how a lot of the technology that we're dealing with, sometimes it's 20, 30 years old. so it has a lot of those same problems. So we have all the new problems that are out with all the new technology and the latest version of things, but we also have the things that have been out for 20 years and everything in between. So it's it's a constant, never boring.

Aaron Crow (1:32.331) never ending level of issues that we're dealing with in these various spaces to keep things running and making sure that your lights turn on and and and the and the trains work and and all the things don't go boom that aren't supposed to go boom.

Lesley Carhart (1:46.998) Yeah, we we don't get bored, that's for sure. It's a challenging space because you have to cover so much ground, so many areas of technology and so many different verticals and devices and pro protocols and things like that. So we definitely don't get bored. we kind of worry about ever being able to retire, but we never get bored.

Aaron Crow (2:5.678) That's very true. That's very true. Well, I mean, last time I talked with you, you were in a com in a different role, I think. And you were obviously you lived in a different place and things are different. So, I mean, all of those things are so big in moving, living in a different continent, a different role, learning different people. Like you all those changes are hard. People don't deal with those things well. We deal with that in our work lives, but all of the time we don't always think about how

Aaron Crow (2:34.222) big a transition of a of moving and changing jobs and changing roles and how all that can impact everything that we do and the friends that we have and the people that we know and you know just where we get our coffee. I mean all those things are valid and huge and and big, especially like I said, the big, big changes you did. So how have you transitioned all those things and done all that while maintaining job and work and and social life and all that type of stuff as well?

Lesley Carhart (3:0.577) I say immigration is pretty much one of the hardest things you can do in the world. It's one of the most stressful, complicated, challenging things that you can commit yourself to. I actually did a talk a a while ago about skills that translate between incident response and immigrating, because a lot of them overlap from having meticulous timelines and plans and documentation and being able to find things to being able to do crisis management and deal with people and

Aaron Crow (3:19.885) Mm-hmm.

Lesley Carhart (3:30.328) You have to do all those things when you're moving countries. And it's turned out all right for me, but it was really it was really hard. It's it's not something to take on lightly. A lot of people are like, yeah, I'll just move to another country. It's very different being a tourist and a guest for a few months and then picking up your life, your whole career and your your everything you own and and moving somewhere else. So very happy I did it. Very, very, very grateful to Australia for letting me live here and stay here.

Aaron Crow (3:37.476) Yeah.

Lesley Carhart (3:56.610) hopefully for the rest of my life. But beyond that, yeah, it's been a it's been a challenging couple of years. People might have noticed that I've been a little quieter than usual on social media and stuff because I've been fantastically busy between my job and incident response and how things are really blowing up in OT and and also starting my life over at 42 years old.

Aaron Crow (4:20.932) Which is no small task and any of those things, it just isolated any one of them, much less all of them on top of each other, while traveling and working and and and all the things as well. So like I said, we we we met up in in Singapore. So the fact that you were on another continent as well, doing work there while while we were there. So all these all these things to say, like all that led to, you know, w what we talked about before and and and you know, what are you doing now? Like you just talked about this big thing that you're

Aaron Crow (4:50.734) you're working on and and all the the the the the history of OT and all the things that we think that hey this will be a thing that I won't have to worry about again. And then years down the road you're we're still seeing the things come up and pop up and they're still relevant at at a large scale in in our industry, unfortunately.

Lesley Carhart (5:10.072) So you really have to have fantastic computer science and computer support foundations to do well in OT cybersecurity. It's not all about process devices and how industrial technologies work. It's also about the stuff is really old and sometimes it's older than you or older than your career and

Lesley Carhart (5:27.640) There's less and less of us over time, I think, who know how to use computers from the aughts and from the 90s and from the 80s. And when they're still running society, that's a a really big challenge. And it makes it very hard to hire people. It makes it very hard to expand operations. And yeah, my my most recent project, I I've been I've been working on a white paper.

Lesley Carhart (5:49.005) Because we've started getting so many calls on a very unique issue globally from customers, prospective customers. And the the issue is they're finding really old, like 10, 20-year-old infections, worms, and commodity malware infections in their industrial environments. For the people out there who are just general cybersecurity people listening to your like, well, that's stupid. Why didn't they clean it up? But you've got to understand these are like Windows XP and older computers, and there's no antivirus that supports them anymore.

Lesley Carhart (6:19.062) And there's no XDR, EDR. The computers are starting to fail for other hardware and software reasons. So the instability added by having a bunch of you know 50, 70 pieces of malware running in the network starts to make things fail. And sometimes really important critical infrastructure things fail. And the problem is you've got no anti-malware. You have

Lesley Carhart (6:42.592) a very limited tool set of old forensics tools to work on these computers. Detection is mostly passive, network-based. So you try to clean it up. You try to start removing these pieces malware and it's like whack-a-mole. So you see these these massive, very cybersecurity, mature multinational companies start trying to clean up these infections and they'll they'll clean up one for a few weeks and then somebody will plug something in or a computer that was powered off will turn on and they'll all get infected again.

Lesley Carhart (7:8.270) And problem starts over and they have to do they have to commit to like operational downtime and and a massive human fronted cleanup effort again and again and again. And it becomes incredibly risky and costly. So the the decision

Aaron Crow (7:8.900) Mm-hmm.

Lesley Carhart (7:23.712) matrix for for what to do in these cases becomes incredibly different because of those limitations and operational risk. Like there's some cases where you have to make a risk decision to leave some of the infection in place. And that's really hard for traditional cybersecurity people to get their head around. So yeah, 16,000 word paper that I'll I'll be I'll be dropping soon through Dracos and doing some some talks on around the world to discuss academically how we deal with this problem.

Lesley Carhart (7:53.857) in systems that can't be replaced or upgraded of getting rid of this commodity junk malware that's starting to cause more and more problems and and also from a regul regulatory perspective, there's organizations that have to clean it up within a certain timeframe. So it's that challenging. It's it's six it's 16,000 words challenging.

Aaron Crow (8:16.495) And it it probably only scratches the surface of what you wanted to say and and had to narrow it down to to to 16,000. And and for folks if if if you're not old enough or don't have enough gray in your beard like I do, that that's something that was a around for like Windows Vista and and service packs, early service packs. And we're talking really, really old stuff. To your point, Leslie, it's like,

Aaron Crow (8:43.082) How do you how do you manage these environments? We know these environments. Every every site I go to has one of these. It has a not necessarily conflictor, but you know, it it it has a an old box that's in the corner that everybody says just don't touch it, leave it alone. It's running this old system that it's really critical. So we're just gonna keep it, we're gonna isolate it, we're gonna segment it, whatever the things are. But sometimes those things get out. And it can be an old PLC, it can be an old Windows machine. I remember in a power plant there was

Aaron Crow (9:12.078) There was a particular system that they literally had like caution tape around it because if you touched or you leaned on the cake the cabinet, it could shut it down and it was it was vibration monitoring for the turbine. Big deal, it can actually trip the unit, right? It can trip the turbine. So they they were really intentional. Instead of fixing the problem because they really couldn't, they they just put caution tape around it and told you not to do it. Like that's some of the times that the way that we solve these.

Aaron Crow (9:40.590) problems in OT is it it makes an IT person or a co a a a more current cybersecurity person, it makes them want to pull their hair out. Like what do you mean? Like why do you let that happen?

Lesley Carhart (9:53.325) And beyond the hair pulling out, it makes some jump to do things that can cause worse things to happen in those environments. You know, you run in, you do incident response strong, you install forensics tools, you try to install XDR, you pull computers offline, you you could be killing people. That's that's a situation where you're it the best case may be impacting process operations at millions of dollars, but

Aaron Crow (9:57.957) Sure. Yeah.

Lesley Carhart (10:14.006) you know, in a worst case you're actually hurting a human being or multiple human beings. So yeah, it's it's challenging. And I had to actually as I was writing the foreword on I was like, I have to say this isn't a joke. I worked really hard on this, this isn't a joke. Like it's it's really very, very challenging to make those decisions and a lot of it is on the the risk crit

Aaron Crow (10:25.996) Okay.

Lesley Carhart (10:33.228) management criteria that go into these choices and what you can and can't hand wave away, you know, putting the the caution tape on things and and then what tools are available to even deal with these problems. And for the people who weren't working in cybersecurity in 2008, what these pieces of malware do and what they are and why they were meaningful at the time, because a lot of the young people weren't there for them. So and we're seeing this like atrophy of skills. Like the whole cybersecurity job market is really saturated right now. It's terrible time to get into the field. Young people

Lesley Carhart (11:3.222) Are not able to find jobs for years after they graduate, but legacy, mainframes, banking, OT, we can't find anyone because people are graduating learning modern tools and they don't learn like old school computer foundations, and we can't teach those in a pipeline. We can teach tools, we can teach pro procedures, we can't teach you to understand how like platter hard drives work or how

Lesley Carhart (11:27.254) you know, old school file systems work and i if you don't have a a substantial understanding of those old systems, we're up the creek and and you're like, we don't I think Erin, do you like to retire someday? I'd like to retire someday. So this kinda worries me. And we're starting to run into these systems and it's that Erin was just saying, you know, like they're out of support, don't touch them. Nobody knows how to fix them. And sometimes it's like the person who installs them is retired or they're dead. So

Aaron Crow (11:40.951) Yeah.

Aaron Crow (11:50.883) Yep. Yeah. And and there's no documentation. And even if there is, it was written fifty years ago and it it's been, you know, adjusted or or customized since then. It's not in the same place that it was. Right. You know, it but these are the things that there's a critical system and and and like I said, it's it's the the it's controlling the turbine or you know, some kind of monitoring, something that is critical and can cause an outage to your point. And lives can get lost, right? There's

Aaron Crow (12:20.046) We're we're dealing with real things here. I I I laughingly, I I remember doing in a set when I was an asset owner, we had a power plant and we found that there was a there was stuxnet running in our in our power plant, right? And I left it because we weren't in outage and it wasn't a semen system.

Aaron Crow (12:45.708) It was in a DCS, wasn't Siemens, there was no Siemens in the system whatsoever. So I left it. Now I planned to come back, and I did come back eventually and clean it out. But in that moment, in the middle of the day, the plant was running. It was in the middle of the summer in Texas at peak time. I was not going to bring down the control system and the cis in the plant to try to start chasing my tail to take that thing out of there when it was doing no harm. Right.

Lesley Carhart (13:9.974) It's textile isn't easy to remove across the network. Yeah. Anything that's wormable, yeah, it's it's it's the the the youth the youths don't know. The youths don't know now that like a lot of ransomware isn't wormable, it's human deployed, like and that's bad enough to clean up. Ransomware is rough enough, but these these old worms, they they spread real fast. They spread they spread by themselves and they spread really fast and they they spread off any media that they can possibly get their hands on and it's

Aaron Crow (13:30.191) Yeah.

Lesley Carhart (13:37.100) Yes, it they're just painful to clean up with like real without real time antivirus, it's a challenge.

Aaron Crow (13:44.409) So I I did another incident where where we were at a let's just say a big place where people travel and very similar type thing, right? Where where it was chasing the tail. And that's another th this comes back to I'd love for you to share a little bit on on the process side of things because that's so important and a lot of people don't real or don't necessarily understand if that especially if they haven't done it before, why.

Aaron Crow (14:12.172) Yes, you can clean out. Yes, there's tools to take things off. Yes, there's a there's a process you can take a fictor off and others off, right? But it's not that. It's like you you you knock one thing down and it pops back up over here. And and there's a process to be able to do that and do it safely to segment things off. Talk a little bit about that and like why is it so hard, especially these older devices, older, you know, kind of risks in these older environments, and and how

Aaron Crow (14:39.136) it spreads so fast and you don't know that you've cleaned it or not and how do you know you're good and and all those that again if you've not experienced that it it it it it's so frustrating and so time consuming.

Lesley Carhart (14:50.988) Yeah, I mean it's it's restarting a whole cleanup effort. And again, you know, you've got to shift your thinking. You have maybe a few hours a quarter of maintenance downtime to do this effort, to do the cleanup. And then you don't have like an EDR you can install on these old systems. They're usually only run back to maybe XP service back to the the ones that are backwards compatible, the ones that are legacy compatible, and that's in limited functionality. So antivirus, you know, you might have the old

Aaron Crow (15:12.420) Break.

Lesley Carhart (15:17.954) DVDs, C D C Ds with installation media, but it's not gonna install correctly anymore. It's not gonna be able to connect to its servers, so that's usually not gonna work. So you're stuck with like something like Clamwin walking from computer to computer, cleaning up the computers manually, and then you miss one. You miss one that like is stuck in a cabinet in a ceiling somewhere, or somebody has it's a laptop and it's sitting on a shelf, or it's the the the USB drive that the vendor comes out with once a month and then it's all infected again.

Lesley Carhart (15:46.855) And there there's there's solutions there. Like, you know, it's it's it's thinking strategically about you know controlling the spread of the malware. So sometimes there's things you can disable network-wise or housework-wise in terms of vulnerable protocols, lateral movement vectors, there's things you can do about removable media in some cases, either hardware or policy controls. So it's kind of thinking about that that segmentation and isolation of the computers and then making a plan to clean them up in a way that they won't immediately be re-infected.

Lesley Carhart (16:16.758) If a computer comes online with that infection. So yeah, I mean, I've kind of broken it down into three solutions. And the first one is you're doing an immediate cleanup. You're investing however much disruption time and money it takes to go in there for whatever reason you think it's going to cause a problem, and it can. If it's starting to like fill up the memory on your embedded computers, we've seen that bring down operations, especially things for like chillers, like building automation systems.

Aaron Crow (16:39.844) Yeah.

Lesley Carhart (16:42.530) those those embedded old Windows computers start going, weird things start happening to your facility. So you know, immediate impact, immediate re regulatory requirement, then you're spending a massive amount of money and disruption and and things like that to do an immediate cleanup. But that's a rare situation and it's incredibly dangerous and expensive. So, you know, do being able to do a a full system upgrade with new Windows systems and stuff is a a massive expense.

Lesley Carhart (17:8.926) the second option is is exactly what you were talking about, Erin, is you you make you make a plan. You say, I can't do it this moment, but I'm gonna make a a sensible plan for exactly how I'm gonna get everything on the first go. I'm going to prevent it from spreading again as much as I can. I know the systems I can and can't replace or or clean up, and I'm gonna deal with those in discrete ways. And I will execute that plan when it's an appropriate safe time and I feel all confident in it. And then the final one is is the deferral of like

Lesley Carhart (17:38.273) You know, we have done a risk assessment and I give the risk assessment criteria that are like life safety, your ability to control the infection, the ability of the infection to potentially cause those life safety impacts, and the the risk of you doing whatever you're planning to do. And in some of those cases you'll say, there's really nothing that we can do about this infection until the next major system replacement.

Lesley Carhart (18:1.630) And I have accepted the risk. I'm going to put it in writing. I'm going to talk to senior leaders and stakeholders about that so they understand the risk. And we are not going to say we're going to ignore this till the end of time, until we retire, and we it's not a problem anymore. But you know, until there's the next 30-year major system upgrade, we really can't do anything about this. So that can change over time. It's something to revisit. But those are kind of the states, the conditions you can end up in with this malware. And it's like, it's just so wild. Like I mean, like could

Lesley Carhart (18:31.504) Ticker was like early in my career, and I would never would have imagined. I would have gone into something else. I would have gone into interpretive dance or something if I had known in 2026 I'd I'd be spending major time and academic study writing formal papers on how to deal with it. But that's where we're at in OT. And it's not because it's new, the infections have been there. It's because people are starting to build their maturity. They're starting to see what's going on in those environments and see how messed up they are, which is good. I'm happy about that. I'm happy about people knowing there's a problem.

Lesley Carhart (19:1.474) and starting to address it because this malware can break things. It certainly absolutely can cause more vulnerabilities. It can be a vector for advanced adversaries to get in. It can cause systems to fail. So it's good that people are starting to know that it exists, but it's challenging because it's so embedded in these environments and there's next to no tools to deal with it.

Aaron Crow (19:20.184) Yeah. Yeah. It's it's it's insane the amount. And and I have these intense conversations at conferences a lot of times. As you know, I I I'm with the ICS Village a lot and a lot of people come up and and I and I get all the people from the IT side and the and recent cyber folks or or maybe they spent their entire career in cyber and and and sometimes it's really hard for them to grasp the concepts of why everything you just said is true.

Aaron Crow (19:47.867) Because then they're just like, well, just rip it out. Just replace it with new stuff. Like why it doesn't make sense to them. And and you you also said something that's that's really important is a lot of these systems are 30-year replacement cycles, right? They they don't upgrade, you know. W if you had a Windows XP machine and you walked into Microsoft and you tried to plug it in, they'd just say, No, here, here's a new laptop. We'll throw that one away. Here's a brand new one, it's not a big deal. There we go. Right. But we can't do that in these OT spaces.

Aaron Crow (20:18.010) it's it's extremely expensive. And sometimes the the other the other irony of that is sometimes the new system is less reliable than the old system with the virus on it. And and and when when we care more about availability and safety than we do, you know, confidentiality, and and and it it drives your decisions and how I respond and the actions that I take.

Aaron Crow (20:47.372) As a business owner, not as a cyber person, but as a business owner, as an asset owner, I'm looking at things differently because this new asset comes in and it doesn't have the same reliability. Like this thing has been running for 30 years. And this new one, I'm not sure it will. And so there's hesitation in around those things. especially if they if they had a bad incident where one failed, the new thing came in to try to replace it and it didn't work and it wasn't as reliable.

Lesley Carhart (21:15.993) Yeah. So I I I I I think this is an interesting podcast. I think this is gonna be the only podcast on Configure. Maybe maybe this week, maybe this month, maybe, maybe this year. So I I hope that all your listeners are are really enjoying learning about learning about how old malware is meaningful and how how difficult it is to find people to clean it up and tools to clean it up.

Lesley Carhart (21:41.264) That'd be a good project for a a student who's like looking for like a lab project. Build a really good remediation tool for like commodity malware that's low touch and runs on like Windows at least XP SP1, if not Windows 2000. That'd be that'd be really cool. yeah, it would have to be low touch. It would have to be like not no installation required. yeah, build a little

Aaron Crow (21:45.037) Yeah.

Lesley Carhart (22:11.247) Build build a little remediation tool for that that's better than Quamwin. That that'd be a good one.

Aaron Crow (22:17.966) That that's something really interesting that to think about, right? Is as we have this, this problem is not going to go away either. a lot of these systems are gonna continue to be there. You know, we're talking 30 years, it's gonna be 50 years, and there's gonna be things that that are out there that are still running. Even the new stuff that we're putting in today, they're gonna be there in 30 years. So we're gonna have some other similar issue with these because then yeah. No.

Lesley Carhart (22:40.585) yeah, I'm not gonna see the end of Windows Pen when I retire. No, no.

Aaron Crow (22:45.914) So this is going to be a constant cycle, you know, repetitive problem that we're going to have, you know, in the in the nuclear environment because of the regulations with the you know with the approval processes to get hardware in things like that. They can't upgrade at all because they're licensed on what they what they've built. So the vendors were really smart in that they started buying back like control system HMIs and PLCs and different things.

Aaron Crow (23:13.966) That were being end of life from a normal, like a manufacturing facility or whatever. They brought them, they bought them back from those customers and recertified them and kept them in their warehouse because they knew if the one at the nuclear power plant failed, they couldn't upgrade it to a new version because they'd have to relicense it. So then they take the old one that they've cleaned up and, you know, put a new label on it and they sell it to them for three times the cost. But that's the only way they can maintain and replace that thing.

Aaron Crow (23:41.400) without a whole bunch of engineering and licensing and regulation and all that kind of stuff. And that doesn't help at three o'clock in the morning on a Sunday when the thing stopped working.

Lesley Carhart (23:52.420) Yeah. It's a it's a interesting challenge. OT is a fascinating space and a lot of what we talk about is like the really major incidents. We talk about, you know, Trisis and Frosty Group and things like that, pipe dream and like the really like cutting edge state adversary stuff. And

Lesley Carhart (24:9.881) What else do we talk about? We talk about like PLCs and hacking PLCs and protocols and modbus and things like that. But a lot of what we do is such janitor work and it's so important. And there's just like nobody like we've had positions open for I've got one open that now that's been open for seven months, like job rec. And I mean we're not like asking for the moon, but I guess we are asking for a unicorn because all we want is like

Lesley Carhart (24:36.889) Forensics people who know how to do forensics on the computers in like industrial environments. And like and I was telling my boss when we put the position up, I'm like, I'm in a group chat with like a hundred people that do that. And it's almost everybody. Like that's it's it's real bad. Like it's it's real, really bad. There might be three people in Australia who do this full time as a dedicated job. So yeah, something to think about.

Lesley Carhart (25:2.635) It's not just all about like, my god, I'm gonna buy a PLC on eBay and I'm gonna hack it. It's a PLC, you can hack it. You can exploit the PLC, I promise. It's more like these these are the problems that are really hard to solve. Like I have 500 computers infected with a random worm from 2010, and there's no antivirusilla run on them. And we it's there's no like PowerShell on these systems, there's no domain.

Aaron Crow (25:8.802) Mm-hmm. Right.

Lesley Carhart (25:32.048) How am I gonna clean them all up? And how am I gonna keep them from like getting immediately reinfected? And what's the risk implications of this infection being in every one of these computers and communicating over like the old network devices and eating up a bunch of bandwidth? And what happens if it fills up memory on like the crown jewel system that's like embedded and has like two gigs of RAM? Like what what happens then? So those are those are the real challenges of working in OT. It's this duct tape.

Lesley Carhart (26:1.611) solution for keeping these old critical infrastructure systems safe and operational and reliable and it never it never stops. Like there's always a new interesting challenge like that. And I can I can write 16,000 words about like a few pieces of malware, a few cases in here and a few tools. And like that's a tiny, tiny piece of the puzzle.

Aaron Crow (26:24.026) Sure. Well, and and as you're saying that with the the the skills gap that we have in the OT side, to your point, because we're aging out, because there's just not enough folks, it gets me back to thinking about the whole, you know, the way we used to do things, the way we do it in other industries. Think about like electricians or or whatever, the whole the whole mentorship side of things where you bring somebody in, of course they don't have the skills. There's no way they could have the skills.

Aaron Crow (26:50.808) They weren't alive when that stuff was out, which is okay, it's not their fault. But how do we train them on those things? Because to your point, I'd like to retire someday. I know you'd like to retire someday. We've got to have somebody that can pick up the baton and run with it. and we're running out of time for that to happen, to have enough people that can actually understand that I actually do understand it to be able to mentor and train the next step of folks that can do those things, mainframe and

Aaron Crow (27:18.958) You know, Windows 311 and NT three three five one and NT four and how those you know directory services worked and how you know NetBIOS worked and like how all of those like some of those things are still embedded into Windows today because of just the way that Windows bolts things on as they've as they've gross. But some of those things have been deprecated a long time and they don't work the same way that they used to. So you think you've heard the term.

Aaron Crow (27:43.853) And you think it works the same way that it used to 20 years ago, but it doesn't because it changed in like NT40 to three to Windows, you know, 2000, it drastically changed the way that they did directory services and NetBIOS and TCP and how the stack worked and

Aaron Crow (28:1.292) All of those things work differently. So if you're attacking it from the current way, you're you're not gonna know how it actually connects, which doesn't need the TCB stack to work at all. And it can do it across NetBIOS and layer two and all these other ways that it is able to communicate because before we had IP. Exactly.

Lesley Carhart (28:14.591) Encapsulated protocols. So yeah. Custom protocols encapsulated inside familiar protocols. Like it it's it was Mayhem for a while. People were just like doing whatever. And we still see new ones every week. And we've got huge catalogs of them with, you know, 600 people working dedicated NOT and we keep finding more stuff. It's it's pretty, pretty crazy out there. And it's it's yeah, it's not gonna change. It's it's

Aaron Crow (28:39.822) No.

Lesley Carhart (28:39.939) The landscape's changing. There's more like homogenization and vendor deployments, but that has upsides and downsides. So it's a it's a fascinating space. And yeah, in terms of like mentorship, we we definitely need to to keep trying to incentivize young people to be interested in the space. But I do have a focus like in a plea to like academics and and university curriculum developers and college teachers, things like that. It's really focused on foundations because again.

Lesley Carhart (29:9.003) I can mentor, I can teach tools, individual tasks, things like that. But like young people who are coming out only knowing how to use like XDR, like

Lesley Carhart (29:18.019) And they don't understand how a hard drive works. They don't understand how memory is written. They don't understand like how a packet works. Like that's way harder for us to work with. It's not for any like gatekeeping or not wanting young people in the field. It's like there's like old computers out there that are not AI, next gen cloud, and they're doing really important stuff. And if people don't have good like computer basics, computer foundations, it makes it so challenging for mentors and you know, certification instructors and and people who are trying to teach.

Lesley Carhart (29:47.946) them on the the far end to to get them where they need to be to take over these roles because yeah I want to retire someday like I I you know like start my little like start little my little bar on a beach in Australia like that that'd be all right but I can't do that unless there's some young people who want to do this stuff and are willing to learn old computers and think through these complicated problems that are not an easy thing that like an LLM can solve for you.

Aaron Crow (30:5.528) Ha ha ha.

Aaron Crow (30:14.646) No, there was no LLM, there was no none of that. Like I we were we were installing operating systems with, you know, thirty five floppy disks and and going one at a time. And if you skipped one you had to start over.

Lesley Carhart (30:22.967) Yeah. Do you know what a floppy disk is? Do you know how to put a floppy disk in a computer? 'cause you're gonna need to you're gonna need to do that. yeah. I mean at least how to burn a C D I do that like every week, like burning C Ds. So yeah, yeah, and yeah.

Aaron Crow (30:28.664) Yeah.

Aaron Crow (30:32.536) Yeah.

Aaron Crow (30:38.756) Mm-hmm. Well, you know, it's it it's crazy that if we think about it, again, you know, a lot of this technology is so old, but the you know, it it it it runs, it works. Again, there was an incident that I had at a at a nuclear power plant where there was very similar to what you're talking about, right? And it was we we noticed that there was an issue. We

Aaron Crow (31:6.104) Because the PLC or the div the the device it was a it was a crane and it was a the controller in the crane.

Lesley Carhart (31:10.147) Yeah. it's always craned. It's always craned.

Aaron Crow (31:14.656) And and and we we cleaned it up and by cleaning it up, we just rolled it back to a known good state, which is usually what you do in OT. The first, especially 15 years ago, you weren't thinking cyber or or or or malware or anything. So you just roll it back to Alaska, known good state, test it out, everything's working good, you're good. The next refueling outage, it it starts doing the same thing right after the outage. And we're like, okay, there's something here.

Aaron Crow (31:43.193) Like something's going on. So then we started diving deeper in. What happened? Well, the last thing that happened is the vendor had come in to do whatever maintenance on the system and plugged in their laptop. Now they'd gone through the scans and done all the things they were supposed to do, but what obviously that's a that's you know a a specific blacklist looking for certain items, and it didn't detect it, right? So they they basically the vendor brought in malicious code that was being he was putting it on.

Aaron Crow (32:12.920) the the the the crane and it caused the problem and and we were doing all the right things but it just goes to show like y you can't depend on the tool to be able to solve the answer. You have to be able to troubleshoot it and understand it and and and work around. And that's that's that's a zero day. That's you know mythos that they claim can do what all the things that you can do. But

Aaron Crow (32:36.724) We also have this old stuff that you could just bring bring things off the shelf and drop it in and it'll do just as bad a things on existing equipment.

Lesley Carhart (32:44.207) Yeah. And yeah, it's it's pretty wild. And yeah, it's always a it's always by the way, it's always a crane or an air conditioner for some reason. Like those are like why do I see so many cranes? So many cranes and so many HVAC systems. So many of those. I don't know why I think they get ignored more than a lot of other systems because they're kinda like just like over there. But they get more they get ignored more than like the PLCs and stuff. So like the typical manufacturing stuff or production stuff. So

Aaron Crow (32:51.374) Yeah. Thermostat, yeah.

Aaron Crow (32:57.241) Yeah.

Aaron Crow (33:3.514) Yep.

Lesley Carhart (33:10.685) what they consider the process networks are kind of tangential. But yeah, cranes and stupid HVAC systems over and over and over and over again.

Aaron Crow (33:19.640) Well, you know, and and the the hard part that I see with this when we talk about the the training, mentorship, etc., none of what we talked about is touted in in school and nobody's talking about it. It's not the sexy thing to go after. There's no degree path for it. You know, we're we're doing a lot of stuff with ICS Village and working with, you know, industry, you know, doing a OT class at Harvard. And like there's all sorts of really cool things that are happening.

Aaron Crow (33:48.911) But even those, the focus is not what you're talking about, right? And that's I see a lot a bigger problem because the the thing I see and the thing I fear is that so many OT sites have told me for years, we've been running this for 40 years and never had a problem. We don't have any issues, we've never been attacked, we've never had an incident. No, you just don't know it. It's running there. And as soon as you start looking, which is how I found.

Aaron Crow (34:16.558) The things that I found and in the places that I found, it's not because I'm brilliant, it's just because I looked. and and other people weren't looking. And when you start looking, you start finding things, these things have probably been spread across these environments for decades and causing issues. And they were, they were re like, I remember going to a control room and they were having a problem. And we when we started monitoring with Splunk, we turned Splunk on and we saw this HMI, this operator workstation in the in the control room.

Aaron Crow (34:46.456) was rebooting every Saturday night at like the same time. We walk in, like I'm trying to get my team, we're looking at diagnosis. We don't see any issues on it. We go in the room. so I just have one of my guys and it's overnight. It's the overnight shift. So I I had one of my guys, I'm like, you're just gonna go sit in the control room and you're gonna watch this machine because it seems to do it between like two and two thirty every night of this Saturday night schedule. So when they when the guy's sitting there, there's an operator sitting in the room. There's

Lesley Carhart (34:50.691) A hole.

Aaron Crow (35:16.184) Shooting the shit, talking. All of a sudden the operator goes, G D, like leans over, reboots the computer. And my guy's like, what are you doing? What did you just do? He goes, every Saturday it kicks off this one process and I don't know why. It always hangs. And the only way I can get it to come back is reboot the computer.

Aaron Crow (35:39.213) It had been doing it for years and he had told people about it. Nobody did anything about it. Nobody cared. So it had gotten forgotten. Nobody thought about it. It was just the way that he got it to work. He rebooted it every Saturday night when he was on his shift at two o'clock in the morning, like clockwork. Right. And it wasn't in any pre-job brief. It wasn't in any operator logs. It didn't come. Only thing that came across in Splunk was system rebooted. Why is it rebooted? Like we didn't know, but we had to sit there and see it.

Aaron Crow (36:6.860) And it was probably something that was eating away memory or some unknown process that was offgoing, but nobody knew it. And how many of those things are NOT across our critical infrastructure that just nobody knows are sitting there just an Easter egg waiting to pop up and say hi?

Lesley Carhart (36:23.959) I mean like the secret is like there's so much junk on these computers, like so many batch scripts and and weird, like custom unsigned vendor applications of things that yes, we're experts at this. Like I do and this has been my whole life for like two decades. And and still routinely I have to be like, give me another computer to compare with this computer. 'Cause like there's like twenty-seven PowerShell scripts, there's PS exec running, there's like seventeen batch scripts. I I

Aaron Crow (36:39.098) Yeah.

Aaron Crow (36:44.506) Yeah.

Lesley Carhart (36:51.831) I just need to like start like sorting through and which ones are in the gold image and which ones aren't. Like 'cause there's just like chaos. And I don't know. It's all custom stuff. Like I d I have no allow list for this. It's it's just like, yeah, I know about the vendor stuff and the file hashes that should like part of the vendor applications, but these environments become so like built on and clutched together over time that it's it's

Lesley Carhart (37:15.373) Sometimes it's just a matter of like finding the person who's been there for 30 years and asking them for a backup image and being like, well, what's what does that do? What is that thing? Like it's but like, you know, and that's that's just part of like real life. Like I re in reality, like of course they reboot the systems because they that's their number one troubleshooting technique, just like it is for us. Something's acting weird, you turn it off and you turn it back on again and you see if that fixes the problem. And

Lesley Carhart (37:41.136) They're doing that all the time because ninety-nine point nine nine nine nine percent of outages and problems in industrial environments aren't cybersecurity related. They're just like something failing, software bug, human error, things like that. And there's tons of redundancies to prevent that from getting really bad. But it's very different when you're dealing with an adversary who really wants to break things and they're like gonna commit multiple efforts to doing that than if it's just something randomly breaking. So

Aaron Crow (37:50.298) Correct.

Lesley Carhart (38:6.755) We do have to be concerned about that small percentile of time where it is malware or it is somebody malicious, it's an insider, it's you know, an infection, a backdoor, whatever. And those are becoming much more prevalent as adversaries great gain that capacity. They have LMs too to look things up and figure out how to blazon the water, break things, and also just there's more access to these environments. They're getting

Aaron Crow (38:22.680) Uh-huh.

Lesley Carhart (38:29.045) inc increasingly connected to everything. So now they're like connected to four different remote access methods and the DMZ and, you know, all these different things that can be a vector for it for more infections to come in and for people to maliciously access the devices.

Aaron Crow (38:32.441) Yeah.

Aaron Crow (38:44.484) Yeah. Yeah, and and all of that is not in any class book, it's not in any certification, it's there you go. Make sure that you read the book the the the the the paper and and put it in your in your in your power plant procedures because I mean again yes no I mean I I seriously.

Lesley Carhart (38:52.119) It will be now I wrote a paper Teach it in your class Make it a sign reading.

Lesley Carhart (39:1.611) Yeah.

Lesley Carhart (39:7.427) I I I need citations. Give me citations, but anyway. No really. I wrote it 'cause there's nothing. There's nothing. So there you go.

Aaron Crow (39:15.512) Yes. Well, so so how do we what is the next step? So i convictor's one of them. How many others are out there? How many others of these are these problems gonna be there?

Lesley Carhart (39:29.652) All the auto-run worms, all the old like info stealers. There was like an era where like worms were the hot thing. Again, young people and it might not remember when like there wasn't it there wasn't so much ransomware. It was more like building botnets and stealing game credentials and banking logins and stuff. So that's what most malware did. Like it it was meant to be quiet. Like you didn't want it to get caught.

Aaron Crow (39:37.689) Mm-hmm.

Aaron Crow (39:45.487) Yeah.

Lesley Carhart (39:53.774) So it would like install on computers and it would spread to all the computers it could get to and it would be like really quiet and it would have commit connections, command and control connections back to a a hacker somewhere and they would tell the computer to do things. Give me your passwords for your banking or your video game, or it could be like you know, attack everybody DDoS this website and that's how they did things back then. So this stuff is headless now. Most of most of them like all the websites have been taken down.

Aaron Crow (39:54.788) Yeah.

Aaron Crow (40:10.746) Right.

Lesley Carhart (40:22.381) doesn't connect to anyone anymore, but like they'll still spread because that's what they're really good at doing is just spread, spread, spread, spread, spread. And most like Windows 11's Windows 10 computers aren't gonna be vulnerable to how they spread. Like that's been cracked down on by Windows. And now Windows comes with like antivirus built in, things like that. But the like these old computers, like that ain't there. It's not there. They get infected again. So it's interesting challenge.

Aaron Crow (40:44.651) Mm mm.

Aaron Crow (40:47.884) Yeah. Yeah, there there's no, you know, the difficulty with the architecture of Windows. You know, there's no pseudo. Like you're you're you're logged in as admin all the time. You have full admin rights on everything. So every process it runs when I'm logged in in every OT environment, they're logged in as admin all the time, especially on these older devices. They don't have multiple logins, they're not locking screens, they've got full access. So

Aaron Crow (41:15.212) All those things can spread like wildfire. And if they have logins, unfortunately, on the one, it probably is the same login that's on the other one. So it it's very, very easy for these things to spread. and it's really, really hard to find them, like you said. And and and even if you found them, did you clear all the remnants of them? Or is there some file that's just waiting for you to walk away and then it repopulates itself, which is what those things are good at. It's gonna keep us busy.

Lesley Carhart (41:45.742) Yeah, thank you for giving me this this soapbox on this too. Like this is really good. again, like I think this is gonna be maybe the only podcast about this problem. And I'll be dropping the paper before OT SUP Singapore. That's the the really big OT cyber security conference for this half of the planet. So in in July. So I'll I'll be getting it out of some peer review right now. So be yeah, so it'll be out before OT SUP where I'm I'm presenting it.

Aaron Crow (41:51.532) Yeah.

Lesley Carhart (42:16.961) an actual paper for the calls for papers. Like we always submit talks where like call for papers, there's no paper. This time there is a paper and you can use it.

Aaron Crow (42:17.348) Very cool. So hopefully this inspires

Aaron Crow (42:22.831) Yes.

Aaron Crow (42:28.552) And hopefully this inspires like, you know, five or ten or fifty or a hundred people to pick up and and create a a product that can help them to help us and the industry to find these types of things and start thinking about none of these things go away. You know, that's that's one of the things, like again, going back to stucks. Yeah.

Lesley Carhart (42:38.479) Mm-hmm.

Lesley Carhart (42:45.145) We need like backwards compatible forensics tools again and backwards compatible anti-malware that's like low touch, super low touch for OT. Like that's something somebody could work on and produce. There's like a there's a market there. And it's not like you want to be really cautious about being, just make semantivirus for Windows ninety five, some kind of security tool for it. Cause I mean there's resource constraints there and that's it's really tough because you can't run out of memory, can't run out of disk on these industrial.

Aaron Crow (43:0.292) Yeah.

Aaron Crow (43:13.123) Yeah.

Lesley Carhart (43:14.615) But at the same time, you don't want to be like, just keep it in production forever. We're gonna make a security tool. You don't want to sell that as like a panacea, like it's gonna be like a fix forever. but there at the same time, there is a space for that, and there will be for a long time. We it and Windows has been Microsoft team has been good about you know producing out-of-band patches for even unsupported operating systems in some of these like really horrific cases.

Aaron Crow (43:32.772) Yeah. Abs

Lesley Carhart (43:44.555) so so people are thinking about critical ext infrastructure to some extent, but yeah, the the A V and the forensics tool market has dried up. It's it's gone now, like for Windows XP, especially pre-service pack two. There's nothing anymore. There's there's like next to nothing. And I understand why because Windows architecture has changed so much. It's hard to make a backwards compatible product for Windows, Linux to some extent too.

Aaron Crow (43:53.434) Yeah.

Lesley Carhart (44:11.137) Even Mac OS has changed drastically since then, obviously a major rewrite. So but there is a space for those types of tools for triage, for scoping incidents, for for cleaning infections, because a lot of it's incredibly manual and requires a lot of study and expertise of old systems to do successfully.

Aaron Crow (44:14.540) yeah.

Aaron Crow (44:33.678) Yeah, absolutely. Well, awesome. This has been a fun conversation for me because it's it's nostalgic. And unfortunately I've been around long enough to to know what all these things are and have have fought this battle in my past. And like you said, I wasn't expecting in twenty twenty six and and to be talking about it again and how how prolific it is and can be in the space. So thank you for for doing the work and and sharing with us here.

Lesley Carhart (44:42.158) Yeah.

Lesley Carhart (45:1.209) Yeah.

Aaron Crow (45:2.602) And definitely will share all the notes and all the things for you for anybody that's interested in in hearing the talk.

Lesley Carhart (45:6.233) Yeah.

Lesley Carhart (45:9.839) Hey, thanks for letting us talk about this today. It was a lot of fun and like I hope a few people learned something.

Aaron Crow (45:17.422) Yeah, absolutely. Well, thank you, Laz. It's great to to see you and talk to you again. And hopefully I'll see you again at a conference and get to see you talk about it directly so I can get the full experience.

Lesley Carhart (45:31.129) That'd be that'd be killer. I'd love to see you again soon. Yeah. absolutely. And anybody reach out anytime. I'm still alive. I'm still happy to talk to you.

Aaron Crow (45:40.260) Awesome. Thank you.

Lesley Carhart (45:43.792) Cheers.

Transcript lightly edited for readability.