AI in OT Cybersecurity: Real-World Risks, Smarter Defenses & the Future of Critical Infrastructure

Aaron Crow: Thank you for joining me on another episode of the PrOTect IT All podcast, where I get to talk to people I've worked with and people I get to call friends. Clark is both. Clark and I have worked together for quite a number of years, and I'm very fortunate to call Clark a friend. Clark, why don't you introduce yourself, tell us a little about yourself and your background.

Clark Liu: Thanks for having me. My name is Clark. I really got into cybersecurity in the NERC CIP era. Unlike today, where people are coming in from cybersecurity as an education, I'm a little too old for that. I came in from networking in the power and energy space utilities, got into the regulatory side, then on the NERC CIP drafting committee, and it all went fast from there. 18 years in power and energy, then really enjoying the dynamics in other areas. Manufacturing, tape manufacturing, food and beverage, all expanded my views from the NERC-CIP myopic view. In NERC CIP you're demonstrating compliance. When you move into other sectors, you're often short on budget and you have to find what moves the risk meter just a little bit in the right direction.

Aaron Crow: We talk about this a lot on the podcast. OT is OT. A power plant is a little different from manufacturing. The technology is different, the business process is different, but ultimately they are very similar in so many ways. You hit on something really important: the budget is vastly different across them. One of the reasons NERC CIP and the power industry is a little bit further ahead is because they don't have a choice. They have to do this or there are fines and compliance issues. Whereas water, wastewater, manufacturing, if I'm not critical manufacturing, there's a limited amount I have to do.

They don't have the same budget cycles and focus, at least they haven't in the past. I see a trend going in the right direction. More and more folks are like, we don't have a choice, we have to do this because it's going to impact my bottom line. More C-suites and boards are starting to understand it's a risk to their business, not just a cyber thing. Not just a cost center. Not just some nerds like me and you who want new firewalls because firewalls are cool. I absolutely think firewalls are cool, but there's a reason behind it.

Clark Liu: Think about what NERC CIP did. Once those fines started rolling out, once those regulatory requirements became enforceable, it created that cottage industry. I feel like it was a good bump in the cyber side where we started getting better tools that fit the OT side of the house.

Aaron Crow: More and more verticals are using NERC CIP and other use cases as reference. Not that they're saying, "We're going to implement NERC CIP in retail or wastewater," but they're using a lot of the framework. Coming from consulting backgrounds, we use different frameworks: NIST CSF, 62443, whatever. Everyone argues, "I think you should use this framework." I don't care what framework you use. The framework is the language we're using to communicate what we're trying to do and what the goals are. Once we have that link, the framework is just the dictionary. This is what we call an incident, an endpoint, an asset. Then you build a program around those for you. It's not like you take NIST off the shelf and there's your program for Duke Energy or Coca-Cola. They have to personalize it. It doesn't matter which framework you use. There's no right answer. There's probably a wrong answer, but I don't think there's a right answer.

Clark Liu: Frameworks are powerful in that, once they choose one, people say compliance is compliance and then we need to do the real cyber work. But for a lot of these electric utilities at that time, this was very new. The journey the regulatory side was going to take them on, as they built up those foundational cybersecurity controls just for compliance, they then got to see, "Hey, look what this is showing me." As they oriented toward their environment, how they're set up, their specific equipment, they were able to make it more than just the regulatory aspect of cybersecurity.

Aaron Crow: Compliance doesn't always equal security. They can be two different goals. We see this in NERC CIP and power utilities a lot. I can be 100% compliant and still have risk in my environment because I'm never quote-unquote secure. The biggest problem I see with NERC CIP is the scope. NERC CIP doesn't include all assets at a site or company. It's a subset. If I have 10 assets and only seven are part of the NERC CIP program, I'm only including those seven. Even if I do the same actions on those other three, which is what you should do, many don't even do that and they're not tracking it. They think, "Well, I've done NERC CIP, so I'm good." But what about those other three? A chain's only as strong as its weakest link. Those are the places they're going to get in. Colonial Pipeline, a lot of these attacks, they come from some supplier's VPN with a back door or an HR system that's not technically OT. Ancillary or tangential systems that cause downstream effects, because they're not part of the program.

Clark Liu: They have the opportunity to. A lot of what we learned since the beginning of NERC CIP is that firewall rules are so different in the enterprise space than how we operate the grid. The grid architecture is so set, the equipment we use, the protocols. Some utilities laid out dual firewalls because of the differences in how things are managed and operated. The mutual distrust model: the IT firewall on the enterprise side, and the OT side has their firewall. The OT side has planned both ports and services for normal operations and emergency operations. When those were merged as one, IT would go and do their annual firewall review: look at traffic, this rule hasn't been hit in 365 days, delete it for performance. If IT did that on an OT firewall and there was an event where they had to use alternative or backup protocols on the network, they'd ask, "Why doesn't it work anymore?"

Aaron Crow: I've been a proponent of having a separate boundary firewall for OT. Really segmenting OT off your IT environment in a physically segmented way. A lot of reasons behind that. Defense in depth. I always assumed my IT organization was captured. You have to. I would put that there. We went as far, in some sites, if corporate had vendor A, let's say Palo Alto, I'd put Fortinet or Cisco or some other vendor. Not because a particular firewall is better or worse. Just that it's different. The same vulnerability that hit the top one wouldn't hit the second one because the OS is different, the vulnerabilities are different.

I can argue both sides of that coin. The bad side is I now have two different platforms, understandings, and skill sets I have to have. Especially on the OT side, we're under-skilled, and I don't mean the people aren't smart. I mean they don't have enough hours in the day. They're working on keeping the plant running and making widgets. They don't spend as much time on firewall and cybersecurity. Not because they don't care. It's just that their job, their boss, their bonus is tied to functionality and operations of the site.

Clark Liu: If you bifurcate that responsibility and ownership so each has their own firewalls, it's not just firewalls. It becomes an entire stack. The OT would then own their own Active Directory, et cetera. Mutual distrust is a powerful pattern.

If we go the other way and it's integrated, the IT/OT firewall is being managed by the IT side. There are a lot of policies the OT people need to be part of so that the operational sense of IT doesn't break any of the run conditions, normal, emergency, or incident response. How many times do we really think about the ability to respond in OT? A lot of times we're going to sever the connection with the mothership and try to keep operational as long as possible, to shut things down safely or to finish a certain production run, even if there is an ongoing attack. That really rattles IT a bit, because they're like, "Shut everything down."

Aaron Crow: We know we can't do that in an OT space. The analogy I always give: you're flying on an airplane, and the airplane has a vulnerability on the avionics, and you're 30,000 feet up with 300 people. Do you want them to patch and update that now? I think not. Let's land the plane, get the people off, then update it. A different level of risk when we're talking about real-life things. It's not "my email server got booted and I lost a couple of sales." Even if a couple of sales means millions, we're talking about loss of human life. A completely different value and risk.

Clark Liu: With the "we're so short staffed," these engineers are working on several other things. The last thing they need to do is help someone change their password. How many of those requests come right through? That's where I had hope for AI. AI is going to enable us to do a lot more in OT than we've ever been able to.

Aaron Crow: It's that double-edged sword. It's going to be so much more effective and efficient. This week I've literally built three websites and I'm not a web designer by any stretch. I updated my podcast website, my personal website, built something for ICS Village. I built all that with AI. I've done this in the past and it looked very 1984, block, "hello world." But now with Perplexity Comet and it ties into Nano Banana or whatever, it creates the images, does the hero, the workflow. It's actually doing an SEO audit: "The images are too big, it takes too long to load, you don't have SEO tags." Especially as someone who's not a web developer, it looks good to me, I wouldn't know it's not great.

Imagine we take that to an OT space and take away all those menial tasks. How do we do it and not add unneeded risk? Because we are bringing risk in. Where's the line where we're comfortable? How do we keep a human in the loop? It doesn't mean you go Terminator and say, "AI, run my plant." Maybe you say, "AI, go do these things and I'll confirm if I agree." You have that last level of submit where it's not actually making changes, just saying, "These are changes I recommend, would you like me to do them?" Think about how many times we forget to back up a switch configuration, or we made a change and didn't submit it, or we fat-finger a password, or because it's so difficult and I'm tired of changing the password, I set it to never expire. I've never done that before, ever, in my life, as an admin. No. Never.

Clark Liu: Those are great places to start. Especially love the change management piece. Change management is both on the operations and maintenance side and on the cybersecurity side. Imagine when we can bring AI along and a controls technician finishes a break-fix and is able to utilize AI to document everything, update the drawing, update the diagrams, add that node in, close it out for the weekend report. Since it's a break-fix, that means no change to even get started on it. Just have AI kick it off and document the whole thing.

Aaron Crow: That is so huge. How many plants have I gone to where I ask for documentation and I get the as-built from 1982, maybe with some red lines, but they haven't updated documentation in years? Sometimes, and I'm not exaggerating, decades. It's all in Bob's head or Sally's head. Not actually documented anywhere, or documented in 50 different places. Bob knows where his stuff is, Sally knows where hers is. There is not one place where if Bob and Sally won the lottery and left the company, Aaron could come in and put it all together.

Clark Liu: I found a substation drawing once. Decades old, but all the pencil, pen, all those little notations, when you put it all together, was up to date, just like that. But you had to know which substation to go to to find that drawing. What happens if they make a fix and don't go back? It looks like the drawing was used as a general notepad for when a substation technician and a protection engineer leave notes for each other: "This is what I last touched."

Aaron Crow: Even as I'm building these sites using AI, I'm publishing in GitHub, a private repo, but AI is self-documenting. Every change. I can go back. Because I'm not a web developer, I have 30, 50, 100 commits to build this site. "I don't like this, I don't like that." Every commit, it's documenting the change, why it was done. If I want to, I can go back at any point. This is a stupid website. Imagine if we had that level of rigor in OT. The main reason we don't do that today is people don't have the time to document all that if they don't have to. Not because they don't see value. They have five minutes and they've got to go on to the next thing, 15 alarms screaming, 15 people asking them to do things. They don't have time to sit for an hour per change.

Clark Liu: I love where you went: "Do we want Aaron configuring this firewall?" You can tell historic firewall rules, who wrote which rules. If you had AI go in and apply those standards, clean up the object names, standardize the method, those firewalls would become much easier to read, much more efficient in terms of having the most-used rules at the top.

Why is all this stuff so new to us? AI has been around in cybersecurity for 10, 15 years, maybe longer. We finally figured out how to enable this for the masses and not just keep AI for researchers and PhDs. AI is now usable by people like us. That's the huge part.

Aaron Crow: The thing I was thinking as you were talking: there's a show called *The Pit*. It's a doctor show based on an emergency room in Pittsburgh. Kind of like *24*, where each season is one day, except *The Pit* is one shift. 12 hour-long episodes, one 12-hour shift of a chaotic emergency room.

This season, spoiler alert if you haven't seen it, there's a cyberattack hitting the hospital. A large hospital chain with multiple locations. A couple of the other locations were hit with ransomware. IT decided to shut down all computer systems at this hospital on a Fourth of July weekend, so nobody had charts, couldn't order anything, nothing digital, screens didn't work. A few doctors and nurses had been there a long time and remembered how to do manual charts. They had to bring in nurses they had laid off or who had retired. Manual charts for every patient. How do I get meds? How do I order X-rays or MRIs when everything is manual? They literally had physical runners who'd take orders and run to the pharmacy, to the X-ray machine.

It shut down everything because 90% of the people there were 30 or younger and had always done everything with technology. That's a TV show, over the top, but the use case is real. That's what we're dealing with in OT ransomware. If it got down to that level, you're literally talking about people's lives. It's why in nuclear environments we have triple-redundant systems and the last system is analog, because they don't ever want that to be a problem.

Clark Liu: For folks who don't practice it on a regular basis, they're going to be in the fire drill and hopefully they still have folks who understand how to do things manually.

Aaron Crow: It's such an archaic skill. Imagine if the power plant went down. Granted, power plants have operators who know how to do things. But there's also a big concern. We're losing experience. Folks who've been there 40 years are retiring. As you hire new folks, they may or may not have the ability to do the same things or know where the bodies are hidden. How do I start this plant without going to the control center and hitting start? There's a manual way to do those things. Everything the control system is doing, you could do by going to the field and turning the valve, putting it in the right place. But you have to know how. That's not something you necessarily train everyone on.

Clark Liu: One of my early things when I walk into a new role is to understand where the criticalities are. How many sites do we have? What are we prepared to do? Can I pull this ethernet cable out if I really need to? And then what is that runway? Really giving legs to that incident response plan. You have to include the controls engineers and the folks from operations and maintenance to understand the full picture.

Aaron Crow: I can be on both sides of the AI thing. I generally think AI is going to bring more good than bad. There's a bunch going on right now with Anthropic and finding vulnerabilities in software very quickly using AI. Zero-day type vulnerabilities. It opens the kimono to all the technical debt in every product in the world. Scary. On the flip side, AI isn't creating those vulnerabilities. They already existed. We just didn't know. Every zero-day was already a vulnerability until somebody found it and exploited it. If we can take AI that can look at things and say, "Here are all the holes," what are you going to do about it? It doesn't mean you have to patch. There are multiple layers of protection: firewall, patching, disabling RDP. Isn't it better to know than not?

Clark Liu: Think from yet the other side. Engineers who want to employ AI into a line. When they begin to think about what's needed for automation equipment to allow AI in, first of all, human in the loop. In the future, maybe closer than I think, when can I allow it to optimize my line? What are all the thresholds, the set points? Where do I now table safety? That combination is how I'm hoping controls engineers look at adoption of AI. Then on the cybersecurity side, I can say, "Okay, I'm counting on these thresholds to exist. They may have been sort of there, sort of not there. Now I can narrow down bad-behavior detection to make sure engineers are using AI safely in OT."

Aaron Crow: It's coming. We're not going to block it. No different than everybody using Netgear routers or D-Link in OT. They're going to find a way. It's like the preacher's kid. Sometimes the preacher's kid goes off the deep end in college, because they never had freedom or flexibility. Our OT and operations folks are going to find a way. "This could really help us, but they keep telling me no, so I'm just going to go do it on my own." That's where it can be dangerous. They don't understand what they don't know. They don't know the risks.

You can't just be in your ivory tower and think you're going to block it. You need to be having conversations and finding ways. On the flip side, organizations that do it and do it well will surpass their competition. They'll be more efficient, more effective. They'll do things people who are saying no won't.

Clark Liu: As fast as the organization adopts AI, the cybersecurity folks need to adopt AI just as fast. As they explode there, you're not going to have the time to analyze and understand all the things you need to protect their adoption. You have to employ the same technology to help you get your arms around it.

Aaron Crow: 100%. What do you see as the biggest gotchas for organizations trying to use AI, and how do you think it's going to be best implemented and protected?

Clark Liu: A lot of folks are looking to ring-fence what AI can do and where AI can connect to. Always worried about that "call home" aspect. When you put so much of your IP into these big engines, you're worried it's going to be used as part of training on the next version, and all of your IP becomes public. You ring-fence that data. And remember: people always find a way. Every time an engineer wants to use the latest AI tool that hasn't been cleared through enterprise yet, they may not always research what happens to the data they put in. So you have a barrage of not only trying to ring-fence your own folks, but also looking at publicly available AI engines and seeing if you can get them to cough up your own IP. I'm not even sure how you get that back.

Aaron Crow: I don't think you can. For folks listening from cybersecurity professionals to my kids to my mother-in-law, let's say you're Joe Blow on the street. What does this mean to you? Look at Open Claw and all these cool AI things. You can do a lot, but you have to be really careful. Many of these things have the ability to log into your email, take actions on your behalf, respond to emails and social media.

Would you give me your password to your email? To your phone? You don't know me that well. Do you trust me? How do you know I'm not going to do something malicious, even accidentally? Just because you can doesn't mean you should. I see Open Claw everywhere. Consultants, contractors, small businesses, soccer moms. "I can do this." Yes, you can. But you need really good cyber posture and foundational things. Not everybody has the same level of segmentation in their home network as I do, firewalls, unique 27-character passwords, multi-factor authentication, YubiKeys. I do this for a living and I force my wife and kids to do it. But I walk into so many places where that's just not the case. They don't have it because they don't know why they need it. They don't read about it, they don't podcast about it, they don't eat, breathe, and sleep this stuff.

Clark Liu: Think about when you enable the bad actor with AI. Just like Claude can identify vulnerabilities in code, bad actors can take advantage of these faster. They can examine different levels of attacks faster. All the things they were iteratively testing can now be done in AI on their behalf. It goes both directions, and defending against that is going to be a nightmare.

Aaron Crow: AI can help us defend, too. If we train it the right way and use it the right way, it can help us find vulnerabilities, plug those holes, document things. It's more about when, not if. Not a matter of "are you going to get hit." It's, "What are you going to do when it happens?" How are you going to respond? How are you going to recover? We've seen airports go down in Europe, train systems go down, power plants, Ukraine, Gaza, war zones where cyber is more of an everyday thing. CISA, FBI, DHS releases about critical infrastructure because of Iran, China, North Korea.

But it's like watching the evening news. Somebody was mugged, a car was broken into, but that's across the highway, until it happens to you or your next-door neighbor. Then it's different. "Wait, my neighbor got broken into? If he could, I could." It changes perspective. Unfortunately in cyber, things don't change until you really see value, or you think it's a real possibility. Bad things have to happen before drastic responses.

Clark Liu: I'm still looking forward to the productivity benefits AI is promising. Cybersecurity hygiene, the everyday stuff. Making sure the things bubble up to the top. "These access reviews need to happen for these particular servers because they're higher criticality." Helping us prioritize to mitigate the highest risk. I have that hope it'll help us direct ourselves faster to where we need to go, instead of always just following processes and dotting i's and crossing t's. Let's look for the i's and t's that matter most. Get those done first.

Aaron Crow: Using AI is going to be like your junior-level intern. If you were assigning them tasks, that's what you assign AI. Not running the nuclear reactor. Filing the TPS report. Something you could train a monkey to do, easy to document. Start there. Then more advanced, more capable, human in the loop. "Okay, now you've done this. Now take the next step." Start small, so impact is low. Not giving it NERC CIP assets. Not having it shut the plant down or start it up. But documenting changes automatically. Scripts I'm currently writing. AI can help update, enhance, document, clean them up.

I don't see it displacing people as much as making people more effective. That engineer has more capability. Maybe I don't have to hire a second person as quickly because one person is more effective. It's no different than an electric screwdriver versus a regular screwdriver. Electric saw. Force multiplier. Another tool, if used correctly.

Clark Liu: When they're updating or building a new line, the engineers involved, all that power going in, once it's delivered, you might just have a technician. If all the as-builts and documentation were there and enabled AI to assist that technician, to identify these problems before they shut the line down, that's where I see the benefit. The engineers are not going to be there all the time.

Aaron Crow: Or even if they are, they have so much going on. Making it easier. No different from why we started doing automation in the first place. A long time ago, all this stuff was hardwired. 4-20 milliamp. Wires directly to the endpoint. Why did we start using Ethernet? Not because it was necessarily better. It was easier and faster. We could do so much more on this backbone. Everything started having an IP address. Then, back in the day, everything was proprietary. Schneider, Foxboro, Emerson, GE, all in-house custom computers and custom OS. Then: "This is a lot of time to keep up with and patch." They started grabbing Microsoft Windows or Linux or Sun Microsystems, because it was easier to add an application on top of a platform than maintain the entire platform, hardware, software, firmware. We just bolted things on.

We did the same thing with networks. Added firewalls. All this stuff has just been added on. AI is going to be the same. Let's learn from our mistakes. Let's make sure we're designing a secure process from the beginning, instead of waiting until something bad happens and having to take it down or respond because a plant went down, or now I've got an AI process running wild, continuing to turn things off I don't want to or on I don't want it to.

Clark Liu: I'd love that at design. Whenever you can get things at design, I don't have to bring a cybersecurity tool to probe, examine, and make a best-educated guess. "Does this look like this signature? Maybe I should block this." Whenever I can get anything at design, it's huge. Look at that automation and those standards, you really want to get your hands on them and see how much you can influence. Now with AI entering the automation space at velocity, it's a little easier. Instead of coming in as a cybersecurity person, you now come in under the guise of making sure you're AI-ready on this particular system.

Aaron Crow: I just got back from RSA in San Francisco. Every product has AI. "AI this, AI that, we're AI in this." Like the IoT thing five, 10 years ago: "You can IoT your toe." This week, our toaster is on the fritz. My wife and I got that toaster as a wedding gift 18 years ago. I started looking on Amazon for a replacement. Most of them had a big touchscreen display on the front. I'm like, I don't want that, that's just going to break. Some were WiFi or Bluetooth enabled. I just want a toaster. You put bread in, you push it down. It's not that hard.

Clark Liu: Have you seen the small makers who are now making dishwashers and washing machines with no circuit boards? They're going back to basics. I made a mistake with my first house. I bought it from the builder. He was an architect. The AC and furnace were very, very old school. You could get parts for it. Not only that, you didn't need branded parts. As long as you could connect things within certain parameters, you could keep it running. That thing was going strong for 30 years. Unfortunately, I modernized it. Now every five to seven years I have to replace it.

Aaron Crow: Everything is like that. Refrigerators, washer-dryers. I have a washer-dryer, the motherboard went out, I called the manufacturer: "Yeah, we don't make that part anymore. You can't buy it on the secondary market." They literally build obsolescence into their products so you have to buy a new one.

I saw this morning, on Instagram or somewhere, a guy sharing that someone is releasing, through Kickstarter, a complete open-source printer. All the parts are serviceable. It uses HP cartridges because they're universal and common, but you can refill the ink, replace the gears, replace all the parts. Uses CUPS printer drivers so it'll work universally. Mount it on the wall. Rolls of paper or cut sheets. Color or black and white. Use it as a plotter. Completely customizable, similar price point.

I just bought a new printer. The $500 one I had for years stopped working. I think the network card went out, but it's not serviceable. Had to buy a whole new one. Ended up buying a cheap Walmart one for $50. It has enough ink for five pieces of paper. Then, "Hey, you've got to buy ink now." That's where they get you. The ink refills are twice as much as the printer.

Clark Liu: All the industries are just going for recurring revenue and trying to drive it. How fast can we get that recurring revenue to happen?

Aaron Crow: So what do you see coming up over the horizon? One scary thing, one exciting thing.

Clark Liu: One and the same. AI is both scary and an incredible opportunity. The opportunity first: when you build out automation lines in any industry, you'll probably have near-built-in predictive maintenance. All of those bonuses that used to be add-ons and separate products from different vendors, you'll integrate into just the controls you have today. Maybe an additional sensor here and there if you want to dedicate something toward predictive maintenance. The opportunity is huge. We're going to get more for what we do.

Unfortunately, getting more for what we do is also going to make it more difficult on the cyber side to detect what's normal and what's an anomaly I need to look into.

Aaron Crow: I think that's going to be the differentiation. You're going to be less dependent on the right product because you're going to be able to use AI. It doesn't mean products won't matter. They definitely do. But you'll approach it differently. I'm excited about it too. Also concerning, because if you don't do it right, it can be dangerous. But I don't fear the change. I'm excited for what's coming. How we grow, how we protect things, how we get more data and capabilities.

It's not just a cyber thing. If we can use this to make processes better, faster, respond quicker, have more capabilities because we don't have to have a vendor create a capability for us. It almost seems like the spreadsheet. I used to put data in a spreadsheet, macros, formulas, data feeds, robust spreadsheet. AI is filling that void. I don't need a spreadsheet anymore. I turn it over to AI and it's a more visual dashboard with integrations and APIs and we can lock it down. If you do it correctly, it's really powerful, but we have to be careful and design it with security and risk in mind from the beginning.

Clark Liu: Right. Having all that in place should be mandatory. They should have those prompts already pre-made and shared with all the engineers to execute on.

Aaron Crow: Awesome. Thanks for digging in. It's always interesting to talk through and get other people's perspectives. Thanks for your time. I can't believe this is the first time you've been on the podcast.

Clark Liu: Thanks for having me on. We could talk for hours when we talk shop.

Aaron Crow: We'll just record it next time we're eating dumplings.

Clark Liu: Exactly. Thanks, Aaron. See you.

Aaron Crow: Thanks, Clark.

Transcript lightly edited for readability.