Episode 14 - Practical Approaches to OT Cybersecurity in Critical Infrastructure

Aaron Crow (0:1.422): Hey, welcome to the podcast. Today I wanted to have an episode to talk about OT cybersecurity products. It's important to understand the unique challenges and solutions in OT. I've been in industry for 25 plus years. I've spent time in IT, I've spent time in OT, and there's definite differences. The technology isn't always vastly different, but the implementation and the impact is.

Aaron Crow (0:28.814): That's why there's a lot of differences in how things are done in OT and IT. And I want to take some time and really talk around what makes, what makes a product OT versus IT. When can I use what? Right. So, you know, first let's say what is OT, right? OT is, OT environments are power plants and substations and trains and manufacturing. And, it is, it is really technology that has a physical touch.

Aaron Crow (0:56.334): If you think about it in like home automation, it's, you know, you've got a light switch that's automated, like I can turn it on from my cell phone. It's different than IT. IT is normally, you know, email and, you know, web servers and my, you know, cloud and all that kind of stuff. It's more back office and sales type stuff, right? The key differences with OT and IT, you know,

Aaron Crow (1:23.086): We talk about the CIA triad, confidentiality, blah, blah, blah, right? The CIA triad in OT is really flipped and in confidentiality is not the most important thing. It's really availability. And when I say availability, I usually like to say availability slash safety. It's available. Like you want your power to work. When you turn your light switch on, there is no outage. There's no downtime. You don't have a...

Aaron Crow (1:49.806): Friday night at midnight where you take the server offline or the power plant offline, you want it to work all the time. And it's the same thing with a safe environment, right? Is you have to have safety being the priority over everything else. Power is a great example in that there's no proprietary way to generate electricity. Power plant A from vendor A and power plant B from vendor B, they use the same technology, right? It's not, that's not their secret sauce.

Aaron Crow (2:17.550): That's not how they differentiate themselves from others. It's more in how they spread and their assets and all that kind of stuff. But a power plant heats up water, creates steam, turns a turbine, turns a generator. Whether it's fuel from solar or wind, well, obviously solar is a little different, but wind or hydro or gas or coal or nuclear. The high level thing that it does is pretty much the same. It's a tea kettle.

Aaron Crow (2:47.438): turn something to steam, to turn a turbine, to turn a generator, to generate electricity. Can IT products be used in OT? I've been doing this a long time and all the way back to, gosh, I think 2010 -ish is when I really started focusing in OT cybersecurity in critical infrastructure specifically as like my primary job.

Aaron Crow (3:15.086): It was never kind of called that before that. NERC SIP really changed the world for the power utility world. But there were no OT specific products back then. So we, we had to adjust and adapt. So I took commercially off the shelf products, you know, traditional firewalls, Palo Alto firewalls and Cisco firewalls and Fortinet and all the different things that were, that were in our IT data centers.

Aaron Crow (3:43.854): and took them and just implemented them in a different way. Cisco switches and gigamons and Ixias and packet capture devices, we were using things like what's up gold for uptime monitoring and SNMP monitoring where you're using NACs to do monitoring on the network layer. But the way that we implemented a NAC was completely different than the way we would implement it in IT.

Aaron Crow (4:13.102): I was using a NAC almost like an asset discovery tool. I was doing no blocking. I was not segmenting networks. I was really just alerting out, hey, this new thing came on and these are things that I see on that device, right? So we set up SNMP traps. We set up a lot of different things. We were also doing full packet capture on the entire network. And again, this is back in 2010. We were doing active scanning. I know that's a controversial subject, but...

Aaron Crow (4:41.326): All the way back in 2010, we were doing active scanning in a production OT environment, power plants, et cetera. Now, were we just going and randomly scanning the same way I would do in an IT world? Of course not. But we were doing active scanning in an OT world. We had to do it carefully. We had to be selective in the devices and the networks that we did it in, but we were absolutely doing it. So it can be done if you know the environments that you're going into and you are careful.

Aaron Crow (5:10.158): and really understand it really takes an understanding and working with the sites. I never went in and said, hey, this is what I'm doing to your environment. It was more of a, hey, these are the requirements that we're trying to do. How can we do this in your environment safely without bringing your environment down, all that kind of stuff, right? So there's a lot of ways that you can use it. The other part of the IT products that don't always necessarily translate into OT is,

Aaron Crow (5:38.990): OT uses a different set of protocols sometimes. They have OT specific protocols. So, Profibus and EtherIP and DMP and, you know, there's these OT specific protocols that don't show up in an IT world. Now, again, this 2010 is when I deployed this, even next gen firewalls like, you know, Apollo Alto, for example, back then, they didn't understand OT protocols. So even if I put it through that,

Aaron Crow (6:7.790): you know, later seven application smart next gen firewall, it didn't know what the protocol was. It saw it, but it didn't classify it. They didn't have that disector. These guys have come a long ways. The Forrester report I put out the other day, like there's a lot of our commercial IT products that have gotten into the OT space and they have OT lines and capabilities in their IT products. So that line is actually getting closer. We have more and more products from the IT side that are

Aaron Crow (6:37.422): that are getting that capability in OT Fortinet and Palo was top right quadrant in that that Forrester example. And that was a great example of how they've done OT protocols in these products so that I already have a firewall. I can I can now start monitoring these these these OT protocols in that space and I get some capability out of that. There is also an impact on, you know, operational uptime.

Aaron Crow (7:4.430): OT a lot of the times is very time sensitive. So if I think about right now, most OT is unencrypted. It is wide open. The main reason for that is because, well, there's a lot of reasons, but one of the reasons for that is time sensitive, right? They don't want to add additional latency by encrypting and decrypting when again, most of their traffic is on site. The physical environment is locked down.

Aaron Crow (7:32.686): So to be a man in the middle, you would physically have to walk into a power plant or a manufacturing facility and plug in, which is not impossible. They've just done more. They focused more on the physical security being that that reducing that risk instead of encrypting the data, right? It's still something that I think we'll get to.

Aaron Crow (7:55.854): and you are starting to see encrypted protocols out there. They're just not widely used specifically in something that was installed 50 years ago. They're not going to go back and replace all their stuff just to encrypt the data. It's a low risk point from their perspective and a high, high risk to availability because all of the stuff doesn't necessarily support it. So those are some of the bigger issues of things. But again, going back to, you know, NACs and firewalls and, and, and

Aaron Crow (8:25.070): switching and you know packet captures and DVRs or net DVRs and you know packet capture devices. All that type of stuff is those are the same tools that you use in IT and OT. The biggest difference is going to be like I said the protocols but again more and more IT products are getting ability. You see other products like Tenable that has you know an IT product and an OT product. You see LandGuard that has them both. You know Industrial Defender you can talk to IT products.

Aaron Crow (8:53.006): as well as OT stuff. Like they don't only talk to OT protocol, they also talk IT, they speak SNMP, they can talk to a firewall, they can talk to a, you know, a VM host and all those types of things. So a lot of the equipment that you see in the OT space is the same that you would see in an IT world. It's just kind of implemented and architected differently. That really goes into why we need some of these OT specific products. Some of these OT specific products, again, may have similar capabilities to their IT cousins.

Aaron Crow (9:21.582): but the implementation may be different. The OT world is highly segmented in that at a single site, I may have multiple network segmentations. I may have multiple active directory forests with no trusts. I may have multiple units and each of those units is a different network environment. Same thing on a manufacturing line. Like each line could be an individual subnet.

Aaron Crow (9:48.846): And there could be a firewall between them to segment those things. You're starting to hear it now with zero trust and all these things. Again, firewall rules, all this stuff comes from IT. And there's a huge benefit to that. OT was already fairly segmented and this gets it down further and gets around some of the limitations in OT. Like you can't patch the same way that you do in IT in the OT environment. You just.

Aaron Crow (10:15.470): It's too risky. A lot of these devices are older. They don't have patches. The patches are not applicable. I mean, you've got Windows XP boxes still running in production and they're air gapped and you know, sometimes they're not, but they're not on the net on the internet, right? They're they're on a segmented network. But the point is, is that it's not necessarily the right thing to just replace that XP box as crazy as that might sound as a cyber security person. It's it's not even supported anymore. Like it's a super high risk.

Aaron Crow (10:45.518): But the alternative to that is that this environment, this control system is running and it's millions of dollars to replace it. Just to get a vulnerability out that I can get out a different way. So a lot of times in OT, I look at a risk and the same risks exist in IT and OT, but many times I'm adjusting the way that I respond to that risk differently than I would in an IT world.

Aaron Crow (11:12.462): That kind of leads me to the next kind of conversation on, you know, in OT, what are some of the common things that you're looking for, right? Is network monitoring, right? You need a lot of, right now, a lot of our OT environments don't have a lot of visibility. Usually there is a segmentation, a firewall, something that stops normal cybersecurity, your SOC and your normal IT folks.

Aaron Crow (11:40.558): they can't see past a certain point. They can't see into the OT network. So it's really this void of unknown things behind that wall. Maybe they're getting out some alerts in a good environment. Some environments they know nothing. So they need network monitoring, intrusion detection, because you're not patching as often. You really wanna monitor, use your Sun, Sue, R, or War, use your strength for your weaknesses and your weaknesses for your strengths, because you're not patching, because you're not updating those things all the time.

Aaron Crow (12:8.718): I'm not surfing the internet. I'm not installing apps. I'm not plugging my, I shouldn't be plugging my phone into these devices. So they shouldn't change that often. So a change in the environment should be easy to spot because it should be the same all the time. So if something is different, I should notice those things. So asset management tools are another really big one in that I want to know when an architect, when a, when a configuration changes. Just like I said a minute ago, right? Is, is

Aaron Crow (12:35.534): the more that I can see change in the environment, the better I'm going to understand. And it's beyond just a cyber security. No, right. I want to know the system went down, right? Not necessarily because China's hacking or because, you know, a bad actor did something. Sometimes it's just, hey, Bob installed something and it's not working anymore. Like who made the last change? So you can go back to a known good state or the last known change and revert back to a previously known good.

Aaron Crow (13:4.846): Right? So that's good from availability from all of that. It's not just a cyber, but it obviously also gives us a lot of insight into risks and firmware and OS and vulnerabilities and all that type of stuff. Right. The next goes down to anomaly detection, threat intelligent platforms. The more that we can get into those spaces in these OTE environments, the better. And another big one that's usually on my list is is secure mode access.

Aaron Crow (13:33.294): We usually have vendors and third parties and even individual within the company that are gaining access to these devices remotely. And remotely can be, hey, this device is on the floor and I wanna be able to access it from my desk. I don't wanna have to walk a mile every time I need to touch that box. Or in the power world, that power plan is six hours away or it's in another state. I need to be able to access it.

Aaron Crow (14:1.838): without having to get on a plane and go to that place. The vendors do the same thing. You know, you've got vendors that are remotely tuning turbines and it would be really expensive if they had to come in at every outage, which is the way they used to do it. But now they're so much more efficient because they can do that on the fly. They're not having to wait till outages to come in and tune those things. They can do that on the fly. So there's a huge benefit in that. That really leads me to the next thing, which is there is no silver bullet.

Aaron Crow (14:32.302): What do I mean by that? You can't buy a single product. Like back in the day, I remember when we first started drilling this stuff out in these OT spaces, we rolled out a firewall and most of the environments thought, okay, well, we got a firewall, we're good. Check the box, I've got cybersecurity. Like it was a box that we had to check. And once I have this thing, I've got oxygen, I've got water. Like it's all I need. I don't need more. I don't need more cybersecurity.

Aaron Crow (14:59.470): I think we've we've realized now that it's more than just a single thing. There's a lot of risk. You look at this, you look at this, you say for 64 for three, any of the of the frameworks, there's a lot of things that we have to consider from disaster recovery and instant response and secure mode access and patching and all the things. Firewall doesn't fix all that. No product does. So there is no silver bullet that answers all of these things. So.

Aaron Crow (15:26.318): when you're looking at your OT environment, it's important to really figure out what's most important. Again, going back to patching, you're probably not gonna patch your OT environment that often, sometimes never. Some of these environments have never been patched. And again, as blasphemous as that may sound, it is not necessarily the wrong decision. A lot of these technologies are coming out now where you can mitigate those controls, those risks.

Aaron Crow (15:55.886): without actually having to patch. Like if I know RDP is a vulnerability on this endpoint, well, I just can make sure that nobody can talk to you on RDP. Especially now with application aware firewalls and protocols, right? It's not just reducing 33 .89, I'm looking at the actual packet and making sure, hey, that's an RDP protocol. Like I don't want that going to this box, that's denied. So there's a lot of capabilities there, but again, it's understanding what needs to be there and know that you

Aaron Crow (16:25.198): It's a never ending, you know, the goalpost is always moving and there is no silver bullet. So it's important to have that multi -layered defense, defense in depth. Yes, you need a firewall and you need, you know, anomaly detection and you need asset management tools and you need a secure mode access solution and, and, and like, it's always going to be making sure that you're covering the right things at the right level.

Aaron Crow (16:53.902): And you're not going to do that universally across all of your sites. Maybe, right? You've got lower, lower value sites to your organization. You're going to probably focus less on some of them than others. But then on the flip side, a chain's only as strong as its weakest link. And you see things like the target attack and they get in with the back door on one, which goes back to defense and depth. If I segment these environments, even if I'm putting one at risk, I don't want to risk the others. So it's really just understanding of those environments.

Aaron Crow (17:23.982): You know, some of the bigger challenges in OT, super highly segmented networks, geographically dispersed sites, you know, I talked about it earlier with the segmented networks, you may go to one side and they've got, I had sites with four different units. We had five different domains, five different secure mode access solutions, five different patching solutions. Like all these things are compounded and none of them had trust. So I had to have a different account on each one.

Aaron Crow (17:52.910): You know, it was, it was difficult and complex, but it had to be that way so that I didn't reduce the risk at the time. Not saying that's necessarily the way I design it today, but that was a reason we did it back then. and also that's the way these environments are designed by default. They, they stand alone so that I can take one unit offline and do maintenance on the other one without impacting either, either, or each other. Right. So I don't want to overlap technologies so that I can't take one of them down without impacting the other.

Aaron Crow (18:22.126): The dispersed sites, again, that's just a geographic area and a time area, like resources. How long is it gonna take me to send a resource after that place? Do I have to hire more people? All of those questions. Speaking of people, it's training, right? A lot of the time you're dealing with very smart systems engineers that are really good at the OT and the systems and making sure the site does what it's supposed to do. Print widgets, electrons, whatever the thing it is.

Aaron Crow (18:49.390): but usually they're not cybersecurity or networking or IT people, though they're very good at it because they have to be, that is not their primary job. So many times they're not trained in all the areas that we would need them to be to be able to fully understand and secure these environments. So sometimes it's a lack of staff and or training of staff that can be a gap from operational expertise to cybersecurity knowledge. Funding.

Aaron Crow (19:18.894): is always a problem. Funding and resource limitations kind of really tie together, but really justifying the need for the right amount of money to, you know, again, this is not something I buy a widget and I'm done. I'm going to have to continually spend money and upgrade. Unfortunately, the adversaries are not stopping. Like you fix one hole and they're drilling another. So we're going to have to constantly be fighting that.

Aaron Crow (19:48.366): which means we also need resources to help with those, right? So we need people. It's not just a technology problem. It's a people process and technology, right? So you need all of those things. You know, vendor support is a big one as well in that a lot of these OEM vendors, they want it deployed. They can only support so many things without naming any names, but some of the bigger ones, you know, they support, you know, vendor A and...

Aaron Crow (20:14.670): the next one supports vendor B. Well, I've got both of them at my site. So now I have to have two different solutions for the same problem, two different antivirus solutions in the same environment, two different firewall technologies in the same environment. And so just one firewall that would be more than enough to handle an entire environment. I have to have two different brands because vendor A supports this brand and vendor B supports a different one. So that gets difficult. And then, you know, old hardware, a lot of these environments, they may have been installed 50 years ago.

Aaron Crow (20:43.438): 40 years ago, 20 years ago. And sometimes the hardware and the software and the applications that are running in them are that old too. Maybe not 50 years old, but definitely 20. You've got Windows XP boxes out there, which leads to how can I patch that? I can't. So patching is not necessarily the answer. And I also can't just go replace them. So I have to come up with other ways to do that. Sometimes those ways are training and awareness. I need to train my people.

Aaron Crow (21:12.878): you know, the importance of educating people on the risks and the vulnerabilities and how, what, you know, what to look for. traditionally they're going to look, you know, if something happens, they're going to think of, you know, availability, but sometimes we need to be thinking about, Hey, did, did we just have something? How did that happen? Is that a cyber incident? Is that something we should look at? you know, we should be looking at regular audits and assessments, bringing in third parties or even just your internal audit teams.

Aaron Crow (21:39.566): bringing in that IT, your IT brother and into these O .T. spaces to take a look, not to touch, not to architect, not to change configurations, just to look and give an opinion like, hey, we've, we manage a lot of firewalls and this is a, this rule here doesn't make sense. Can you explain that to me? Hey, maybe if you did it this way, it might be better, right? Looking at that kind of thing. I'm a vast advocate for vendor collaboration.

Aaron Crow (22:7.598): customer has the ability to kind of bring the vendors to the table and say, Hey, vendor, A, B, C, and D, we have you all. We want to have a unified cybersecurity program, and I'm not going to have 15 different products solving similar problems. So these are the ones that we're looking at. Let's come to the table. Let's find a solution in architecture and find a way that we can reduce that tech stack a little bit. Maybe it's not down to one or two, but maybe it's down to 10 or, you know, eight.

Aaron Crow (22:36.206): so that we can work collaboratively and with the right tool set and be able to understand them. Because the problem is, is the more tools I have, the less competent I'm going to be in any one of them as an organization, which is a compounding problem. Leveraging technology is huge, right? Use technology, find advanced cybersecurity tools that'll help in OT environments. Use these things to help you with asset inventory and...

Aaron Crow (23:2.414): you know, all of these problems will help on the operational side, not just the cybersecurity side. So when you're having a problem, you can see the last time a configuration change, you can look at, you know, the new, Hey, there was a new asset that showed up. Hey, this box tried to talk to somebody else, you know, somebody remotely logged in, like all of these things give you visibility into what's happening in your environment that helps on the operational side as well beyond just cybersecurity. and then, you know, building a resilient culture in that, you know, promoting a security first, just like we have a

Aaron Crow (23:31.566): a safety first mindset in OT, we should have a security first mindset. Like we should really be thinking, you know, in OT a lot of times it's Saturday night, five o 'clock in the morning or whatever time, I just seem to get it working. So I bypass all the things that I don't understand because I need it to work. And I don't know how else all this other stuff works. So I bypass a firewall or I plug it directly into something. Right. And all those things can happen and do happen. But we need to think about, OK,

Aaron Crow (24:0.846): If I'm gonna do that, I got it up and running check, we need to come back around to that tomorrow morning and I need to get the right people on the phone and we need to fix it the right way. Instead of just leaving this thing here and with a piece of paper or tape on it says, do not touch. So obviously that's a lot, but really we just wanna dive into what the difference is, the challenges in OT, what's different between IT and OT, what is an OT product versus an IT product.

Aaron Crow (24:29.230): kind of explain how sometimes they're very similar, sometimes they're pretty different. I think that line is getting grayer and grayer and you're seeing more and more, you know, really great products from, you know, Palo to Fortinet and, you know, Tenable and all of them that have these great IT footholds and they're really doing a lot in the OT space, which is really awesome because I can do, you know, I've got an organization that uses product A,

Aaron Crow (24:57.134): I can incorporate that into product B and I have all these resources that already know how to use those products and can help architect and design and roll out the right policies and configurations on these endpoints. So I encourage you to ask the questions, bring in your IT folks, ask your vendors, get collaboration, bring people to the table, ask the question of, do we have a product in -house that can do this? Could it work in OT?

Aaron Crow (25:25.902): How could we make it work? Cause a lot of times they're going to tell you, no, well, it's in the cloud. Well, do you have an on -prem version? Could we have a separate instance that's on -prem versus in the cloud? Do we have a different version? Cause I don't want to integrate with the IT active directory. Okay, well, could we have a second version? Like just ask those questions. Don't just take it at face value that the answer is no or that the answer is yes. Cause you'll also get the opposite. well we can just use this product. Okay, but how? This is the unique environment that we have. How would we use that?

Aaron Crow (25:54.766): in our example and make sure that you have the right understanding of your environment, the tools, the needs, and then choose the right tools and products that fit your different use case. They don't all have to say OT on them, because many times we're only dealing with IT. We're dealing with firewalls and switches and VM servers and Windows machines and Linux boxes. And a lot of the stuff that's in these OT spaces are really just IT equipment that's physically in an OT environment that's doing an OT function.

Aaron Crow (26:22.574): And then yes, there are also OT specific devices from PLCs and, you know, control processors and IDs and all that different type of stuff. But those things even are getting more and more IT aware as, as we go and we get these next, you know, industry, industry 4 .0 type stuff. So thank you for listening. Please like and subscribe on all the different podcast places, YouTube, et cetera. Definitely share it, reach out with any questions or concerns.

Aaron Crow (26:52.174): you know, if you want to be on the episode, on a podcast, definitely reach out. We're always looking for good content and good conversations to have. Thanks a lot and have a great day.

Transcript lightly edited for readability.