Compliance Isn’t Security: NERC CIP 15 and the Real Gaps in OT Network Monitoring

Aaron Crow (0:1.560): Hey everyone. Welcome back to Protect It All podcast. I'm your host, Aaron Crowe. Today I want to dive into something I've probably talked about a bit. But it's something that as it gets closer to time, more and more folks and organizations are going to be fighting for. And honestly, it's something that I think in general, we've been kind of dancing around and not actually

Aaron Crow (0:31.502): hitting it directly. And I'm talking about network monitoring in OT environments. Specifically, east-west traffic. think we've done a lot. Organizations have done a lot as far as north-south where we're talking about, you know, I'll put a perimeter firewall, even firewall if I have a segmented environment, segmenting or monitoring that traffic that's leaving a certain environment or zone in an area.

Aaron Crow (1:1.014): But it's that East West, right? So things that are on the same switch, in the same subnet, in the same network. How are you monitoring those things? And in an IT organization, you have, you know, NACs and things that are monitoring, which MAC address is this? Does it have a policy? Does it have antivirus? Like all that kind of stuff. We're not necessarily doing those in our OT spaces. So what's happening inside of your...

Aaron Crow (1:29.242): NERC CIP calls it electronic security parameter. There's a blind spot, right? There's usually this monitoring or data that we see from the firewall, but everything behind that is kind of like a dark hole of just not exactly knowing. Like you'll get some stuff that's reporting out, but you're not really getting everything, right? So the problem is, is that your most critical stuff, right? That's the critical systems that are in those spaces. So NERC...

Aaron Crow (1:56.862): with electric providers, power utilities, transmission distribution, NERC CIP 15 is really focused on attacking that, right? And by attacking, mean going after, knowing that's a gap in our security posture and how do we dive into that? So what is that north-south, I call it a trap of feeling like you've got

Aaron Crow (2:23.662): monitoring or what you do, but you're limited, right? So when I go into most OT environments, power plant substations, manufacturing facilities, and I ask about if they have monitoring, let's say that, you know, obviously many don't even have anything. But if they are doing any network monitoring, nine times out of 10, you know, they're monitoring at that north south, right? They're only getting things that are traversing through a router, through a firewall. And that's

Aaron Crow (2:52.014): That's the extent of it, right? They've got IDS, they've got what's going in and out of that ESP, and that's a different section in NERC, monitoring at that ESP, electronic security perimeter. And it's been the standard in power utility and other manufacturing and other OT spaces as well. But what happens when things are...

Aaron Crow (3:17.486): beyond that, right? What happens when something doesn't go through that path? Like I can't tell you how many times I've walked into these spaces and I'm not specifically talking about NERC SIP, but just in an OT environment where they have a firewall and they are monitoring at that firewall, but then there's a path around it. Like they're bringing in a laptop. They have remote access that they directly plug into the OT devices that bypasses all of that stuff.

Aaron Crow (3:42.690): you're really flying blind because you don't know those things are happening. You're not monitoring those things. You don't know to look for those things, right? And this isn't theoretical. Like we've been watching this play out for years. You Ukraine, pipe dream, there's all sorts of things, examples of attackers who have, you know, knocked on the front door and gotten turned away, but found other ways to get in. That's the risk of, you know, living off the,

Aaron Crow (4:11.840): off the land techniques that these guys are using. using, as we know in these spaces and OT spaces, we don't have encryption. We don't have a lot of those things that you would have in an IT world. So if I'm inside the protected space, I have carte blanche to do what I want. I can send a command to a PLC and it is going to do what I ask it to do. It's not usually asking for credentials. It's not asking for authorization.

Aaron Crow (4:40.298): It assumes if I'm speaking its language and I ask it to do something, it's going to do it.

Aaron Crow (4:47.630): There's one side of me that says, or a lot of arguments in the space of, we need to encrypt everything, we need to lock everything down. I don't necessarily disagree with those things, but there's a lot of reason why that's a problem in an OT space. But at least we need to be monitoring in these spaces to understand that, you know, we know that the MP3 Modbus, a lot of these things are open protocols. How do we know that somebody's in that space and these commands are valid?

Aaron Crow (5:15.852): your IDS is not going to be triggered by, you know, somebody sending a correct command to an HMI. It's not malware. It's not virus. It is actual commands that should be sent. It just isn't being sent by somebody that should be sending them. And if you're not monitoring that East West communication, that internal network of things that are plugged into switches, things that are in those spaces, engineering machines, engineering workstations, et cetera.

Aaron Crow (5:41.510): PLCs, HMIs, historians, know, engineering workstations, you're not going to have visibility into knowing what those things are, right? So you can't tell the difference between an operator doing their job and a bad actor, you know, doing something they shouldn't be doing. So specifically NERC SIP 15, for those of you in the power utility space, you're probably very familiar with it. Those of you that are not in,

Aaron Crow (6:9.122): the power utility industry, NERC and FERC, there is a regulation around the power utility industry. NERC SIP is kind of the implementation of those things and the, these are the rules that you have to follow to be compliant. Everybody knows what compliance standards are, whether PCI, whatever. It's very similar to that, right? So in that, it is a compliance standard that is held against these organizations. Those things...

Aaron Crow (6:37.806): they are a good start. let's start there. A lot of OT spaces don't have that regular regulation side of the, of the, uh, arm. So you look at water utilities, there isn't any regulatory body that is pushing, Hey, you must do these things. Um, kind of same thing in manufacturing, although there's, know, I, know, 2,700,000, one there's 62443, um, in, in the UK there's, there's NIS2 there's, there are standards out there. I,

Aaron Crow (7:6.862): Obviously you've spent a lot of time in NercSIP, so I have that as an example. I also think it's pretty far advanced as far as the implementation. It's been around a long time. They've been constantly improving it. But to be really clear, there's a lot of noise. It hasn't been implemented yet. There's a lot of people. Compliance sometimes can be gray, meaning I interpret this to mean this. So there's going to be a lot of vendor-driven confusion of, yeah, our product will do this. It'll do that.

Aaron Crow (7:36.978): and it's going to have to be kind of played out with auditors and with the regulatory bodies to say, this is what we're looking at. Does this fit? So what are the requirements? The requirements are saying that you need to implement network data feeds. It's going to have to use like a risk-based rationale to monitor network activity inside your ESP. So you're going to have to really look at connections in devices and communications.

Aaron Crow (8:5.272): You're have to be collecting data that gets into that visibility of what's going on behind the firewall on that control network, that OT network. And many times at these spaces, there's multiple of those networks. I've got various switches spread across my environment. I have to know if somebody's plugging into them, if there's malicious activities going on with those things. It can get really complex, really, really fast. As of right now, it is only being required at medium and higher facilities, so a low impact NERC site.

Aaron Crow (8:34.488): today would not need to have this, but it is really good practice. It's just going to be, you you look at some of these, won't call them out by name, but you look at some of these larger, you know, utilities that have thousands of substations across the country, know, hundreds of power, you know, power generation sites. Those are, it's going to be really tedious to deploy this at scale across your organization. And then even after it's deployed, how do you monitor? What does response look like?

Aaron Crow (9:4.398): It's not simple. That doesn't mean that we shouldn't be doing it, but it is absolutely not simple, right? So you need to be evaluating, you know, anomalous activity, what actions require, you're have to document those things. And the thing around a compliance program is I have to show proof of compliance, right? I have to show that I monitor these things. If I had something come up, what was my reaction? And I have to document those things. So there's this entire documentation side of a compliance program that has nothing to

Aaron Crow (9:33.902): quote unquote, nothing to do with compliance or nothing to do with security and everything to do with showing that I'm following the rules and I am compliant with this regulation. So that's that yin and yang of I can be 100 % compliant and not secure. I can be fairly secure and not compliant. Unfortunately in compliant organizations like this, 99.9999 % compliant is not compliant.

Aaron Crow (10:2.274): You are out of compliance, right? It's a hundred percent or nothing. So, NERC is saying, excuse me, NERC is not saying that you have to do full packet capture everywhere across your entire environment on day one. They're more so saying, deploy where the threats are, think about where the advisory behavior looks like. So, really deploy into your environment, make deliberate choices of where you're pushing these, pushing, placing,

Aaron Crow (10:30.862): these network data feeds and look at the bigger problem beyond just what is the technology that can do network monitoring. A lot of these environments, they have really old equipment. been in a lot of places that switches are 20 years old. You've got old school switches that yeah, they can do port mirroring or spanning, but they only support eight out of the 24 ports can be monitored at a certain time. So what are you doing that?

Aaron Crow (11:1.030): and that capability, are you putting in taps? Are you replacing the switch with a more modern switch that supports full mirroring? And then on the flip side of that, I've also seen spaces where older switches, when they're doing port mirroring or spanning, they're actually using the networking plane to do that, which means you're taking away throughput or cycles.

Aaron Crow (11:29.388): from your switch to do monitoring, which is a problem. So there's a lot of ways around that. We don't have to go 100 % into the architecture of those things, but this is why I bring these things up to just talk about why it's so complex to really dive into these things. know, SIP 15 is really applying to high impact and medium with external routable connectivity. If you're in a high impact facility, it is mandatory.

Aaron Crow (11:59.010): you know, good thing with is there are no high generating sites. It's going to be more, you know, control centers and things like that. But there are mediums, there are medium generating sites. Obviously there's a lot of substations and control centers. So backup control centers, you know, it sounds like there's a lot of this in the future. Just be ready. It's going to be deployed, I would assume everywhere. So, you know, what is the reality check is, that

Aaron Crow (12:28.502): I've not seen too many organizations sold on a tool that they think will solve the problem. INSM and network monitoring in these O.T. spaces is not easy. You know, I was a product vendor, every product in the space, it's not only a product problem. There are absolutely products that will do this. And again, I'm not trying to solution or architect this.

Aaron Crow (12:57.486): But the whole point is that you were going to have to find a product that understands OT protocols that can be in these spaces and determine what good looks like so that I can communicate when something bad happens. If you look at baselining, specifically talks about establishing a baseline for your network traffic in this NERC ZIP 15, it's one of the requirements so that you can detect anomalous activity. But in an IT environment,

Aaron Crow (13:27.630): There's plenty of tools because we know what HTPS looks like. We know what SSH looks like. Those protocols are, there's so many products that do those things. In an OT environment, obviously we have lot of OT network monitoring tools out there, but there's not as many as there are in an IT space. You've got obviously those OT protocols, DMP3, Modbus, TCP, Profibus, Ethernet IP, et cetera, et cetera, et cetera.

Aaron Crow (13:58.363): And each vendor is going to implement things in the way they do things and the PLC is a little different. So it's not just the protocol. It's about the function codes that are within them. Like there's just so many things that go into those things. And then the infrastructure challenge that I spoke about, you you go to a power plant or any of these spaces, I may have 200 switches deployed in all over the place and there is no, you know, fiber backbone that I can use to do spans or get all this data back. So

Aaron Crow (14:27.502): there's going to be a lot of infrastructure that goes into actually deploying these things, managing switches, span ports, network taps, you know, how am I getting these things back and monitoring them? And then you talk about compliance. I'm not going to put them on an IT network backbone. So it's got to be dedicated. So it's just like all of these problems just kind of compound on top of each other. Now I've implemented this in scale way before this is a requirement at power plants and it is absolutely doable.

Aaron Crow (14:55.000): but it is, you know, it's hardware dependent. The infrastructure, the architecture is a little complex. Once you get it, it's there, right? The cool thing is, is once you implement this type of, you know, monitoring on the network layer, then I can put in a tool that I want, right? So we did this to, we put spans off of every single switch, going into a tap, like Gigamon or Garland or, you know, any of Ixia, any of those, you know, network taps.

Aaron Crow (15:23.752): We really did that so that we would not break segmentation. But then you go into some kind of port aggregator, right? So I've got 200 switches. We did two spans on every switch. That means 400 spans, right? Going into a port aggregator. So then I could just have, say, a tool port. Hey, this is going to, you know, network monitoring tool. This is going to a SIM, whatever you're doing with it. Then you have a lot more flexibility down the road to

Aaron Crow (15:52.281): change tools to be able to add, remove, change without once that network architecture layer is done, all the rest of it becomes a little bit easier down the road, right? You're not locked into, well, I have to use vendor A on the tool side as far as the monitoring because I've already implemented, it's too hard to change them. And, I have to deploy a hundred sensors out in the field. If you do the architecture, in my perspective, you should do that architecture at the network layer so that you have

Aaron Crow (16:18.626): You don't have to have a sensor at every switch or at every, you know, substation. Like you could do that a little differently. You're still going to have a lot of implementation, a lot of architecture, but it's going to definitely look different. And in IT world, you're probably just doing a span on a core switch and feeding all of that to one place. It gets a little simpler. OT is definitely a little bit more complex just because of the risk of segmentation, the risk of the downtime that can happen when those things happen, right?

Aaron Crow (16:48.814): And then you do all that. Like we've monitored the network, we've got the tool, we've set it up, we've tuned it, we know what good looks like. We're monitoring for those things. Now the people problem comes into this. Like who's monitoring it? Who's looking at the dashboard? Like are there people that understand what good looks like? What are their playbooks look like? What do the run books look like? Right? So you can have the best tools and architecture in the world.

Aaron Crow (17:15.138): But if you don't have somebody sitting in the chair that's looking at it, that knows what they're doing, then it's going to be difficult. Like you're not going to get the same value. Is this normal DMP communication versus malicious ones? And again, some of these are not malicious DMP traffic. Some of this is going to be, we're not an outage and I know that this site would not be doing those types of commands during this window, without a window. know, did they trip? Did the unit go down and now they're doing these things?

Aaron Crow (17:45.255): But you have to understand that at a different level. And this is not a network architecture. It's not a cybersecurity understanding. This is where I talked about it a couple of weeks ago, where the difference between bringing in an operator, bringing in somebody with the engineering, operator, I talked about it with Dean Parsons, about how important it is to have those teams be co-mingled and really understand from an operational level.

Aaron Crow (18:11.863): what these things are doing, not just from a cyber level, what good looks like, but also from an operational level. Cause ultimately that's the real thing that you're doing. Otherwise you're going to be drowning false positives of, know, and you're going to miss the good stuff. You're going to be drowning in the things that you think are bad or not really. And you're going to miss the things that really are bad, but didn't, you know, raise a flag. you know, so kind of transitioning that to compliance versus security, which is the real conversation here. you know,

Aaron Crow (18:40.929): Anytime we talk about NERC CIP, there's always that compliance and security lens that you look at. One of my previous bosses, used to have this drawing. think I've mentioned it before. He gives me a hard time about it because he loved the drawing. I hated it at the time. I don't hate it as much anymore. I never hated it, but it communicated the point. Anyways, the document or the

Aaron Crow (19:10.585): presentation was two targets. So like think of a an archery target or a whatever, right? So a bullseye, two bullseyes. And one is labeled compliance. One is labeled security. I can be bullseye in both of them, but they don't necessarily overlap, right? So I can be compliant and have a big security vulnerability. I can be more secure and not meet my compliance requirements.

Aaron Crow (19:37.423): Right, so you have to consider both of those things when I'm doing them. So I can't just be compliant and check the box. can't tell you how many times I've walked into a site, like, oh, we just passed our NERC SIP auditor, whatever thing, so we're good. Like, okay, but how many things are in that? Like, if I go to a power plant, only a fraction of the actual devices are part of the compliance program. So what about the rest of the devices that I can get in from one of those devices that are not part of the compliance program, and then I can potentially impact?

Aaron Crow (20:7.055): So there's all these things to consider as I'm looking at them, right? So, and like I said earlier, I continue to say it, 99.999 % compliant is not compliant. 100 % compliant is not necessarily secure. So, you the standard of what FERC and NERC have determined minimally acceptable posture is not necessarily the best thing that you should do or the end goal of what it is. NERC SIP compliance or any of these compliance programs

Aaron Crow (20:35.671): from my perspective is the bare minimum. It is the bare minimum that you can do. It should not be your end state. Like that should be the, okay, we know we have to do this. How much more are we going to do? Where do we feel comfortable from a risk, from a business risk perspective and from an understanding my environment, what else do we need to do? That doesn't mean you have to go from zero to a hundred, but the goal in my perspective has never been that NERC SIP is the end. Like I do that, I'm done, right?

Aaron Crow (21:6.391): So, know, NERC CIP 15, you know, organizations really approaching this INSM network monitoring, deploying a tool, configuring some alerts, know, understanding risk base, understanding your architecture. But again, you really got to really make sure that somebody understands it's reviewing the data. What does it look like so that you can start alerting on those things? So what is that actionable steps? You know, make sure that you've assessed your network.

Aaron Crow (21:34.148): you know where your systems are in these spaces you're going to have again, NARC SIP, you're going to have already had, you know, understand what your assets are, what your, you know, high impact systems, medium impact systems, external routable communications, but bigger picture is, is really understanding where those segmentations are. what am I going to be monitoring for? and as you're doing is you're installing this, this, this plumbing for all of these things, it is so much easier. It's just like when you're running,

Aaron Crow (22:2.861): wiring for a house, if you need one, run two. Like if you're gonna go in this area and you're saying, hey, that's not part of my compliance program, so I don't need it, but it's sitting right next to it, monitor it. It doesn't have to be part of your compliance program. You can have it a different environment. Maybe you don't even turn it on, but have the piping there. If you're gonna run fiber, run extra fiber. Don't run one pair, run five pair, because the cost of the fiber is insignificant compared to how much it costs to run it. We know in these spaces you're gonna break fiber,

Aaron Crow (22:32.803): You're going to need extra stuff. You never know what you're going to need it for. know, so really look at those things, you know, assess your network architecture to make sure it's ready for monitoring. Look at the better way to do these things. How can I monitor inside of these spaces? Do my switches support spam ports? Like all these types of things that we talked about. And then the bigger picture is, is baselining what it's good to look like, right? So you're going to have to understand what these

Aaron Crow (22:54.883): you know, control systems, what the PLC is like, you're going to need an expert in all of those spaces. I go to a site that's got an Emerson and a, and a GE mark six and a, and you know, a third party controls. I'm going to have to understand each one of those, what good looks like. And I'm going to have different things at every site. the blue boxes, SELs like you, what is good look like in all of these spaces? That's going to be the bigger picture, the harder part. Again, we're not talking about only compliance.

Aaron Crow (23:22.383): We're talking about actually securing these spaces and knowing what good looks like so that I can lower my risk and protect these spaces. Visibility is amazing. I'm excited about where this is gonna go and how this can take us further in the space. But I also see that this is going to be a big hurdle that's gonna take a lot of work to get to. So anyways, all that said, I'd to hear anybody's perspective on NERX at 15. How have you been implementing it? What tools are you using?

Aaron Crow (23:51.011): What are the architecture strategies you're looking at? How are you, are you replacing switches? Are you adding ports and taps and all the things? Definitely reach out, let us know. It's definitely something that I think is gonna be fun and exciting over the upcoming years and it'll probably spread to other critical infrastructures beyond just NERC SIP and Power Utility. All right, I appreciate everyone listening.

Aaron Crow (24:18.438): Definitely dig in and let me know what you think in the space.

Transcript lightly edited for readability.