Securing Remote Access in OT: Visibility, Segmentation, and What Compliance Misses

Aaron Crow (0:1.432): Hey everyone, welcome to another episode of the Protect It All podcast. This time I want to talk through all things connectivity in the OT space. One of the main issues that we normally see struggling with, obviously outside of asset inventory, visibility, is remote connection.

Aaron Crow (0:31.207): and because we have so many older systems, unpatched systems, we don't necessarily know all the systems, vendors hands are in things and lots of connections and backdoor backdoor is probably not the right word, you know, alternate paths into and out of our environments. controlling access is vitally important to these environments.

Aaron Crow (0:58.396): obviously a lot of, there's a lot of expertise in engineering that goes into these spaces, designing these control systems and, PLCs and, all of these very, very specific environments. So we really many times are leaning upon our vendors to be able to help us to support, to provide that level of expertise when we're having issues, all of that type of stuff. Right. So, that, that leans heavily on.

Aaron Crow (1:23.850): really being able to protect that. you know, if you had to look at this CSF or, you know, the control areas, one of the factors or the stronger areas that I believe that we should most organizations need to really focus on and be strong on is that secure remote access. It's that connectivity to these spaces. How can I, how do I allow people to access? And that doesn't necessarily mean remote being, you know, at another, you know, physical location. It could be.

Aaron Crow (1:51.606): you know, even just how my technicians are accessing these networks. Are they bringing in transient devices? Are they plugging in their laptops? Is there a dedicated device? Are they connecting from their office that's across the street? Are they able to connect from the corporate headquarters that's down the road in another city? Can I connect from another country? Do I have third party vendors that are connecting in? All of these things are factors that cause impacts or could cause impacts to these environments and beyond just cyber issues.

Aaron Crow (2:22.092): There's many of these issues and this goes back to, need to understand what's happening in my environment. Like when we walk into a place and something stops working, I literally just got out of my son's truck and it wouldn't start. It's an older vehicle and it wouldn't start. So what's the last thing that's changed? That's what I'm looking for. Who made a change? What's the biggest impact that could have caused this thing? And let's start reverse engineering. Where did I get to so that I try saying, hey, if,

Aaron Crow (2:50.636): he just installed new lights on his truck, maybe he forgot to connect the battery or anything like that, right? So it's really just being able to look back at those things. So specifically today, I wanna talk about CESA and the UK National Cyber Security Center and along with a bunch of other different entities have jointly published practical guidance on secure connectivity principles for OT.

Aaron Crow (3:19.054): They offer an eight point framework to design, secure, manage connectivity and OT environments as organizations face raising business, blah, blah, blah, blah, blah, blah. Right. So what does that mean? Like they're really giving guidance because obviously someone else agrees and believes that this is an issue and a risk and something that people need guidance on. Because we know that the days of 20 years ago, when I started having these conversations and these OT spaces, that it was always just like, well, we'll just disconnect.

Aaron Crow (3:47.288): we're not going to allow remote access into our environment, you've got to be on site to be able to touch that. I think that has changed, obviously, over the years. know GE is very large. They have their Atlanta Data Highway as part of their control system, and they did a lot of monitoring across that network, which was value add to their customers. But that also meant that they were able to access that from across the country, across the world, depending on where your facility was located.

Aaron Crow (4:14.446): And yes, you want GE doing that. You don't want a bad actor being able to do that or even a disgruntled employee or even just something accidentally happening, right? So what does that mean? So the eight sections, it's really around, just like we harp on and I even started this conversation with, the design connectivity for documented business seed.

Aaron Crow (4:41.248): What does that mean? Right? I really need to understand my environment and make sure that I'm not just doing an any, any, I'm not just allowing access from anywhere to anywhere at all times for all users. It sounds obvious, but many times when we're walking in these places, they have something that was set up five years ago. They don't exactly know why it's still there. They're hesitant to turn it off because they don't know what's going to break. The problem with that is, that

Aaron Crow (5:8.230): anybody could be getting in and add on to the fact of that that we may or may not be monitoring those environments or that network connection or that device, then that adds more problems, right? So, you know, want to, by designing connectivity and reducing that, you you're reducing your unnecessary attack surface. I'm not just allowing RDP from anywhere on my corporate network or from anywhere even just within a certain subset.

Aaron Crow (5:34.634): It also limits who is able to get in. Like even when I was an asset owner and I was the one responsible for OT in those spaces, my team would call the local site before we logged in and made changes. And the way my leadership explained it, Scott Rosenberger, if you're out there and you're listening, you you don't walk into somebody, at least not in Texas, you don't walk into somebody's house and start eating out of their fridge without letting them know you're coming over.

Aaron Crow (6:2.826): That would give you a probably a bad experience, especially in Texas. you know, reducing unnecessary tax surface and not allowing people in, you're not making changes that nobody's aware of. I don't know what changes are going on at this site. So I don't want to be rebooting something when you may be in the middle of an outage. You may be in the middle of troubleshooting something. I just rebooted your system. It sounds innocent enough, but that happens and that can cause problems, right? So all of those, all of those

Aaron Crow (6:31.564): connections need to have a valid documented business justification. Who is owning it? Who's responsible for it? Why is it there? Is it open all the time? When does it turn off? is it, it should time out, it should expire. I shouldn't just have this RDP connection open all the time. The best case scenario is I turn it on, I let the people in, do the work, and then I turn it back off like a work order.

Aaron Crow (6:55.030): It should have a review date, there should be approval, who can approves it, should be in your change management documented, like everything should be documented, right? But that all leads down to, you need to build and maintain a definitive OT inventory. Before you enable access, you wanna understand what they're getting to, because if you don't have a inventory, and I'm giving access to system A,

Aaron Crow (7:24.224): And I don't really understand my network and what other systems that they're giving access to system A could mean that I'm also giving them access to system B, C, D, E, and F, which could cause problems. Obviously pivoting, could be on a system they're not aware or meaning to. They could accidentally connect to things like, there's just all sorts of things, right? So you really need to secure or let me rephrase that. can't.

Aaron Crow (7:51.534): I hate to say can't, know Dale Peterson doesn't like this, you can't secure what you can't see. You can secure things you can't see, but it makes it really hard. It makes it much harder to secure an accurately, effectively secure an environment that I don't know what's on it. Yes, if I just turn everything off and I don't allow anything in and I'm not, no traffic is coming in like in a nuclear environment that's completely, really only air-gapped place that unless they have like a data diode, like a waterfall,

Aaron Crow (8:21.048): there's no data coming in and out. Yeah, I don't necessarily have to know what's behind the fence. But most OT environments are not like that, right? They are connected. They are allowing remote access. They are allowing pie data out. They're allowing, you know, they're patching, they're getting data from other places. And, you know, they're updating systems and servers and databases and all of those things are inherent risks. So you really need to understand them, right? So a continually updated

Aaron Crow (8:50.122): Record of assets, firmware versions, flows, ownership, roles and responsibilities, support contracts, those are all, they sound so simple and that it's not sexy and it's not AI, but it is the thing. Probably one of the biggest things that people can do is manage and understand their environment, their assets and their network sections, right? You wanna minimize and isolate connections.

Aaron Crow (9:19.338): I'm a very, very strong believer in segmentation and least privilege. You need to isolate environments as much as you can. My system A, micro system, target control system doesn't necessarily need to be connected to my balance of plant. Maybe there are some connections that need to be there, but they need to be intentional. I shouldn't just automatically connect everything on network A to network B.

Aaron Crow (9:46.966): And again, this is not, this is just good network architecture. This is just good system design, even beyond just cyber and risk of a cyber attack and ransomware and all of that. It's just a good idea. I've got many stories and I've told this story. I don't know if I've told it on the podcast, but you know, prime example of a site where they were doing a control system upgrade on one unit and unit.

Aaron Crow (10:12.970): Unit one and common were online. Unit two was offline. They were upgrading unit two, but it had active directory and it was all one big shared network. Long story short, the upgrades that the vendor did to unit two impacted unit one to the point that the operators had to punch the unit out because all of the data on their screens in the control room went blank. They had just smurf data, nothing, right? The plant was running fine, but they couldn't see it or control it in their theory or

Aaron Crow (10:42.382): what they could see so they turned it off until they figured out what was going on. So there's a prime example, it had nothing to do with cybersecurity. It wasn't a nation state, it wasn't a bad actor, there was no ransomware, none of that. It was just it impacted operations. And for a plant manager or for a site, they don't care. It doesn't necessarily matter if it was China or if it was some third party country or some bad actor, did it impact my environment? Was it an OT attack? It doesn't matter, did it impact me?

Aaron Crow (11:13.695): you know, V lands are good. Do you really want, you know, ACLs? Yeah, sure. Firewalls are better. Zero trust. Micro segmentation is, is, is best in some scenarios. but you know, you really need to understand when you're building out your asset inventory, you need to understand the things that are critical, what their job and their role function is so that I can make sure that on those critical assets, I'm understanding them, right? Colonial pipeline. You need to understand where your OTE assets are, which ones are critical.

Aaron Crow (11:40.462): and which ones can go down or have an issue. You need to have authentication and authorize all of those connections. You wanna require strong multi-factor authentication, know, dual factor, YubiKey's or tokens and you know, whatever your preferred method is, you wanna have some kind of MFA for accessing these environments, right? You wanna take away the, you know,

Aaron Crow (12:7.884): the vulnerability of passwords being your only protection in these spaces. Especially because most people are going to use the same password in their home life as they are on here, but just because they have so many passwords to remember. And if you're not using a password manager, which you all should be, then it makes it really hard to remember all those passwords. you you're going to prevent credential use.

Aaron Crow (12:33.642): simple brute force attacks, you lateral pivot attacks, just because I have access on this system doesn't mean I have access on that one. know, certificates and session tokens, jump servers, where you want to record sessions, recording sessions get you a lot of things, but you know, some level of understanding what happened in my space, I can't tell you how many times a vendor would log in, make a change, something changed, we weren't sure what we call the vendor, I didn't make any change.

Aaron Crow (13:3.626): And then we go back to the logs and say, yeah, who made this change right here? And we found out, hey, that was Bob. Bob made this change. Why did Bob make this change? May not be a bad thing, maybe a good thing. He just didn't tell us or update the work order or whatever that may be. You want constrained and direction limited communications, right? I, I, you not usually in all, most of my architectures, I'm not going to allow somebody from outside my corporate network

Aaron Crow (13:33.314): to initiate a connection directly all the way to the OT environment, right? I'm going to have some level of relay in between there. They're gonna have to authenticate into my network, multi-factor VPN, and then I'm gonna hit them with a jump server that's going to then authenticate through different credentials from the outside and the inside, know, some level of, you know, delegation of duties in those spaces, but I definitely don't wanna allow something way out to come in.

Aaron Crow (14:2.198): A lot of these newer, really amazing, secure mode access solutions enable a lot of these things from having the password management to having credentials and certificates and recording sessions and a lot of these things. Not to mention that they're splitting protocols, meaning they're not using, it's not RDP from the internet, you're not connecting your laptop directly to it, it's a protocol break.

Aaron Crow (14:30.542): you know, HTPS on the outsides, you're accessing it from a webpage and then it's doing RDP directly from the backend, but you don't have the ability to copy and paste files directly onto the desktop. So it's harder to pivot. I've got to go through a system which it's actually scanning the files to make sure they're not malicious. It doesn't mean it's going to find it because it's a white list, black list. But the point is, it's making all of these things harder. The struggle that I've seen in a lot of these spaces is

Aaron Crow (14:59.788): because it's also harder for the good guys, right? It's adding a step, it's adding a little bit of complexity. So we have to make sure that we're not over-engineering it so hard that the normal day-to-day actions are so difficult that I can't do my job, the engineers can't do their job, the operators can't do their job, because then they'll just find workarounds. And we don't want that thing either. So you want constrained direction-limited communications, but it has to be able to work.

Aaron Crow (15:30.252): We talked a bit about it, but you wanna make sure that you're doing visibility, logging, telemetry, session recording. Those are all super important things for obvious reasons. The session recording is huge. Again, if nothing else, it can even help with documentation. Hey, I've got this vendor coming in to make these changes and update the system. I can document all the steps that he took so that the next time this happens, I've got a documented process on how to do those things.

Aaron Crow (15:58.508): and update my work orders, update Maximo or whatever my system is. All those things are great. You want to make sure that your provisioning and vendor channels have firmware and you're updating your firmware in a right way. Like you're not just randomly going. Most, most, many, I won't say most,

Aaron Crow (16:25.358): of your VARs and vendors are going to provide you a secure place to be able to go get your firmware, right? They're not just going to drop it. You you're not going to have to just go to the random website and get it. Like they've got hashes that you can confirm and go through. You know, all of this is to make sure that your environment stays your environment and it's not impacted. The bad thing is, is if somebody gets in,

Aaron Crow (16:58.134): It's really hard to get them out. hard to see that they're there and it's hard to get them out before they do bad things. There's multiple scenarios where there was no secure mode access, somebody brings in a transient device, their own laptop, whatever, plug it in, now they've got a connection going out. Because it's really easy to monitor everything initiating from outside in and block things. Many times I've seen firewall rules where they don't allow anything in, but they allow almost anything out.

Aaron Crow (17:25.990): And that's where those communications can be established from inside, get reaching out. So we need to make sure that we're monitoring those things. So all this to say, it's a good process. They're digging into, limiting exposure connectivity, centralizing network connections, using standardized and secure protocols, harden your OT boundary, limit the impact of your compromise. Like all of these things are things that we talk about all the time. So it's not.

Aaron Crow (17:54.392): It's not rocket science what they're talking about here, but it also goes to show that these are basic principles that there's still so many that are not have not implemented yet, at least not consistently, right? So how do you guys feel about these types of things coming out from government entities? Do you think it'll help? Does it help you with your conversations? I can see this being a good thing to come across to be able to rise up to justify scope.

Aaron Crow (18:23.434): resources, et cetera, to really help to push these narratives through to get budgets aligned and assigned, you know, to justify spending and resources and all of those things. But my fear is, is that a piece of paper or something like this alone may not be enough, right? Does there, you know, in the power utility world and on the US side, NERC SIP has come along and yes, there's a stick.

Aaron Crow (18:51.790): There's a fine, there's an impact, a negative impact, financial negative impact if things aren't done. So does it need to have that? Some will say yes, some will say no. I can argue both sides of that coin. I think there's a place for it. also, I don't love compliance because then I think it's very easy to miss the force for the trees that I'm compliant, but I'm not secure. It's a slippery slope for sure. But I definitely believe that we have to do

Aaron Crow (19:20.202): more and do something to be able to push this through. So, I'd to hear what you guys think. If you all think this is a good thing, if you disagree, if you like the wording, if you think it's too vague, if you don't like that it's so pointed, what are your thoughts and feelings on it? Make sure that you like, subscribe, all the things, leave comments, always looking for folks, topics like this one. appreciate Perry for sending me this to have a conversation around.

Aaron Crow (19:50.147): Definitely an interesting one and definitely one that I'd love to see what the response from the industry is and how this actually helps or hinders depending on how people look at it from this point. So thanks a lot. Until next time.

Transcript lightly edited for readability.