Understanding IT OT Convergence: Dealing with Challenges and Building Trust

Aaron Crow (0:4.568): Hey, thank you for joining me today. I want to talk about a topic that I bring up sometimes on the podcast and, and, but I really wanted to bring, have a deep dive into it. And it's really about, we've all heard the term ITOT convergence. It's called a lot of different things, but that ITOT and where the differences are and how they're getting smaller or closer together or whatever you want to look at. But I really want to talk about that and, and

Aaron Crow (0:32.834): the struggles, the challenges, as well as the successes, in, in that it OT convergence. and you know, why it's such a crucial conversations, why it keeps coming up. It's also one that people kind of avoid. Some folks avoid it, just because it's kind of a buzz buzzword, et cetera. Right. So, I've dove into it before, but obviously there's a difference between it and OT, the technology, the T's the same. The I is different. The O is different. the, roles are different.

Aaron Crow (1:3.491): Those roles are probably the biggest difference specifically around this topic in that it's really about...

Aaron Crow (1:19.062): your experience and your coverage, right? So an IT person is usually dealing, they're working in the corporate headquarters, they're dealing with traditional IT things, they have, you know, maintenance windows on the weekends, they have outages, they're patching, like they're looking at things from a different lens. The problems that are having, like if the server goes down, it's a different impact than if, you know, a manufacturing line goes down or a power plant goes down or

Aaron Crow (1:48.024): wastewater treatment plan or a train stops running, right? There's a difference in those things. It's the, again, the technology is this many times very similar. It's got routers and switches and firewalls and, you know, all those types of things. But obviously there are some differences, PLCs, IoT, different types of devices that are controlling endpoints, automation, et cetera. But traditionally

Aaron Crow (2:15.372): the IT folks don't necessarily understand the struggles in the OT space. So even though in my experience, many times the IT team is trying to do the right thing and they're trying to do good, some of the things that they may have tried to push or may want to push down into the OT space, patching, GPOs and active directory, just to name a couple.

Aaron Crow (2:42.722): They don't necessarily understand the implications of that, right? So one of the biggest issues I've seen and challenges I've seen in that ITOT conversions has been lack of trust. Where did that lack of trust come from? It didn't just inherently, hey, I don't like anybody, I don't trust anybody, because they do trust people. They inherently usually don't trust corporate and IT. And the reason for that distrust is because of historical reasons. In the

Aaron Crow (3:12.584): usually the corporate or IT organization has provided a service or tried to help and that help didn't work out. It ended up hurting more than it helped and they had to fight to either undo it or go around it or yank it out, right? One of the teams that I had when I was working as an asset owner in power,

Aaron Crow (3:35.318): was, you know, we always talked about, I'm gonna, I'm gonna give everybody shirts and it's gonna say, you know, I'm from, I'm from corporate or I'm from IT and I'm here to help as a joke because they don't necessarily want your help. And the reason for that is not because they don't want help. It's because they don't trust that you can actually help them. They feel that even though you're, you may be in that organization, may be very technically capable. They're not necessarily able to provide value in the space because they don't understand it. And the impact of that,

Aaron Crow (4:3.990): historical mistrust has been outages, right? You roll out a technology, a prime example, I talked about GPOs, you put Active Directory in and there's a computer that's sitting in a control room that is providing screens for the operator to see real -time awareness of the environment. And they drop that machine into a group policy that gets patched or it gets screen savers turned on and screens locked, et cetera.

Aaron Crow (4:31.534): in an environment that's 24 seven is always operated. That doesn't sound like a big deal, but it is when that operator doesn't know the login and it doesn't, know, the screen that they're, they're watching because there's a procedure going on just went to screensaver and they can't get it out. And they're in the middle of a process or a procedure that now they can't monitor because the screen that was up, you know, in the control room in a protected area is now locked because of a group policy change that it did. Right. So those are the,

Aaron Crow (5:0.434): pendulum swings drastically. Hey, I don't want anything from you because you can't even do basic stuff like this. And that that's a big thing to overcome. Those those failures are big and they happen. the problem is, is that I T, know, traditionally, again, I've worked in it as well. You know, it's Monday through Friday, eight to five, nine to four, whatever the numbers are. But you're working traditional hours, you're off on the weekends, maybe you have an on call schedule, but you're not there,

Aaron Crow (5:30.830): And those plants, those OT environments, many of them are 24 seven, especially if you're looking at critical infrastructure like power utility or wastewater, like you want your water and your electricity to work 24 seven, those plants are running. So there's operators there 24 seven. So if that thing breaks at three o 'clock in the morning on a Saturday, they're the ones that get called. They're the ones that are on call to show up and try to get that thing fixed. And usually they don't even have

Aaron Crow (5:58.498): the right person from IT or know how to get a hold of them to fix the thing. So there's some trust gap that has to be built. So many times coming in as an outsider, it's really hard to overcome that lack of trust. They don't know me, they don't trust me. Sometimes I can bridge that gap just because I've worn those hard hats and I can speak their language and they can see at least I understand why they are hesitant. But still there's a difficulty. One of the plants

Aaron Crow (6:27.298): that I started working at and we onboarded, we actually purchased the plant and there was a local person there and I'm sure he's listening. He may or may not be listening to this. And if he is, he knows exactly who I'm talking about. And we butted heads very a lot in the beginning. And it was because he didn't trust me. He didn't trust that I knew his environment and that I wasn't going to cause problems. That I wasn't going to make his environment less reliable.

Aaron Crow (6:56.654): and make his job difficult, right? He was the one responsible for it. And he didn't have the trust that I was going to that I knew how to help him without causing problems. So it took a while for me to build up that trust. So that gets the kind of the next section of the piece that I want to talk about is, is building that trust and collaboration. I've seen too many times, both as an asset owner, going and trying to push things down to power plants. Also as a consultant coming in and again,

Aaron Crow (7:25.516): these large scale programs, all good things. But at the end of the day, you have to get buy in from the, from the end user, from the plant manager, from the control system owner, from, from the, the engineer, from the operators, right? If they don't own it, or if they don't, if they don't get buy in on those things, they're going to have, they're going to push back. And ultimately they're the ones responsible and, have the, the authority to say, you're not going to do that in my place. Right.

Aaron Crow (7:54.922): How do you push that? Of course you can go up the chain and try to escalate and force it, but that's not the best way to do it. It's like forcing your kid to eat food by prying their mouth open and shoving it into their face. Maybe it gets the job done, but they're not going to like it. They're never going to learn to like it, and they're constantly going to fight, and it's going to take a lot of energy. It's not the right way. So how do you build trust in these environments?

Aaron Crow (8:22.786): The biggest thing I could recommend is going into these environments with an open eye. Listen more than you talk. Go in with with goals instead of prescriptions. What do I mean by that, right? Instead of saying, hey, I'm going to push Active Directory Group policies and we're going to lock all these workstations and we're going to patch all these systems because these things are insecure and you've got Windows XP over there, so we need to replace that and and going in with the know the 8th grader or 12th grade teacher, you English teacher, you know, writing red all over your

Aaron Crow (8:52.588): walk in and say, here are our concerns. How can we solve these problems together? Like what are ways the screen is unlocked? Our policy with corporate is, is that we don't have unlocked screens. Let's talk through why that's not an issue here or how we can approach that problem. Well, in this particular environment, that's in a control room that has multiple barriers of physical security to get into. And it's got a person sitting at that chair

Aaron Crow (9:22.286): 24 seven, 365, meaning that yes, they're not locked. So, but at the same time, not just a random person can walk up and start typing on the keyboard and take over because there's a person at that keyboard at all times. So if a random person walks in the control room and tries to put hands on a keyboard, it's not going to go well. They're, going to get escorted. The fact that they got that far.

Aaron Crow (9:47.678): says a lot of things, but it's very unlikely that they're going to get to the point where they're actually at a computer and they're making changes without somebody noticing them and physically restraining them, especially if you're in Texas at a power plant. So I promise that'll happen. But go walk in with a an open mind and being willing to listen. Right. The biggest difference, one of the big differences in it and OT is just how I approach the problem. The problems exist.

Aaron Crow (10:15.660): I have, I need to secure mode access. need to patch systems. I've got vulnerable systems. I've got, you know, legacy equipment. I've got all the same problems that I have in an IT space, but I don't always approach the problems or remediate the problems in the same way. So going to the plants, going to the environments and saying, Hey, these are the concerns that we have. Maybe we did an assessment. Hey, these are the things, the findings from, you know, the third party or internal audit that did the assessment. How can we approach these things and work together for a solution?

Aaron Crow (10:45.166): Because if you work together for a solution, then implementing that solution, you're gonna be more likely to be successful. You're gonna be able to check off the boxes that you're looking to check. It just may look different than how you did it on the IT side. And that's okay. Your policy for success doesn't have to be the same playbook. You're playing, it's a different game. Like you can't take a football playbook and take it to baseball and expect to win. It's just a different game.

Aaron Crow (11:9.838): It doesn't mean they're not both athletes. It doesn't mean there are similarities like they both need to be in shape or they need to be, you know, physically capable. Like there's a lot of similarities, but there's also a lot of differences. Maybe baseball and football is a little extreme, but you know, the point is, is that there's differences in those two things, right? so building that trust and I've had a lot of success in that. And the main success I've had with that is by going in, proving, you know, who I am and kind of my experiences and coming with a, with an open,

Aaron Crow (11:40.202): I and willingness to listen, hear their concerns and be able to respond to their concerns with, you know, resolution. Right. This is how I've done it in the past. This is how we've approached this problem. And if I don't know, say, I don't know. Right. And don't be tied to a particular way to solve a problem. Like you've got to do this or else. Like I'm going to force you to do these things because again, it's not going to work well. Like you're going to end up losing or

Aaron Crow (12:9.410): you're going to end up break it, you know, causing more strife in that relationship. That's going to cause it'll be problematic down the road. Maybe you win this one, you win the battle, but you lose the war. Ultimately you want to have a, an open dialogue with the, with these, you know, different groups so that it's not a one -time problem. I'm not going in and rolling out technology and I'm wiping my hands and I'm done now check. I've, I've secured the environment. It's, it's not, it's not that easy.

Aaron Crow (12:39.118): I'm constantly gonna have to go back in. They're gonna do control system upgrades. We're gonna need to upgrade and replace systems. They have new capabilities and technologies that are coming out or end of life. Like there's infinite number of things and the more that you're close with those groups, the better that you're gonna have long -term to have secure by design in the future. Systems that are already there, obviously they weren't necessarily secured 40 years ago. Security wasn't even part of the architecture.

Aaron Crow (13:8.322): or the engineering in a system. again, the closer that you build these relationships, the more that you're likely to have those things. Building those relationships, what does that look like? Obviously, it's really simple. It's building relationships just like you do in any way, right? You break bread, you buy lunch, you go have conversations, you go there without an agenda.

Aaron Crow (13:31.704): I'm not trying to make friends with you so that I can force you to do something. It's like when somebody friends you on LinkedIn and then the very first five seconds later, they're sending you a sales pitch. Just like, I don't want to connect with you for that, right? If you're not actually looking to connect and build a relationship, you're just trying to sell me on something and then you're going to go away as fast as you came in. I'm not interested in that, right? And these connections shouldn't be that way either. At the end of the

Aaron Crow (13:58.936): The way we should be looking at it as an organization is where we're different teams on the same or different players on the same team, right? We're all in the same ship. We're trying to go to the same goal. Yes, I'm wearing I'm on offense and you're on defense or or I'm I'm wearing blue and you're wearing red. I've got an eye in front of my title and you've got an O in front of yours. None of those things matter. Ultimately, in that we need to be realizing we're going for the same goal

Aaron Crow (14:27.746): We want to do it in a way that everybody wins. At the end of the day, that's the goal. We want to win. We want to be secure. We want to make sure that the site is available, that the operators and engineers and all the things work in the way that they're supposed to. And you want to be invited back. Like if you build those relationships, the funny thing is, is the next time the vendor comes in with an opportunity or an upgrade or a solution, they pick up the phone and say, hey, the vendors want to sell us these firewalls. But I know you guys have firewalls. Like, what do you think about these?

Aaron Crow (14:57.922): And you can be there to have dialogue and say, yeah, those are great, but what if we did these? What if we did this? No, you shouldn't do that because of these things like here, ask them these questions, or maybe even they start bringing you to the table. Ultimately, that's the best goal. The best goal in the world would be that you bring a representative from your firewall team, from your networking team, from your server team, from all those different environments to these meetings in the design conversations when you're troubleshooting.

Aaron Crow (15:26.446): My team was successful. One of the things I was most proud of my team being successful with is, you we supported 45 something power plants across the state of Texas. And we would get calls from the operators, right? From the engineers, the control system owners, the plant managers. They would have an issue and they weren't happy with the response or maybe support they were getting from the vendor. And they would call my team. And the reason they would call my team is because we built that trust and they knew,

Aaron Crow (15:55.274): You guys know about these environments. You know about the technology, the networking, all of these active directory, like all of those different systems. And those are some of the things that we're concerned about. Can you come and sit at this table and ask the right questions to the vendor to make sure that we're choosing the right options? So again, I say this a lot and it's something I feel very passionate about and that we move, know, business in general, we move at the speed

Aaron Crow (16:24.802): trust. And ultimately that just means that the faster that I can trust you, the faster we can move forward. Like if I don't trust you, I'm not going to move forward, right? It doesn't matter what you're selling. could be the greatest thing in the world, but if I don't trust you, if I don't have that trust, then I'm not going to necessarily listen and understand it because I'm, I'm hesitant because I don't have that trust and understanding as in that relationship. Right? So the faster that we can build those things and grow those things, the better that you're going to be able to go

Aaron Crow (16:54.092): All that to say, thank you for listening. Definitely reach out. We're going to be at Black Hat and Def Con in Vegas in August. So definitely come reach out, say hi. We'll have some swag, maybe some t -shirts and stickers and all that kind of fun stuff. Come drop by. Morgan Franklin will be at the House of Blues during Black Hat and I'll be there as well as at Def Con supporting the ICS Village.

Aaron Crow (17:22.088): which I love being a part of. DEF CON's a lot of fun. It's hacker summer camp. So be there with and having a lot of conversations and seeing all the interesting people and technologies that are there. So thanks a lot and until next

Transcript lightly edited for readability.