Navigating Cybersecurity Challenges: AI, Tabletop Exercises, and Operational Technology

Aaron Crow (0:2.434): Awesome. thank you for joining the podcast protected all, today this is a cybersecurity awareness month. So we wanted to kind of dive into some basic topics around cyber cyber hygiene and some of the things that are in the zeitgeist of cybersecurity, for this month. So, today I've got with me Mike and Clint, Mike, why don't you introduce yourself, tell us who you are, and then we'll kick it over to Clint.

Michael Welch (0:26.924): Yeah, thanks, Aaron. Yeah, Mike Welch, Managing Director at Morgan Franklin Cyber, focusing in the utility, power gen, oil, water, gas, critical infrastructure. Looking forward to this conversation with both of you guys today.

Aaron Crow (0:41.540): Awesome, thank you. Clint.

Clint (0:43.966): Clint Bowdungeon. I'm a director at Morgan Franklin Cyber, director of cybersecurity innovation. also the founder of ThreatGen and specialize in OT as well as now AI.

Aaron Crow (0:59.494): AI, everybody's loving talking about AI and all the things. mean, Clint and I have even had, you he didn't fund conversations around the use case of it and it's coming, right? So, I mean, honestly, let's just start there. Like, how are we using and how can we see the use of AI in both enterprise and, you know, more so on the OT and critical infrastructure side? How do you see that being used, Clint, from a...

Aaron Crow (1:27.696): from a perspective of people that maybe have a limited to no OT program, maybe they don't have anything and they're just trying to get started, how can it help them? And then maybe the more advanced folks that have a program and they wanna kind of get that, they've got the 80-20 rule and they've already done the 80%, but they're trying to get that last 20, how can it help those guys? Because I think it's a different use case depending on where you are on that road.

Clint (1:49.746): Yeah, it is. And first of all, I'll clarify that, you know, at this stage in the game, I don't think anybody should be using AI to make final decisions and especially not making any changes in any type of configuration firewalls or anything like that, right, especially OT. But I think the two use cases you'll see are from basics. think Genitive AI when used properly and

Clint (2:19.400): securely and privately to where you're not spewing your private data out to the cloud. Exactly. You know, you don't want to stay there. Exactly. Right. We're not talking about chat GPT here for private data, but you can do it with like, we'll talk, can, maybe we'll talk later about, can do it locally. but from the basic use case, I do think that generative AI has, and we're talking about specifically here, generative AI.

Aaron Crow (2:25.062): That's a key there. We need to make sure that we, you don't copy and paste in this chat GPT online.

Clint (2:48.486): and not like the traditional machine learning kind of thing. So that's the hot topic right now. So generative AI can really help when in your basic process with helping you analyze data, helping you understand what decisions to make, right? It can look at data. Generative AI is really kind of a superhuman when it comes to data analysis. That's its superpower, right? And that's how it enables basic users is I have all this data.

Clint (3:17.924): And to your point earlier, kind of talking in the green room, my data is key. If you don't have the data, then AI will make its own data up and you don't want that. Right. So having the proper data, it's going to help you analyze the data assets, whatever. And it's going to help help you make informed decisions. Right. Where do I need to put my blocking and tackling? What do I need to tackle next? What vulnerabilities do need to tackle? That sort of thing. From an advanced perspective.

Clint (3:45.360): It's really the same thing in terms of the use case, right? It helps sock analysts. It helps those advanced, more advanced, more technical folks analyze data and help supercharge your sock program and analyzing data, analyzing threads, analyzing threat intelligence, because that's what it does. It analyzes data much like a human does, but it can do it quicker, faster, more efficiently. And in many cases, not all cases more accurately. And then there's a case in the middle.

Clint (4:15.036): to where generative AI can help your GRC program. It can help you with that heavy lifting that everybody hates, creating your documentation, right? It can help you create your cybersecurity policies, analyze your cybersecurity policies, help you create your incident response plan and all of that. And then, you know, because that's at least your first draft, right? It'll help you give you your first draft and then kind of the edge cases are it can help you with your cybersecurity training, your cybersecurity awareness.

Clint (4:43.558): And it's in response training like tabletops, are in, I'll just touch on it, but tabletops is a pain point for a lot of people and AI, generative AI can help you make your tabletops more approachable.

Michael Welch (4:56.814): So, Clint, real quick there, sorry, Aaron. When it comes to the data and utilizing the GEN.AI, how do we determine the accuracy or the appropriate, making sure it's not biased in any one way or an organization that's starting that journey that they're not violating their own policies in a sense? Because that's where we see a lot of startups and new companies that are trying to jump on this technology, but they don't understand the risks that may be there with that cut and paste or whatever.

Aaron Crow (4:57.006): Yeah. No, yeah.

Clint (5:26.676): Right. Yeah. So number one, first and foremost, the key rule is to always keep the human in the loop, making sure, you know, trust, but verify or don't trust and verify. Right. And so making sure that you're keeping the human in the loop and validating that data. It also goes back to the training process. Make sure that you are feeding the training in the fine tuning process and training and fine tuning are different things, but fine tuning was most people are going to do or providing that retrieval augmented generation of a library of documents for it to use.

Clint (5:55.390): So providing it the necessary data for fine tuning and in your rag system is going to help maintain that accuracy. So feeding it the proper documents, keeping the human in the loop. And then now you can use an agentive process to where it double checks itself. So there's a process to where you feed it the information, you get the information back before, but before it gives it to you, it feeds that data to another agent and says, hey,

Clint (6:22.258): double check this for me for, know, fact check this for me. And then it'll do things like go out to the internet for fact checking. It'll fact check the documents that you put in the document database. And so these are the typical techniques for maintaining that lack of hallucination, if you will.

Aaron Crow (6:41.338): Yeah, I mean, you we've all, we've all heard the, the, the stories of, know, lawyers that are using it. and then it comes back and has all these citations. And then you actually look up the, cases that it cited and they don't exist. It just created them. I was actually listening to a podcast yesterday where they were, they were talking about voice and AI voice, but ultimately underneath it is it basically, they had two AIs talking to each other. and they were making things up.

Clint (6:55.198): Right. Yeah, that's the loop.

Aaron Crow (7:8.550): and they were having conflict and resolving conflict, but a lot of the stuff is just made up. Like they were calling call centers and they would ask for like a credit card number or your social security number. And it was just making numbers up like nine, four, nine, eight, seven, six, five, four, three, two, one, five, five, five, five, five, right? And the customer service person was like, that's not a card. Like that's not a real number. And it had no idea, cause it was just throwing out numbers. It didn't have any of that fact check thing there. So it was just.

Clint (7:32.552): Yep. Yeah.

Aaron Crow (7:38.178): It sounded real. it gave numbers cause it knew it had to give a number, but it didn't actually have a discover card or a master card or whatever. Like it wasn't an actual account number.

Clint (7:47.526): Yeah, and due to the reason or the due to the way that generative AI works, that's its nature. If it doesn't have the hard data, it's going to make up data because it just predicts what that data should be. And that's why, you know, you shouldn't be using AI in any official capacity if you don't understand how to use it and apply those techniques to minimize hallucinations and minimizing and even alleviating

Clint (8:16.712): For the most part, almost completely hallucinations is possible, but you have to know what you're doing before you use it.

Aaron Crow (8:20.486): Sure. Yeah. And that goes back to the conversation, like you said, we were having in the green room where, you know, I really struggled to see how much you can use AI beyond advising on, you're missing this, you're missing that. Because as we walk in, know, Mike and I've walked into a hundred of these places and they don't have any idea from a data perspective.

Aaron Crow (8:42.871): what assets they have, which assets are critical, how their environment looks, not at the high level that you would need to be able to give that. So you could feed stuff into AI and they wouldn't know the answers came out were true or not, even the person, right? So if the person can't tell if it's true, how can we expect the AI to understand if it's true? So there is no check and balance. So the very first thing you've got to do is understand what is true. And then I can start using AI to enhance that to get me multiple capabilities.

Aaron Crow (9:11.748): to some more of those advanced use cases.

Michael Welch (9:13.974): Yeah, like even to your point, sorry Clint, to your point where you said AI is its superpower, right, is just intelligent. But it still always will need some level of human interaction, judgment to validate because without that you're just asking for challenges and risks.

Clint (9:14.118): One that we need.

Clint (9:30.620): Yeah. And that's the key point, right? Is that you don't have generative AI to do someone's job for them. You use generative AI to help someone do their job better, which is why keeping the human in the loop is necessary. It's it's an AI companion, right? It's not it's an AI assistant, not an AI employee.

Aaron Crow (9:53.774): So the example I always use is that when I was in my freshman year college, I had to buy a calculator, a scientific calculator, TI-82, I think it was, right? This is back in the 90s, so it tells you how long ago that was. But in those engineering classes, I took university physics and calculus and those kinds of things. They would let us use these calculators, right? And those calculators were being used. I still had to know how to do the math.

Aaron Crow (10:20.378): the calculator, I could put in the formula and I could get the answer in the test. So like I'd have the question and I could put in the formula into my calculator, assuming I knew which formula to use and I would get the answer. But the test was not, did you get the answer? How did you get to the answer? And you had to show your work, right? So the calculator, I can't just put it in there and just expect it to do it for me. It was an assistant, it was a tool. It's no different than a cordless drill. I can use a manual saw or I can use a electric powered saw.

Aaron Crow (10:50.459): It's just a tool. can do it either way. I just have to know what it is. It's not gonna build it for me. I still have to do it.

Michael Welch (10:58.486): I'm curious, Clint, as a resident AI expert on this panel and across the industry as a whole, we're talking about security practitioners and organizations leveraging it. How do you see the bad actors out there using it to make what they're doing more efficient? Do you see AI enhancing their capabilities?

Aaron Crow (11:16.614): Good question.

Clint (11:18.952): Yeah. So there's already tons of articles out there where people are talking about not only proof of concepts, proofs of proofs of concept, but also actual use cases and enter, I guess reports to where they are using generative AI to create malware automate the process of creating malware. There are models out there that are

Clint (11:45.968): specialized and under the they'll take these base foundational open source models, and they'll fine tune them to uncensor them, and then provide them with specific cybersecurity and even malicious red team malicious, offensive documentation, training data and techniques, and build these models to specifically help with the malware creating process, right? And, and because the bad guys don't necessarily care about

Clint (12:15.294): how a bad decision affects my production, they are using it to help automate more and make decisions and to completely, I guess for the most part, create autonomous command and control servers and things like that. There are already cases of them doing this. And if you really think about it, it's for the most part, anything that we're on the professional side, so helping automate

Clint (12:45.074): red team testing OSINT that sort of thing. The bad guys are doing the same thing, but they're taking it to the next level and

Aaron Crow (12:52.068): Yeah. And they don't have the restrictions that we do because of fiduciary duty and, and laws and they're already breaking laws. So they don't care that they're doing that. Right. So if you, if you take the, the, the latest or the known vulnerabilities of these, of these systems and you pipe that into, if you know a system and you know which vulnerabilities it has, you can very easily pipe that into even just genitive AI and how would I attack this?

Clint (12:57.438): Bye.

Aaron Crow (13:20.644): what are the known vulnerabilities and what is the best way to attack this and pivot from there. It's going to help you, right? Even if you don't know anything about those systems, it's going to help you because of the general knowledge that's in the internet. And I can feed it, you know, documentation, can feed it, you know, all the vulnerability and alerts and the, all of that, the MVD, that database is good for us to know what we need to fix, but it's also good for the bad actors to say, Hey, I know you have this vulnerability and you have a patch.

Michael Welch (13:48.302): Yeah, well, I'm curious how these SOCs out there are going to be able to build their detection engineering rules to be able to identify attacks and alerts and stuff generated by AI activities versus a human, right? We know what to look for. We have rules built for detection engineering rules for what those normal processes, tools, tactics, and techniques. I'm curious how that's going to change when they're leveraging AI. Is it going to cover their tracks, make them more efficient?

Michael Welch (14:18.008): keep them under the radar where they're not getting the alerts. It's interesting to find out where that's gonna go.

Clint (14:25.108): Well, just remember that number one on the on the the blue team side, we're already using things like machine learning to help with the process and the sock. And but here's the on the blue team. This is both the scary part and the good part. The good news, bad news is this is the this is the least advanced and the worst that the technology is ever going to be.

Aaron Crow (14:50.918): Correct.

Michael Welch (14:51.438): Yeah.

Clint (14:51.801): You know, it's only going to get better and more advanced. So I'm not trying to throw FUD out there, but it's fact. Yeah.

Aaron Crow (14:55.802): And quickly, and quickly, like that's the other thing that I think is important. So I was talking with some folks around, you know, Tesla or not Tesla, but Elon Musk's team and many others, it's a race for honestly electricity because electricity is the thing that is currently holding it back because the more power they can throw at it, the faster it's gonna grow, right? So.

Clint (15:22.036): Yeah. And who was it? I forget one of the companies. I forget who maybe open AR. So I forgot who it was that they're like buying and reactivating through my island. Right. So they can have their own nuclear plant to power the AI. Right. I mean, that's how you get Terminators. You know what Terminators.

Aaron Crow (15:31.052): Mm-hmm. Now it's Elon Musk, team.

Aaron Crow (15:35.280): Yep. Yep. And, and, they're, they're, they're actually looking to build data centers around old decommissioned power plants that they can, they can retool and spin up. Because again, power is the problem, right? So throw away the battery technology and anything else they need, like they need power. And you know, nuclear power is obviously the cleanest and most efficient.

Aaron Crow (16:1.414): But, you know, they're looking at all, all sources of, of, of power. Solar's not going to do it. So they, they need, you know, gas fired turbines and they need coal fire power plants and they need nuclear to, to, to get these things going. And it's a race, right? Because China is building coal fire power plants left and right. And they don't care again, just like the bad actors, not necessarily saying that China is a bad actor, knock on wood. But you know, they don't care about the emissions stuff that we do in the U S right. So they're, they're just.

Aaron Crow (16:29.914): trying to win the race and that they pull all the restrictions out. they're building them federally where, for instance, in Texas, it's a deregulated environment and our power utilities are not able to build the power because how do they fund it? It's not a government entity.

Clint (16:47.420): I heard somewhere that people make good batteries.

Aaron Crow (16:50.244): Yeah, the matrix. So I've doubled down quite a bit on the AI topic. Let's shift gears a little bit, Mike, and kind of in an area that you and I've been in a lot. And that's, you know, we see a lot of needs in the space, and I see more and more regulation coming down in areas because

Clint (16:53.438): Going back to the whole thing. Yeah, yeah, exactly. Yeah.

Aaron Crow (17:15.226): people need to justify and validate and actually provide funding to do the things, the bare minimum, right? So we need critical infrastructure. So NERC SIP has been huge in this space, but we're seeing more with TSA pipeline and, you know, 62443 and a lot of these things that are coming out that are really pushing, you know, requirements down to these, these industries because we see, Hey, you've got to do something saying that, you know, I'm going to disconnect it from the network.

Aaron Crow (17:40.846): or put in a firewall and that's all I need to do to have a security system as we know it's not enough. So talk to me a little bit about the regulation environment as well as the difference between compliance and security and kind of how folks can kind of kick that off in whatever regulated or even non-regulated environment they may be in.

Michael Welch (17:59.362): Yeah, no, that's a great question. It's a great topic, especially like you mentioned when we look at power, utility, generation, transmission, substation for the last whatever 15 years, once we've been working together in different capacities. NERC SIPP, actually when it was the 1200 series, right? When it was optional and then it became NERC SIPP and there's so all these iterations of it, which is one of the few in critical infrastructure that have that level of regulation right now. Like you mentioned, oil and gas.

Michael Welch (18:26.338): Water doesn't have really any and you look at yesterday or a couple of days ago, they had the water breach. Maybe on the IT side, we don't know much about it yet. Compliance, I always looked at as a practitioner, I always want security, right? Compliance is that vehicle that you can use to get security put in place. As you know, everybody's fighting for dollars for budget and this and that. So how do we...

Michael Welch (18:55.202): get the money we need to put in security. A lot of folks use compliance, but as a practitioner, compliance doesn't mean you're secure, right? I was reading a thread yesterday on LinkedIn and I was talking about that. Why are we using cybersecurity frameworks if we're still having all these issues? The challenge is people are using compliance and it should be the floor, right? Yes, we have to be compliant. We need to make sure that, but that shouldn't be your high watermark, right? You have to meet that.

Michael Welch (19:23.010): We understand there's issues with resources and budgets and this and that, but you can't use that control as your ceiling. It should be your floor and you should always build best practices, right? Across all industries, understanding the challenge is where do we get the money from? Of course, in the NERC SIP world, there's regulated, unregulated. Sometimes you can do rate cases and you can get money back. So you see a lot of those companies in those regions doing more.

Michael Welch (19:52.526): But at the end of the day, we have to do better as an industry, as practitioners, even the government to make sure that we're getting funding, grants, whatever, to help organizations build their program, even if it's a starting point. Where do they go from? You mentioned 62443 from when we're looking at industrial control, not NERC SIP, right? That's a specific revision of 853. They're all very similar in nature.

Aaron Crow (19:52.966): All right.

Michael Welch (20:21.016): There's a lot of commonalities across them for organizations that have to meet NERC SIP, maybe PCI. There's lots of ways of economies of scale to try to leverage it across the enterprise as a whole. But unfortunately, we still work very silo based and it ends up costing the organizations more money. So I think we'll always have that debate. The compliance is needed because it becomes the vehicle to get the budget and the funding, but it has to be

Michael Welch (20:50.626): really pushed down at the enterprise level, either by the government, of course with NERC SIP, we got FERC, we got NERC, and then we got the regional entities. There's a big ecosystem there that has taken a lot of time to build to get to where we are, and it's still changing. mean, just coming out now, I think it was NERC SIP 15, where now we're finally looking at East West.

Michael Welch (21:12.898): We're not just looking, it's always been north-south, right? It's always been in and out of the firewall. But what about that insider or that person never really were we talking about east-west? Well, that's lateral movement. That's privilege escalation, right? So it's gonna be interesting how that's applied to the industry when we're looking at power for highs and mediums. And eventually it's gonna come down to the lows and it's gonna be a new whirlwind.

Michael Welch (21:38.922): Exciting to see but yeah, that's my thought there when we talk about compliance versus security They work together, but just because you're compliant doesn't mean you're secure. You have to do more least that's my thoughts

Aaron Crow (21:49.722): Yeah, yeah. Agreed. And it's one of those things where, you know, especially in NERC SIP, know, compliance is, is a hundred percent compliant, right? So the thing I would always say is 99.9999 % compliant is not compliant. You are in compliant because you are not 100, you are either compliant or you are not. There is no gray area in there, right? So, so you are getting fine because you are out of compliance or you're at least having a recordable or whatever, right? So.

Aaron Crow (22:17.338): where to your point, I can be compliant and still have a security issue, right? So when you look at a lot of these large utilities, maybe 40 or 50 or 70 % of their assets, their critical assets are part of the compliance program. And there's the other 30 % that's not. That doesn't mean those 30 % are not important. That just means they don't meet the criteria to be part of the compliance program, right? So we see Colonial Pipeline, for instance, right? Going back to that attack not so long ago,

Aaron Crow (22:47.163): that was not an OT system, but it impacted OT. Like I get so tired of hearing these arguments from pundits and social media people like, well, that wasn't even a cyber. That wasn't even an OT attack. Who cares? It impacted OT.

Michael Welch (23:0.526): Then you didn't do a business impact assessment to determine what you bring that IT system down is going to impact that system, right? So it goes back to resilience, right? Even tabletops, going back around that. You really have to understand the system as a whole, which is, know, like with cyber-informed engineering and really understanding every component, the dependency upstream as well as downstream because...

Aaron Crow (23:9.190): Correct. Correct.

Michael Welch (23:28.780): becomes so much critical because for a long time we were talking about the convergence of IT and OT. Really they should never converge, but IT is always gonna need data from OT for maintenance, for scheduling, for outages, right? You're always gonna need that. There is a secure way of doing that, but it becomes a challenge because once you open up those holes, now you have dependencies, right? Understanding those dependencies and if I bring this down,

Michael Welch (23:57.570): What's going to impact? So it's not something you're going to do overnight. It's planning, right? It's strategy. It's understanding the systems, working with your partners, right? Really to be able to come together and understand what resilience really means.

Clint (24:14.494): You know, that brings up a good point. You mentioned cyber informed engineering, which is kind of another buzzword flavor of the decade. But, you know, when it comes to the OT, IT bridge kind of thing, and the way that people are assessing that risk or those hazards, right? I think it needs to be flipped, right? When you're talking about OT and you know, this may be twisting the terminology a little bit.

Clint (24:42.418): But instead of having cyber informed engineering, I think you need to have engineering informed cyber, right? Because when it comes to your resiliency and because like you mentioned before, if it impacts OT, it impacts OP OT. It doesn't matter if it was targeting OT or not. And understanding how a cyber vector can affect the OT is important. And the only reason a lot of cyber folks don't understand

Clint (25:12.296): that whether it's IT, cyber, the only way you're truly going to understand that is to understand the processes, the operations, and you need to engage with the engineers, with the operators on that side. And when it comes to OT cyber, and this is kind of a soapbox of mine, but when it comes to OT cyber, the whole process should start on the operation side and work its way outward, not the other way around. I typically see IT and cyber, they have this

Clint (25:42.032): mentality of, guy, hey, OT folks, we need to protect you from yourselves. And that's not the fact of the matter is that operators and engineers have been protecting against impacts to the process to the environment to production for a very long time. And they're good at it. Cyber is just a new hazard in the the whole process hazards analysis, you know.

Aaron Crow (25:47.162): right.

Aaron Crow (26:5.606): Correct. 100%. And no, I was just going to say, I agree 100%. And that's really what the cyber informed engineering out of Idaho national labs is really trying to say is you have to include, because it can't be me as a cyber security professional designing a cyber system to push down to the plant. You know, and I I've been doing this way longer than it was called cyber informed engineering or anything like that.

Michael Welch (26:6.444): I mean for the longest, yeah, I'm sorry, go on.

Aaron Crow (26:31.340): I was working with the asset owners. was working with the systems administrators and the systems owners and the control engineers and those people and saying, hey, this is your system. You need remote access. We need to make sure that that's secure. So instead of just putting this on the internet, let's do this in a secure manner, right? But it's not me. My security solutions are not the thing that drive the thing, right? And I saw this from a...

Aaron Crow (27:0.312): Again, being a former asset owner and a power utility here in Texas, I saw a lot of things that came down from IT. IT had the budget. They had, you know, all these, these policies or procedures and they tried to cram them down our throat. Cause I worked at the plan in the business, right? But what, and I had this conversation a lot, even all the way up to, you know, the CISO of our company. And I said, look guys, you have to realize you're the tail. You don't wag the dog.

Aaron Crow (27:26.246): Like you have more money than me, but if my stuff goes down, our company doesn't exist. If your stuff goes down, it's a bad day, right? You've got executives upset or whatever. But if my stuff goes down, the government is calling us because we've taken, you know, 30,000 megawatts of generation off the grid and we have a big, big, big problem, right? You need to set those things separately. And to your point, Clint, is cyber is just a risk.

Aaron Crow (27:53.776): that they have to design into the system. And that's the pieces is up until now, those engineers don't have the skillset, nor should they have the skillset to truly understand what that risk is. So we need to be working together. all in the same team. It should not be IT against OT. It should not be IT against the plants and the business and the manufacturing and all this kind of stuff. We should be sitting at the table together because on the same, at the same time, OT needs IT as well.

Aaron Crow (28:21.360): to provide secure mode access, to provide firewall support. Like you don't, shouldn't expect your control engineers to be managing a firewall and be managing virtual environments and to be managing the networking environments and the routing and all that kind of stuff. Like let them focus on what they do, but the only way that's gonna happen is if you build trust between those organizations, because I'm not going to give anything to IT if I think they're going to break my system. So a lot of that distrust is because IT tries to cram things down OTs

Aaron Crow (28:51.226): throat and they say, if you're going to do that, get out of my house. I'll do it in house. I'd rather do it poorly than allow you in my house because you break stuff. And I'm the one that gets called at three o'clock in the morning when the thing goes down while you're having mimosas, you know, six hours away and don't even know what happened. Right.

Clint (29:9.694): Yeah, that's key communication, right? The whenever you have a culture of working collaboratively with OT and it and they communicate and actually take the time to say, hey, teach me what you do. Teach me about your process, whether it's it or OT. I want to understand that is how you you know, that is how you build that trust. Right. I mean, 18 years of marriage and I'm not divorced yet.

Clint (29:39.153): has taught me communication is key and OTIT, it is a relationship, right? And in any relationship, communication, understanding, compromise, meeting in the middle is key.

Michael Welch (29:52.632): Yeah, no, I would say even when we were talking, you mentioned Idaho National Labs, Aaron, even they're consequence driven, right? It goes back to, it's similar to business impact assessments. It's understanding the consequence. And then the only way you could truly understand those consequences is getting everybody together, right? And walking through it. And that even includes the manufacturer. I remember when I was at a large utility, worked there for a number of years, we used to do factory acceptance testing, right? We would have,

Michael Welch (30:20.774): IT, OT, us, the distributor, everybody at the FAT, right? Really walking through the process. And then we would do it again at the site. So site acceptance testing, right? Now, not enough, now it takes time, it takes dollars to do that, but not enough entities out there to do it. Because at the end of the day, we're always looking at security by design. Or I think the new thing is security by default, unfortunately.

Michael Welch (30:47.564): I don't know if we'll ever get there. Security still seems like it's a bolt on. At some point in the future, it will be brought in early in the conversation. But the earlier you bring it, the better chances you have to build a program that you can really work with, be more proactive rather than always reacting. Because if we're always reacting, we're losing.

Aaron Crow (31:9.488): Well, I I 100 % agree and we've made progress because I remember, I don't know, back in 2010 or something, was again, an asset owner, power utility, and we were doing control system upgrades across our fleet. And I think we did, I don't know, 15 that year. So I spent a lot of time in, I won't say the cities, but other places doing factor acceptance tests with the control vendors, right?

Aaron Crow (31:33.964): And as we were, were specking out those systems. And when I say we, I was part of the team. It wasn't me doing it, but I was invited in not by choice. They didn't have a choice. was like NERC SIP was coming out. We had to have it secure. There was the vendor actually put in front of the plant manager, the secure and the insecure option for their control system. Right. And, and as soon as I saw that, I'm like, are you freaking kidding me? Like you actually have that. That's the title of your proposal.

Michael Welch (31:54.382): Yeah.

Aaron Crow (32:1.594): The secure option is this and the insecure option is this one. Which one do you want? And of course the plant manager wanted the insecure. Why? Because it was a hundred thousand dollars cheaper or whatever that number was. And I said, okay, this is not on the table. You cannot do this one. So throw it out and don't ever show anybody at this company an insecure option again, by definite. Like why would you even have that name? Of course now nobody would do that. But back then it was, it still wasn't.

Michael Welch (32:27.426): It was acceptable.

Aaron Crow (32:28.518): it was acceptable like most people it was you know no don't know Active Directory no network no firewall no any of that it was just you know raw dog in it.

Michael Welch (32:36.760): It was about the service level agreement. How can we make sure we meet that SLA? It wasn't about making sure we can control or secure the environment,

Aaron Crow (32:39.984): Yeah. Yeah.

Aaron Crow (32:46.169): Absolutely.

Clint (32:46.952): You know, and that really kind of brings up an interesting pivot in this conversation. The convenience versus security factor, especially since we're talking about this in the spirit of cybersecurity awareness. There was an interesting thing that happened to me this past weekend sitting on the couch. you know, I'm going to go ahead and I'm going to come out here. I switched from Android.

Clint (33:12.466): to iPhone and I'm coming out of the closet here and the and so I'm getting everything set up and you know when you switch over you know I use a password manager and everything but you gotta start switching everything over and you know getting all my social media accounts and my Netflix and everything set up and I forget which one it was and you know my wife sitting there next to me and I'm I start screaming at my phone because

Clint (33:41.928): I'm like, are you kidding me? I have to go through. We're no longer in the era of dual factor and multi-factor. I literally had to go through five different stage gates of security. it wasn't even anything important. It was Steam. was, I'm screaming at my computer saying, are you kidding me? It's a dang video game platform. And I have to go through more stage gates of security.

Clint (34:9.000): than I do for my Microsoft account, what gives? And then I realized right then, and I'm a cybersecurity professional and I'm like, this is why I hate cybersecurity. if I'm a cybersecurity professional and that's my reaction, that is, you're getting 10 times that reaction from the average user, right? And that is why cybersecurity fails in many cases is because we're over-engineering the cybersecurity in many places.

Clint (34:38.684): And anytime something seriously severely overrides convenience, like there's a compromise, but when it's so hard to use, you're either gonna not use that service or you're gonna find ways to circumvent the service, right? And so I think my mantra is when it comes to cybersecurity awareness, everybody's already aware, right? So the new meaning of cybersecurity awareness is not how do you make people aware of cyber threats?

Clint (35:8.368): It is all about how do you find a balance between convenience and security? How do you get people willing fully willing willingly, sorry, willing fully? I can't talk today. How do you get people willingly engaged in the cybersecurity equation? And, you know, I'll just leave that as a conversation.

Michael Welch (35:28.480): Well, engagement is always the challenge and I know it's always been that the end user is the weakest link and because we're talking about cybersecurity, realistically, the end user should be part of your solution, the first line of defense, the last line of defense, but you do have to make it easy because it always is about the user experience. It doesn't matter if you're doing multi-factor authentication or you're doing this, the user experience has to be acceptable.

Michael Welch (35:54.104): but we can't risk security at the same time, right? So it's about how do we engage them? If you force it down their throat, they're not gonna be happy, they're gonna find ways to circumvent it, and then none of us win, right? So I always look at the end user, the business, whatever it is, we have to engage them because they are part of our solution, and without them, we're not gonna win.

Aaron Crow (36:15.750): Absolutely. And you said something right there that that's super important. And I've seen it in OT all the time. Availability, right? And if you make it so complex, why do we not have 100 character passwords? Because nobody can remember them, right? And I can't tell you how many power plants and manufacturing facilities I walk into and the password is printed or written out on a sticky note sitting right by the operator workstation, right? People will find workarounds.

Aaron Crow (36:42.734): If you make it so complex, they will find a way around it. They'll bypass your firewall, they'll bypass your security, they'll do whatever, because when it's Saturday morning at three o'clock in the morning and they have to get the system and they've got the plant manager screaming behind their head and the CEO is screaming at them, they're about to lose their job, they don't care about your cybersecurity stuff, they just need to get the system working, right? Availability trumps everything. When there's no electricity and the power plant just tripped and you've got to get it back online,

Aaron Crow (37:9.956): The last thing in your mind is thinking about the cyber security. Is this site safe? Like safe safety from a, from a human human, you know, life perspective. Yes, is always there, but we've got to start thinking about, Hey, we can't make it so complex that they're going to just bypass it because they will to your point, right? They will find a way around it or they'll kick you out and say, I'm not doing that. I'm going to accept this risk and you're take that someplace else. Cause I don't have time for.

Michael Welch (37:29.292): Yeah.

Michael Welch (37:37.048): Yeah, I remember I was at a conference a few years ago and the guy that was talking was talking about the blast radius. If something happens, what is that blast radius? And if it becomes so complicated to try to restore, you're doing something wrong, right? I think you mentioned earlier, Clint, I mean, keep it simple. There's ways to implement cybersecurity that are not overly complex or you used over-engineering, right? It's how to find that balance because there is good ways and it's...

Michael Welch (38:4.790): Right sized for the function, for the business, it's not one size fits all. And I think that's also the challenge with compliance sometimes, right? It's not always one size fits all. This business risk is different than this one's appetite because of what they do and it's how you implement it. it goes back to that you can't implement every control at once and think you're gonna be successful, right? You implement your controls, you mature those controls, then you add on.

Michael Welch (38:34.762): Additional controls as needed based on your attack surface and the different things that the industry you're in the benchmarks, right? It's a it's a Continue we always use continuous monitoring continuous this it's a continuous process, right? We can never put our guard down Because the bad actors don't and they for them. It's all about money and they want your data So we always are working and I mean, it's it's a challenge for anybody out there

Aaron Crow (38:55.920): Yep.

Michael Welch (39:1.912): But if you overcomplicate it, it's also gonna be a larger challenge to try to be successful. Or if you have to restore because of something happened, it could be attack, it could be just a misconfiguration or an accident. But if it's so complicated to restore, you've lost.

Clint (39:20.978): Yeah. And, know, and I think going back to the conversation, so, so how do you manage that over engineering the compromise of accessibility and availability versus security? And my experience this past weekend to go back to that example was that, you know, no, no, no, no, Well, I mean, to be fair, I did find out that my password manager

Aaron Crow (39:21.253): Yeah.

Michael Welch (39:40.206): Go back to the Android?

Clint (39:48.402): because I use a, a platform agnostic password manager and I'm not going to tell everybody what that is because that's, that's a security issue. But, but, but I, I use that and number one, using a password manager. I don't know any of my passwords and yes, I do have 30 plus character passwords. because I, the convenience factor. So if you engineer convenience into security, flipping the script.

Aaron Crow (40:8.570): Mm-hmm.

Clint (40:17.618): then you have security, right? So using a password manager that can not auto-fill in the insecure traditional sense, but with the click of a button can auto-fill passwords into your browser and things like that. And that's what I was using. I would not have been able to get everything switched over effectively because onto my phone had I not had a password manager and be able to manage that. So making the password managers

Clint (40:44.420): easy to use, right? Making your third party authenticator. So I'm using different, you know, whether you use Microsoft, indicator or Google or whatever, making it easy to do the security password managers, authenticators with one click button authentication. Cause you know, Google has a fantastic one, right? Is this me? Yes. You know, and I, there are ways to circumvent that. But when you make security easy, then you, you flip that script. Like I said before, and that's the problem.

Clint (41:14.312): You know, again, going back to examples of the week and everything and my wife, my wife yells at me because he's trying to get into Netflix and I have this super long password and she grabs at me because why do you have this? How do you remember that? I I don't remember it. I have it on a password manager. And then going back to the user mentality, she refuses to use the password manager because it's new technology for her to learn. So it's this constant cat and mouse chase of we just made it more convenient for you user. And the user says,

Clint (41:43.870): But it's new technology I have to learn and that's not convenient. It's a...

Aaron Crow (41:47.330): Right. Yeah. And so I love that. And a hundred percent like I've got my family on a password manager. So we have one and I've got the family plan. So my kids have their own logins and my wife has one so they can create their own logins and they all know how to do it. And it's funny because, you know, they're doing it at school and the other kids are looking at them like they're weird. Like, why do you have, what is that? Like they don't understand it, but you know, they're like, my dad's in cybersecurity. So.

Aaron Crow (42:15.750): And they're like, okay. know, but, driving to that, those conversations, I think that's the let's, I'm going to lead it back to those tabletops and, why I want to lead it to those tabletops is cause I think that's kind of a great place that can be used in most of the time are not being used properly. I don't believe because most places that do a tabletop, they do it once a year and the cyber executives and you know, the CISO and the C-suite folks.

Aaron Crow (42:44.506): go through that exercise. They have some of the facilities people, whatever, but they don't have all of the right people involved. And part of that is because it's hard to do. They're paying third party companies to come in and they can't get everybody in the same room. So Clint, know that you've with ThreatGen have a product and this isn't to pitch that, but more so around why, what we can do using AI and things like that.

Aaron Crow (43:14.448): to make those conversations better. And the outcome of those things can be that IT and OT and plant managers and executives all better understand and their perspectives are put into the process so that when you come out, you come out with a more accurate understanding of my risks, the mitigations that I need to do, the concerns that I have, the places that I've got gaps.

Clint (43:35.955): Okay.

Aaron Crow (43:43.002): the places that I'm strong in, know, all of that type of stuff can come in because I have everybody at the table. So all that lead up to say, let's dig into tabletops.

Clint (43:53.938): Yeah. So, you know, the tabletops are the are one of those things to where it's an it's the age old adage where practice makes perfect or perfect practice makes perfect, right. And but it's a measurement tool. And it is muscle memory. The problem is, number one, the set or the symptom is that we're doing annual tabletops. You can't

Clint (44:20.646): really get much out of an annual tabletop other than checking a box going back to compliance, right? But if you want to get the true value of the tabletops, you need to do them more frequently. You know, it's you're, you're not going to get as fit as Aaron there with by wanting by doing one ruck. You know, you got to do it regularly, right? You got to exercise regularly, or it doesn't get into your muscle memory. And you know, what's the old saying, you know, everybody has a plan until you get punched in the mouth. Well, but if you but if that plan is in

Clint (44:49.892): ingrained into your reaction, your muscle memory, then you're going to be fine when it hits the fan. And you're going to know what to do. So but why? Why do we only do it once a year? And in aside from apathy and just complacency, it's because tabletops have traditionally been hard, right? I mean, it takes a lot of time to plan one effectively and how to do it. And then you know, we're creatures of comfort. Yeah, exactly. We're creatures of comfort.

Aaron Crow (45:15.673): and boring.

Clint (45:19.492): And we don't do things that are difficult very easily. Right. And so this is that is where the generative AI does come in. In that, because it's a superhuman, right, it analyzes things like human and it can do it. It augments the human capabilities to write documents, to create things, to come up with stories. Right. Using generative AI, you can take all that heavy lifting off of

Clint (45:48.264): the resources that it takes to plan the tabletop exercise, you can use generative AI to create your scenarios, realistic, effective scenarios. Any tabletop in the past that I've been involved in the traditional way, you never have enough artifacts and collateral, right? You're always going to have somebody that says, well, what do those logs look like? Or that's not how our system works. Or, you know, you have those problems. So using generative AI, you can either preemptively create a ton of

Clint (46:17.626): artifacts and collateral to be able to show your team, your audience, or you can create it on the fly, right? And that's kind of one of the things that I've created, which is this dynamic system that will create the scenario and facilitate the scenario and create injects and run it. And when somebody says, hey, how do you, know, what are the logs look like? It will create very detailed logs. It'll create this cohesive, congruent story.

Clint (46:47.654): And that's what generative AI does, right? And so all of a sudden now, when we don't have to spend tons of time and resources to create the scenarios and run the tabletop, get everybody in the same room, we can do them more effectively, right? We can run them twice a week, we can run them quarterly. And now you have a measurement tool that says, well, we've improved, we've improved, we've improved. Okay, we put some changes in place, let's run it again. Or how about this? You got a new threat that comes out and you're unsure whether or not you're prepared for that threat.

Clint (47:17.182): Can we do we have the time to create a traditional tabletop to put that thread in there, right? The scenario do all this with generative AI. You can simply take the details of that threat, feed it to generative AI and say, let's go. And now all of a sudden you are running a tabletop instantly on that thread. So, you know, to me, going back, you know, this should probably be my closing thought, but I'm going say it now. You know, generative AI.

Clint (47:47.504): is the Jerry Maguire of technology. You you complete me. That's what it is.

Aaron Crow (47:52.324): Right. Yeah. You know,

Michael Welch (47:53.934): But what I love about it is, let's say, I think with the traditional tabletops is the logistical piece of getting everybody involved becomes a challenge. But you can leverage the automation of it and then build, like you said, Clint, build new injects, bring in other groups that a lot of times forgotten, facilities, building management, right? And then you can take that previous one and then put injects that are specific to them.

Michael Welch (48:19.660): that now they can get value all almost on the fly, right? So it really is quick to turn around and then you can get now those different business areas are, or once again, we use that word engaged because now it's specific to their environment, their function. And then it helps them build a plan on recovery, resilience, business continuity, right? Whatever it may be.

Aaron Crow (48:47.172): Yeah, absolutely. And if you take it a step further, like the use cases is open ended in that you can expand it so I can run it with this group and then I can take the results and feed it back to the system and then send it out to my systems engineers and let them run through it. And then their input is adjusting the model. And then I send it to my IT people and then I send it to my OT people. And then I come back together and say, Hey, after we fed all these things through, we made some assumptions in the first round that

Aaron Crow (49:15.098): you know, our systems engineers or our plant managers or our this group, hey, they brought up some things that we didn't think about. Let's go back through it again. And it's like, you know, to Clint's point, it's like a muscle, right? So if I do this and I learn, well, I should have initiated my incident response plan two steps before when I go through that same scenario again, like, I remember last time I waited too long. Let's go ahead and initiate, right? It's the same reason why we do fire drills.

Aaron Crow (49:42.884): like in buildings, right? Is we don't want the first time that somebody's having to evacuate for a fire to be when there's an actual fire, right? We want them to understand where's the muster point. I don't use the elevators. There's a floor warden. You know, what do I do? Where do I evacuate to? I go two floors down. Like if I'm in, you know, a building in you know, in New York, I'm going two floors down or whatever that procedure is.

Aaron Crow (50:7.300): The tabletops allow us to do similar things and do them repetitively without this super overhead and expense of having to bring everybody in the same room and have a third party to hold it. Like it just really opens up the capability and allows it to be used like a tool.

Clint (50:23.400): You know, it's so this is this panel here is the perfect illustration of tabletops. You have Aaron and Mike are the epitome of the regular practice, regular exercising on the annual tabletop. Who do you want to be?

Aaron Crow (50:45.711): you

Michael Welch (50:46.476): Awesome.

Aaron Crow (50:49.200): All right, so always kind of wrap these things up with a question I ask everyone. And it's the next five to 10 years, what's one thing that you see that's concerning and maybe one thing that you're excited about coming up over the rise in cyber? So Mike, I'll go with you first. What's one thing that you're kind of excited for, maybe one thing you're concerned about that we need to make sure that we stop at the pass?

Michael Welch (51:11.318): I actually, think it's the conversation we had today, AI, excited about its capabilities, but concerned about its capabilities. I mean, and that's like Clint was saying, we're at its infancy, right? As you mentioned, when power and electricity becomes available, that brain is just gonna get larger and larger and larger.

Aaron Crow (51:18.266): Sure. Yeah.

Clint (51:18.526): Yeah.

Michael Welch (51:31.298): That will bring a lot of value, but at the same time, will bring a lot of risk that as practitioners, we have to always be thinking about that. So it's, I'm glad I went first, cause that was the easy answer.

Aaron Crow (51:42.982): It's double edged sword, I agree. Go ahead, Clint.

Clint (51:43.404): Yeah, yeah. Yeah. And so let's just go with the assumption. I'm not allowed to say the same thing, right? Because you knew that's what I was gonna say. All right. So you know, I think that the thing that I'm that I that I'm not concerned about, but the thing that I see potential in and I think is that in the slowly but surely

Michael Welch (51:52.758): Exactly.

Clint (52:13.020): we are making progress in cyber, right? mean, especially in, from the OT perspective, right? I mean, 10, 20 years ago there was resistance and now there is, dare I say awareness and concern and willing participation, right? So I am seeing progress in the willingness to address cyber security. Okay. And I think on the flip side, what I'm concerned about

Clint (52:42.160): is budgets, because without budget, you're not going to hire the right personnel. You're not going to retain personnel and staff. You're not going to get what you need. You're going to continue to search the bottom of the barrel to find whatever tools you can to do the best you can with what you have. Right. And so I think budgets have always been a concern. And I don't think that while I'm seeing some progress from the sea level,

Clint (53:11.398): and everything like that. I think I'm seeing far too often that you don't have enough buy in from sea level to allocate the necessary budget people aren't seeing security as a business process. They're seeing it as insurance and I don't like that. So that's my concern.

Aaron Crow (53:32.302): Yeah, kind of adding on to that, I see a lack of understanding and acceptance of risk. And part of that is because the executives are accepting risk because somebody told them, yeah, we're OK with this or whatever. And they're not truly understanding the risk because I think if they really understood it, they wouldn't accept that risk. Right. So I think a lot of risks that are accepted, but whether they're pushing it off to cyber insurance or whether they're saying, well, that's not a problem or whatever, I think a lot of that is because they are misinformed.

Aaron Crow (54:2.094): are not accurately informed. And I don't think it's malicious. think it's, yeah, they think it's like that false sense of security in that they think they're better off than they are. They're really strong on their IT side. They've got really good firewalls. They've got processes and procedures. But you look at the target attack, you look at a lot of these things, all it takes is a chain is only as strong as its weakest link. If I've got one device that's connected to the internet and bypasses all of this great security stuff,

Aaron Crow (54:32.102): It didn't matter, right? I've got all of this capability, but I just bypassed it because you made the security solution too hard. So some engineer at the plant just plugged it directly in the internet, right? You know, I've done assessments where the cable modem that's providing tertiary or backup communications is plugged directly into a network and it's got the Wi-Fi built into it. And I could bypass all of your firewalls and all the enterprise stuff that you've got by connecting to the cable modem router and I'm directly in, right? And it's not...

Aaron Crow (55:2.168): Uncommon, right? We have systems that are old. We have systems that are Windows XP. We have a lot of this stuff and that doesn't mean you've got to rip them out, but you truly need to understand those risks and make sure you're mitigating those risks, not just accepting the risks blindly because honestly, most people that are accepting the risk probably don't even have the authority to accept the risk and it's not accurately getting bubbled up enough to translate that risk and communicate what the true risks is.

Aaron Crow (55:29.156): so that the people that actually have the authority, the CISO and the C-suite executives are able to make the right informed decision and accept and or mitigate those risks accordingly.

Michael Welch (55:38.636): I think that's a topic we have to have another conversation on Aaron because it's, I mean, you can go in so many areas. It might be a future one because even when you look at reputational risk, we always used to be, hey, if you're breached and you're a public company, your stock's going to go down. But how many stocks gone down, people buy the dip and then next thing you know, six months later, the stock is through the roof higher than ever, right?

Michael Welch (56:2.904): So it's understanding the risks and the impact and how you have that conversation with the C-level, the C-suite.

Aaron Crow (56:11.322): Yeah, absolutely. awesome gents. really appreciate the time today. Some great conversations, know, Clint and I didn't fight over AI or anything like that.

Clint (56:20.486): I don't do my own fighting anymore. let AI fight my battles now.

Michael Welch (56:21.631): I was the mediator.

Aaron Crow (56:27.255): That podcast I listened to really, they basically trained their voice and they had it connected to a phone line. So they would actually have it call customer service for them. So I can absolutely see that being something I want in my future. If you need to disconnect your cable or you need to whatever, just have your AI be your assistant and take care of that stuff. Right now, luckily I have a wife and she does that stuff for me, but it'd be really cool if I had an AI that I could just say, hey,

Michael Welch (56:27.885): Awesome.

Aaron Crow (56:53.402): Go do this stuff. Negotiate my contract for my cell phone and don't give up until they give me a discount.

Clint (57:0.542): Talk to my boss about my performance review for me.

Aaron Crow (57:3.404): Exactly.

Aaron Crow (57:7.204): go on job applications. There's all sorts of endless opportunities that people will use AI for that really have nothing to do with cybersecurity. They're just fun to talk about.

Michael Welch (57:7.744): Oz. Yup.

Michael Welch (57:17.102): 100%.

Aaron Crow (57:18.726): All right, gentlemen, I appreciate your time.

Michael Welch (57:20.682): Awesome, be good.

Clint (57:21.076): All right, thanks guys.

Transcript lightly edited for readability.