Bridging IT and OT in Cybersecurity for Power Plants with Jori VanAntwerp

Aaron Crow (0:1.332): Awesome. Thank you. Thank you for joining me today on the podcast. Jory, why don't you just introduce yourself? Tell us who you are and a little bit about your, about your background as, as well as maybe even a little bit about that painting that's behind you.

Jori VanAntwerp (0:13.512): Sure. Aaron, thanks for having me. So my name is Jory Van Antwerp. I am the founder of a small company called Enver OT. Long time geek. I'm going to really show off this gray in my beard, actually. I started off actually building PCs at home and then graduated to writing key gens and crackers and all of that fun stuff that we shouldn't have been doing, but I was doing anyways.

Jori VanAntwerp (0:40.544): And that led to a fantastic career that evolved into IT. And as I tell a lot of people, whether you're in systems network or you've done the whole gamut, all roads eventually lead to security. Because once you build a solution, you want to protect it. You want it to run right. People want to use it. It's just a natural evolution. So I started in cybersecurity actually, man, almost 20 years ago at this point. And that evolved into a really neat career.

Jori VanAntwerp (1:9.590): of running around and eventually touched on something that you and I are heavily involved in, which is industrial. And I started being exposed to industrial through manufacturing plants and kind of the, the IT edge that touches industrial. And then eventually I was picked up by an industrial company. I'm sure nobody's heard of it's called Drago's. And I was sent out to a site to actually help with an onsite assessment. And that was at a large dam here in Arizona.

Jori VanAntwerp (1:38.878): And it completely blew my mind. It finished that paradigm shift between IT and OT and showed me just kind of what you're dealing with, why it's so important and why it's so different. And I have been absolutely immersed in it ever since.

Aaron Crow (1:54.770): Yeah, it's amazing. It's like you see behind the Oz, you you see the wizard behind behind the curtain and it's it's eye opening and it's exciting. You know, the amount of things and the need and, know, the protection, you know, as a self-proclaimed patriot, right? I want to protect our country. want to protect my kids, our way of life beyond just this country, like in general, like stopping bad actors from doing things and making sure, you know,

Aaron Crow (2:22.708): People take for granted when they turn their light switch on that how the electricity gets to them or when there's a natural disaster, you know, they don't really understand why it's so complex. I know in Nashville or Nashville, Asheville in North Carolina here recently, like, you know, we've got all of these basic systems that are not working. I know there was a retail store in the area that couldn't sell anything for like two weeks.

Aaron Crow (2:51.432): because they didn't have any way to process payment. didn't have, you couldn't go, ATMs didn't work. They couldn't go to the bank because the banks were all closed. They couldn't get cash. They couldn't accept PayPal. Like nothing worked. So they have the stuff and there's no way to transfer value other than the old school bartering or just giving it away because they didn't have basic processes. And all of those things, as we know, once you see behind the curtain,

Aaron Crow (3:17.396): There's all of these connections, power and manufacturing and transportation and all these critical infrastructures, they're all linked. There's a reason why they're critical because something bad happens in all those things. If one of those dominoes falls, our entire ecosystem falls around it and it's very hard to get it stood back up and it takes time and it's difficult. So that's why it's so important to think about these things as we say, left of boom.

Aaron Crow (3:43.925): and start having these conversations before something bad happens instead of waiting until after it happened and trying to say, crap, now what do I do?

Jori VanAntwerp (3:52.960): Absolutely. And that actually that keys into a really interesting conversation that I've had quite a few times recently, which is around modernization in these environments. You know, you and I both know that it's incredibly important that we start to utilize some of the technology that we've created in the last, you know, 10, 15 years. And even the stuff that's coming out now, the issue that comes into it though, as you just stated is, some of that automation comes at a price.

Jori VanAntwerp (4:21.502): And I talk a lot about modernization in these environments and where should we stop? Well, one of the things that I think is incredibly important is keeping those manual controls around, actually having an operator being able to turn knobs, flip breakers, whatever they may need to do to keep that operation running should they need to without the automation. Because if we don't have that, you run into situations that we've actually seen pop up in Ukraine and other places in Europe where they have modernization.

Jori VanAntwerp (4:52.679): And once those modern systems had an issue, whether that's a security breach or not, there's nothing you can do except go and repair that system. And we're lucky here in the US in many, instances, we have those manual controls. So the power does stay on as long as possible or that we can get it up in an interim status, right, as we're replacing the system.

Aaron Crow (5:15.122): Yeah, you know, came, lot of my career has been spinning and specifically power generation trends, you know, transmission, distribution and generation. and we have, you know, entire playbooks on, if the entire grid goes down, how do we get it back up from, from, no, but people don't understand in a lot of these spaces is it takes power to start power. So if, if, if the grid goes down and, the, everything is dark, how do I get

Aaron Crow (5:43.806): power back going when I have to have power. It's kind of the chicken and egg thing. I have to have power to get power going. So we have black starts. There's generators like, but if any of those things fail, so I have to start a small plant to get enough power on the grid to get the next plant going to increase the phase and all the electrical things that go into into this. And there's also a reason why, and I know I'm nerding out on the power side, but there's a reason why Texas is its own grid and the East Coast is different than the West Coast.

Aaron Crow (6:12.742): And, you know, people want to throw rocks and stones because Texas has had some issues in in in Urquhart and our grid here lately, but it was designed on purpose. This wasn't a Texas thing. This was a, how do we make sure that the country maintains and we don't have a something in the East coast in New York and Manhattan that can roll a blackout across the whole country, which can take forever to repair and replace. Whereas with Texas in the middle and DC ties to everything.

Aaron Crow (6:40.282): It is the protection from from Texas to be able to get the East Coast or West Coast go back up and going. Same thing in Mexico, etc. So it just goes to show and the bigger picture of this is when you talk about modernization is is, you you look at a nuclear power plant, a lot of those things were built in the 50s or designed in the 50s, at least maybe they came online in the 80s. And you look at those and we've digitized, we've modernized a lot of those things, but we didn't replace the analog.

Aaron Crow (7:8.114): we put the digital in series with the analog, right? So the analog is still there. It is still our tertiary, know, break glass in case of emergency and our people are trained on it. So if all the digital stuff fails, we can still do it manually, which is important when you're talking about nuclear power.

Jori VanAntwerp (7:8.430): correct.

Jori VanAntwerp (7:27.182): That's incredibly important.

Aaron Crow (7:32.958): You there? Yeah.

Jori VanAntwerp (7:37.578): I am. I lost you for a second, so...

Aaron Crow (7:42.664): what's going on.

Jori VanAntwerp (7:46.402): That's all good. No, it's incredibly important, especially when it comes to nuclear power. you know, what you're talking about, the interconnects, you know, there's interconnects that are spread out throughout the United States and they even lead up into Canada. And that is part of the reason it's also for sharing power in particular instances. You know, I live in Arizona and our neighbors to the West have a lot of power problems. So we share power a lot with our neighbors to the West.

Jori VanAntwerp (8:14.818): on a regular basis through one of those interconnects. But those interconnects are there so that the thing that everyone fears can't happen, which is, we hear in the news and a lot of fighting on it, but gonna turn off the grid. There's a lot of grids. Turning off the grid is not a, we're not gonna flip a switch. know, a threat actor is not gonna be able to hack a singleized point and just take out all of the US. And a lot of that is due to those separate grids and those interconnects.

Aaron Crow (8:30.077): Right.

Jori VanAntwerp (8:42.636): So there's actually a lot of redundancy and forethought that goes into how power is put together. And I think it's a, you know, can absolutely geek out on it with you because it's a great way to look at how to build something that's robust because in the U S we have one of the most stable power grids in the world. And it's something that we do very, very well. And, you know, all our technology shows that, but it's, it is redundant and that redundancy helps even with security.

Aaron Crow (9:9.022): Yeah. And that just goes to show that even with power, think power, if you look at the critical infrastructures, my feeling is power is one of the more mature because of NERC SIP and some of those other regulations on a cyber perspective. And yet we're still behind the eight ball. Like there's still a lot of work that needs to be done and modernization and a lot of processes and proceedings. And it's not all technology. This isn't a technology problem, not only a technology problem, right?

Aaron Crow (9:38.216): We need process and people and technology altogether to fix some of these things and design these things, right? When these systems were designed, cyber wasn't an issue. And these systems also weren't connected to corporate or the internet or any of those things. So they were designed in a vacuum. And then we started bolting all these things on because we wanted access and capabilities and functionality. And as we did these things, we brought in these risks. In the beginning, we didn't necessarily understand the risks that we were bringing into these environments.

Jori VanAntwerp (9:49.026): Mm-mm.

Aaron Crow (10:6.332): And then once we have them, they're like, why now we're dependent upon those capabilities. But also I have to be careful because now I, I brought in this risk that I want to mitigate and I can't make it so difficult that it brings the process down or the plant down, but I also need to make sure that, you know, a bad actor can't get in or even, you know, most of the time and, and what we see is even just a misconfiguration. Like if a misconfiguration can bring down my environment, it doesn't matter that it's.

Aaron Crow (10:35.920): not a nation state or it's not a cyber attack or you know there's some popular people I won't even say their names but they're like there's there's all this ot stuff nobody's talking about vulnerabilities there's not that much stuff here and you know fierce fierce selling and like okay let's say that's true but you can still bring down environments whether or not it's ot colonial pipeline's a great example that wasn't an ot attack but it brought down ot so does it matter

Jori VanAntwerp (10:57.260): Not at all.

Jori VanAntwerp (11:0.556): Yeah, it's, it's a discussion that we really need. We really need to have more often publicly because the, you know, the greater public, they don't know the intricacies of, of, of the systems that we deal with. And you just brought up a fantastic point, which is, know, we went from having 36, know, 36 guys, 12 trucks driving out to locations, checking on things to actually bringing, bringing information back to a centralized area, which one.

Jori VanAntwerp (11:28.834): It allowed us to actually be more efficient, but we could also get metrics and telemetry status, all these things from these environments that can help us do things better, but they weren't intended to be connected. And there's a fine line to walk with that as well, because as we're modernizing these environments, we introduce more vulnerabilities, different vulnerabilities. We introduce inefficiencies as well. One of the areas that interests me the most right now is the fact that

Jori VanAntwerp (11:57.762): doesn't matter if you're in energy, know, advanced manufacturing or pharmaceutical, many people don't monitor at the operation, which, you know, we call Purdue level one, but they don't, they don't monitor at that particular operation. It's difficult. The only point I mean by that is that we don't have enough data about what's actually transpiring there to see what threats, if any, or reconnaissance are happening there in that same token. One of the things that's been,

Jori VanAntwerp (12:27.138): that's been proposed to help with that level one problem, that operational problem, is to update the hardware with things that have more traditional operating systems, whether that be Linux, et cetera, so that you can actually house what in IT we call defense in depth, right? You'd have EDR, you'd have all sorts of defense. There's an issue with that, and it's actually twofold, and it's very difficult for folks outside not to understand the systems that we talk about, those programmable logic controllers that...

Jori VanAntwerp (12:56.418): that we have at energy there. I mean, I'll point this way. Half a mile that way is literally the substation that's feeding my house right now. Runs on Foxboro controllers. It is, I want to say it's almost 19 years old at this point. Might be a little bit more than that. And it will be running. It's not up for a Harbor refresh. The devices in there sometimes reach temperatures. The air temperature in there is 145 degrees Fahrenheit. I mean, it's,

Jori VanAntwerp (13:25.474): blistering in the summer. There's no air conditioning. There's no nothing. And one of the reasons that they run so well is because they're very specifically made for that task. So in modern computing terms, we have very down tuned processors. They don't create much heat, means they don't eat much cooling. Everything is, you know, sparkless, fanless. It's all built to run that way. The second that you introduce a legitimate operating system, suddenly I need more power. We need more cooling.

Jori VanAntwerp (13:53.902): and, and, and we've introduced complexity and complexity is actually the absolute enemy of robustness. Simplicity is where you get robustness from. So just from keeping your lights on, keeping that simple and robust is incredibly important. That may be at not the sacrifice, but the knowing, decision to say, okay, well, we can't implement these types of security measures at X, but we can have a 99 point trailing nines uptime.

Aaron Crow (14:1.886): Mm-hmm.

Jori VanAntwerp (14:24.242): At any given point in time. The second is cost you and I live in very hot environments, which means we use AC a lot It's very expensive in the summer Could you imagine if they decided they wanted to upgrade that particular? Substation to the latest and greatest outside cycle how much your power bill would be because it's gonna go to the subscribers. We're the ratepayers so you want to upgrade that substation suddenly, you know, my power bill is now thousands of dollars a month and

Jori VanAntwerp (14:52.364): That's just not tenable for most people. So it's a difficult problem to solve when we start talking about how specialized those particular pieces are.

Aaron Crow (15:3.432): Well, and it just goes to show like we have to you can't take a single thing and it's really you look at cyber people and they're good at, know, they see vulnerabilities or they see whatever, but you have to really truly understand the overall system to understand what that risk is. Right. So yes. And we see this all the time, right? Is, Hey, I've got it. I've got a system running windows XP running in a, critical environment. Well, that's really risky. You would never allow that in your it work.

Aaron Crow (15:31.356): Like if somebody tried to play to Windows XP machine into your corporate network, in theory, hopefully you have an environment that's going to block it. It's going to kick it off. It's going to say, yeah, go away. You can't use that here. We'll give you a brand new laptop. Don't try to bring that XP machine back in. Right. But in the OT world, that's the only system that can work. You can't upgrade it. You can't patch it because it's the only thing that works. And to replace it, it's not just replacing a computer like with a, okay, I'll just buy a thousand dollar laptop. No, you can't do that. You've got to replace the whole control system.

Aaron Crow (16:1.588): And then you've got to bring in air conditioning because it won't set all of those other factors that go into that. And I just, by doing all of that, I reduced my uptime, my reliability decreased by mitigating this cyber concern. Instead of, is there other ways that I could mitigate that cyber concern? Instead of reducing my availability, can I?

Aaron Crow (16:28.468): put other controls around it, can I harden the outside shell that I can reduce that cyber risk while maintaining my availability numbers that I'm looking for? And that's the conversation we have. And it's the same conversation we have in IT. We just have different factors. that's, to me, that's the biggest difference that we have in industrial controls and OT. It's the same problem. We just have different factors. The equation, the variables in the equation are just a little different.

Aaron Crow (16:55.944): Right? And we just have, you have to make sure that you really understand it. You can't take a business equation and dump it into the OT world and expect it to work. It's going to, you're going to get wrong answers. You put it into your calculator and you got 12, but it's really five or 105, like whatever the thing is, it's just off. And unless you know the business, you won't know that because I did it over there. It should work over here, right? It's all technology. The T's the same. So it should be the same results. No.

Jori VanAntwerp (17:24.378): And it's a fantastic point. It actually plays into cybersecurity in these spaces as a whole. When we talk about, like I said, Foxboro is the manufacturer of most of the controllers in this environment. Well, Foxboro could also be at an Amazon fulfillment center. It could be baggage claim. It could be part of pharmaceutical process. How do I know that? Right. And when I say I, I'm simulating an attacker, like how do I know what that does?

Jori VanAntwerp (17:51.522): That's where reconnaissance comes in. It's so incredibly important. And it goes back to my previous statement of what's going on at level one. Are we actually seeing reconnaissance? And this gets into a question of physical versus, you know, digital security. And as you said, there's so many things that we can do to mitigate that. And one of the best things that you can do in an industrial environment is properly segment your networks, right? So actually create those logical separations where it's not air gap, but it's logically separated and very difficult to get to. But that also has.

Jori VanAntwerp (18:20.974): in itself issues around, how do I know what's going on in that particular environment? Right? If it is as sealed off as it can be and that physical concern comes into again, that substation down the street. If I, if I want to walk down there with a pair of bolt cutters, I can get inside the fence. can get inside the building. And if I don't take down any type of, you know, operation that's going on, I can run reconnaissance all day long. I may be even able to leave something behind for later.

Jori VanAntwerp (18:50.412): Now we're crossing that realm of what's a cyber attack and what's physical when they really should be considered, especially in OT as being very similar or at least merged together in the threat itself down there at level one. And these things go back to one point I really want to get into, which is asset inventories. And I'm going be the first to tell you I've been in the industry for a long time. I know you have too. It's something we've been talking about forever and everyone claims to have an answer to.

Aaron Crow (19:18.098): and still have the problem.

Jori VanAntwerp (19:19.980): Yes, we still have the problem. And what's interesting is, and I want to give operators massive credit here. I've heard this stated on so many other podcasts, publications, et cetera. Operators don't know what's in their environment. No, no, no, no, that's false. It's completely false. Operators do know what's in their environment. What's difficult for them is over the last 25 years, what's changed? What's been updated? What's plugged in where?

Aaron Crow (19:36.852): Mm-hmm.

Jori VanAntwerp (19:48.386): Have I had any, you know, assets that have been replaced possibly with a newer model, newer firmware, different, you know, different set of problems that comes with that. And it's something that they struggle with. So when we say they need asset inventory, it's not because they don't know what's in their environments, because it's a very manual process and it may be 20 years old, right now to date. and they're walking around and visually checking off this item is here, but they don't know what's actually happening on that item. The asset inventory becomes a really big.

Jori VanAntwerp (20:18.186): issue at that point. But before we delve into that, just going back to the physical necessity of security in these environments is incredible because it does play into cyber. Cyber attacks in these areas, colonial pipelines are a fantastic example. Like you said, it wasn't an OT attack, right? But it affected OT. And that was simply because of how that particular environment was set up and the policies that were in place in it. And in that particular instance,

Jori VanAntwerp (20:47.736): talking a jump box. So they lost visibility into that environment, which from an operational standpoint, and, you know, a lot of folks don't understand this. It wasn't shut down because they couldn't build was shut down because they couldn't see what was going on in the pipeline. You don't want to pump thousands of gallons of oil in the middle of nowhere and not have an idea around it. So they shut it down and frankly, it was the right thing to do. I mean, it caused a lot of panic, but it was the right thing to do. I just don't think it was ever really explained very well, but that, that type of event.

Jori VanAntwerp (21:16.939): goes back to what you and I were just speaking to around segmentation. If you're properly segmented and you have highly available or redundant systems to be able to monitor in that particular environment or a way to manually go and check those environments, you're in a safer position.

Aaron Crow (21:34.046): Correct. Yeah. And that's the key. And I'm glad you brought that up on asset inventory and physical, right? These engineers and these operators, these plant managers, they know their environment very well. They troubleshoot it daily. If something breaks, they know exactly where to go to fix it, right? What they don't necessarily know is the cyber vulnerabilities, the firmware version.

Aaron Crow (21:58.804): some of the things that we as cyber professionals are looking for. So when we say they don't have an accurate asset inventory, that's what we're talking about, right? It's more along those lines. Like they know their system, they know how it works. They know how to troubleshoot it because they do it on a daily basis. They are constantly maintaining and keeping that system up and running, which is why it's running. So that's a very good point. I haven't, I haven't hit on that and it comes easily to me, obviously, because I see it and I know, but

Aaron Crow (22:26.386): others probably think about that. Well, how could they not know what they have? And to your point, it's not that they don't know what they have. It's that they don't know it from that perspective, from the perspective that we are looking for the knowledge that we want to export out to help them on the cyber risk side. They understand it from an asset availability and from a functionality of their system. They don't look at all those components as individual pieces necessarily. They look at them, they categorize them more as an entire system.

Jori VanAntwerp (22:54.766): And it's a different language. And this is an area where you run into that IT OT communication gap. it's, you know, you may have areas where it's better, but for the majority of, you know, plants that I've seen, you end up in a situation where OT is talking about a cabinet and the name of a controller. And IT is going, this is 10.160.1.3. The operator doesn't know what that is.

Jori VanAntwerp (23:22.456): Right. And if you're, if the naming scheme in that particular environment isn't documented, you can't tell them what it is. So asset inventory becomes incredibly important for bridging that gap as well in being able to tell an OT professional what's going on. And, it's not just cybersecurity, you know, I think one of the things, and I'll follow my sword here that we've failed to do for the OT, the OT folks as cybersecurity professionals.

Jori VanAntwerp (23:47.348): is to make it accessible and provide OT folks value. It's not that OT doesn't understand security or doesn't have security in mind, it's just that they're focused on resiliency, efficiency and safety. And those, we think of those things as one in the same, right? As cybersecurity, but there are, we're gathering information in these environments that could be used to help them. Whether that's preventative maintenance or just frankly,

Jori VanAntwerp (24:14.702): Letting them know if something's a little off in their particular environment these guys aren't normally looking at traffic patterns For example, right like someone would be in a sock as you just said, they're not trained in that way But if we could let them know that something's off with a PLC or an HMI and give them the location This is substation X cabinet B, you know device a That's something that I think is really powerful and really helpful

Jori VanAntwerp (24:42.156): And we need to start thinking about how to help those operators because remember they're in there day to day. The SOC analysts are involved. have, we have security folks that are involved, but that tends to be at larger organizations. What about the co-ops? What about the munis? What about, you know, your water treatment plants where there's 12 people in the entire organization. There's one person that's doing all of it. Your operators are actually out there controlling. need to involve them, but do so in a way that's additive to their job, their daily routine.

Aaron Crow (25:12.988): And that you hit on something so key. And I go back to when I started in rolling out what I call OT or cybersecurity in the industrial space and specifically around power plants. Like I came in and it was because of NERC SIP and we were doing compliance activities and we were doing segmentation and a lot of the stuff that I was doing, but I didn't have any budget. So I was going to these plants and I was saying, Hey, you're doing a control system upgrade Foxboro.

Aaron Crow (25:41.076): GE turbo control systems, Emerson control systems, Honeywell, like all these different systems we're doing upgrades on. And you have to add another 300,000 or whatever that number is to your budget because we have to do it in a compliant way. Meaning I need more stuff that's gonna have to be coming. And I don't have any money, which means you're gonna have to pay for it out of your outage budget, which means you're gonna have to choose to not do something else. Cause this money doesn't come out of nowhere. So they had to basically

Aaron Crow (26:9.768): you know, not, Hey, we're not going to do this other maintenance because we, got it. It got cut off the line. So I didn't go to them with a, we're going to do this as cyber. The way that I won friends and influence people is, you know, we talked about the, it's amazing how all these cyber security tools that, that Splunk and all the data that you can get out in all of these different types of tools, if done correctly, you can also get a lot of things that are valuable to the asset owner and valuable to the operator.

Aaron Crow (26:39.292): and knowing when systems are down or rebooted or you've got a power supply that's not functioning and it's beeping at you. One of the first plants we deployed this entire architecture to, they had redundancy. their switches, their entire network was redundant. Every device had dual NICs and whole nine yards. Completely redundant power, everything. Like a really great designed system. We plug in the system and one of the switches in this cabinet, in this back room was

Aaron Crow (27:9.384): complaining nobody knew it because nobody happened to open the cabinet and see the blinking light and it was it was not working. So it was a redundant system. So it didn't fail, but they didn't have redundancy at this place because one of the switches was off. And when we, when we pulled it up, we started getting all these alerts and we walk over to the cabinet and one of the power supplies wasn't working because it was overheating. We went in there and looked and there was a zip tie stuck in the fan. So we

Aaron Crow (27:38.484): pull the zip tie out and the fan started spinning and the switch came up and booted and started working again. So as simple as a problem, it took us 15 minutes to find the problem, but nobody knew to look. That thing had probably been complaining about that for who the hell knows how a zip tie got stuck in the fan. It probably fell when they were doing something else and nobody noticed it. But it's just a great example of instead of just focusing on fear selling around, you know, cyber and all these things,

Aaron Crow (28:7.098): make sure that we're considering the value to the overall environment and the organization, not just the risk, not just the fear, not just the China's attacking us or the grid's going to go down, but also what value can we bring to make their jobs easier? A lot of times cyber is bolting on and making things harder, making the process harder. It's hard to log in. It's harder, longer passwords, like all those things are more difficult, but there's a lot of benefit and capability we can give these operators and these asset owners.

Aaron Crow (28:36.626): by bringing the knowledge to them so that, like you said, hey, your system in rack two building four is blinking and you should probably go take a look at it, something's not right. And that's hugely beneficial to them.

Jori VanAntwerp (28:51.052): It is and I have a customer that I've been working with.

Jori VanAntwerp (28:57.902): that had an outage that was reoccurring, but it was random. now it didn't take the plant down itself, but the outage was happening. It would take down a controller specifically, and that controller would then have to be rebooted. Once that controller was rebooted, everything came back. It was all fine. Now, again, this is part of a redundant system. It didn't take anything down. They had been working with the vendor to try and fix this.

Jori VanAntwerp (29:26.220): And every time they worked with the vendor, the vendor would come in, they'd look at the configuration, they'd look at log files, they couldn't find what they needed, and they would ask for one thing, a packet capture. Most of these environments don't have a way to do packet captures. So the way that we really helped that particular customer was say, hey, well, let's turn on a packet capture and let's make sure that you have a rolling packet capture at all times so that you...

Jori VanAntwerp (29:53.386): If you have an incident, any point can go back and say, Hey, that happened at nine 10 this morning. I have a packet capture from nine until 10 30 AM. I'm going to go ahead and grab that and send it to the vendor and tell them the timestamp on when this particular outage happened. And then next time that it happened, they were able to submit that packet capture. They were able to identify which, what the actual problem was, which was a, it was a bad control set actually. well bad command that was being sent to that particular, PLC.

Jori VanAntwerp (30:23.124): And the vendor was able to fix it. They haven't had it out since. And that's one of those things where packet captures are incredibly important to us. You know, from a cyber perspective, I want to be able to go back historically and look for issues or hunt with new detections, right? Or new IOCs, IOAs, et cetera. But just for an operator, from a maintenance perspective, the vendor needs a packet capture. You've got it.

Aaron Crow (30:46.256): Yep. Well, and it goes packet captures and logs from windows systems that have a similar story where same thing, the power plant. We we, we noticed that the system kept getting rebooted and we didn't know why. and, and so we, basically sat a person at that spot and it was, it was always like Saturday morning at 2 AM and we didn't see anything going on. There was nothing in the logs. Like we didn't understand what was going on and we came in and

Aaron Crow (31:14.900): I had a person basically sitting in the control room right next to this computer and around 2 a.m. operator came in and he yelled some explicits at some point during this time of the day and he rebooted the machine. And my guy was sitting there watching and he was like, hey, why'd you do that? He goes, why did he do what? He goes, you just rebooted your machine? Like, yeah, freaking happens every time when I'm in this certain process and I do this certain thing.

Aaron Crow (31:43.314): I have to reboot it when it comes back up, it'll work. But for whatever reason, the first time it's like timing out. Well, it had a memory leak in the process and by Saturday it came around and when they tried to kick it off, it just froze. And the only way he knew how to fix it was to reboot it. But he didn't log it in his logs because it wasn't really a problem because he just rebooted it and it fixed it. For him, that was enough because he knew how to make it work. Like he knew the workaround, but he wasn't telling anybody. Cause again, for him, wasn't a big deal. It took 10 minutes. It was annoying.

Aaron Crow (32:12.690): but he knew how to fix it and everybody before him had trained him. If this happens, this is what you do. So it wasn't a non known issue. Everybody knew about it, but from us as an outsider and we were trying to help the overall system and the vendor and nobody understood what was going on until we physically put a person in that chair to watch it. Right. And then we started getting logs and all that kind of stuff, but it just goes to show again, another example of how little things like that. wasn't a cyber issue.

Aaron Crow (32:42.118): It wasn't any of those things, but cyber tools can help with a lot of these things that we're talking about beyond just cybersecurity risk availability and information that can help the asset owner and the vendors and the bars and all these, these people that tie all these things together. These are extremely complex systems with so many parts and components and pieces and ties and interconnects and third party connections and all these things that all of these things have to work perfectly.

Aaron Crow (33:11.616): or it can all crumble down. And these people, they do this on a daily basis. And we come in wanting to change their system without truly understanding. And that's where my push for vendors, for cyber people, internal, for business, the C-suite folks, to make sure you truly understand the process before you start dictating changes be done in these environments. Because...

Aaron Crow (33:35.666): what you're trying, sometimes what you're trying to do to help can actually be a problem and actually cause problems and impact your environment and actually cost you money directly beyond just a bill, but also loss revenue and safety and availability and so many things that can impact. And you don't realize it, obviously you wouldn't do it intentionally, but some of the reasons, sometimes that's what the only thing that stands between that is an operator that's willing to stand up against the CEO and say, no, we're not doing that.

Aaron Crow (34:4.296): And I've seen that so many times.

Jori VanAntwerp (34:7.596): That's actually, that's a really good point. And I'm sure all of us have heard horror stories about, when I'll say IT centric technology has been brought into OT environments and has caused an, an outage, you know, or a major, a major issue. And it's something that, that is done by someone who, who isn't hasn't had that paradigm shift yet, right? They're not intimately familiar with what's going on in an OT environment to know.

Jori VanAntwerp (34:34.466): just some of the basic things that can go wrong inside one of those controllers. And so I'd love to actually give an example of that for folks, especially if you're new to OT. So let's talk about PLCs, programmable logic controllers. They are custom built. They are usually running a real time operating system. VxWorks tends to be one that's run a lot. That real time operating system for all intents and purposes is almost firmware at this point. And all it does is interpret ladder logic. And that ladder logic,

Jori VanAntwerp (35:3.690): is then sent out over basically IO at the backend to do something, turn a valve, et cetera. That is so simple that we forget there's an IP stack on that, but it's not an IP stack like a Linux system has or Mac or Windows, where if you ping my particular system on, let's say, I don't know, port 1096.

Jori VanAntwerp (35:29.870): My system's gonna actually check and see if there's anything running on 1096. So that's gonna open, it's gonna check it, and then it's not there, and it's gonna turn it off. And it's not even gonna, the stack is not even gonna care. That is not the way that PLCs work. So you could open that port. Now it's listening on 1096. Now you have resource contention. Now that actually leads to, later on, can add to a memory leak or even other issues that where

Jori VanAntwerp (35:59.850): that PLC has gone down. And it's why you hear a lot of us say, we don't want active scanning when the truth is that there are ways to do active in these environments that's safe. Right. So you can't, you can't blanket statement that, but if you were to bring in something that just did sin floods, that's a really bad idea. Right. That's a really bad idea. But from an IT, an IT, you know, sock analysts never done anything in OT. They they're going in there thinking they're harmless, right? We're going to go in and they're going to run a quick scan.

Jori VanAntwerp (36:28.076): You see what's in the environment, see what they can do to help. And they may have just taken down three devices. And it is ignorance, but it's not willful by any means, right? But it's something that I want to bring up because it's a horror story that I hear all the time that makes OT standoffish when IT solutions are brought up. And it's not that we can't utilize these tools, even if they are IT-centric tools.

Jori VanAntwerp (36:55.776): It's that we got to understand the environment first and the people that know that as the operator. So if you're, if you're an IT person, even at a large organization, you know, like the OG and ease the SOCOs, the Dukes, you know, you want to go sit with your operators and your plant manager and understand the operation. Understand how it works. And then maybe even contact the vendor and talk to the vendor about how their systems work to get an understanding of how you can protect them. Because in some cases, what we would do.

Jori VanAntwerp (37:25.482): in an IT environment that would be very successful could be extremely harmful.

Aaron Crow (37:30.196): Right. Well, and it goes to in some of those large environments, I worked for Duke Energy, right? And I did, you know, very large OT program across three years. touched, you know, 900 substations and stuff. We deployed technology and OT cybersecurity across, you know, a big swath of their environment from generation to transmission, distribution, nuclear, kind of the whole gamut. But the part of that was

Aaron Crow (37:59.166): to do that successfully, to get business buy-in, because we were pushing all this stuff down, we got so much resistance. I pushed for building a lab. And it sounds silly, you know, building a lab, like it's expensive, it's hard, but we, you know, I justified it and I explained it and I got buy-in and it became one of the more valuable things that we did during that engagement. And the reason for that is because we had a like for like representation. So you mentioned Foxboro.

Aaron Crow (38:27.022): So we had a Foxboro control system. We had their Emerson control system. We had their GE Mark 6 turbo control system. We had the full stack of SEL switches that are in the substation, right? We had, you know, all the stuff that's in distribution, all the IEDs and all the components that are there. We didn't have all of them, but we had a good representation of the hardware that exists. So what that became was two things. We used it as a staging floor.

Aaron Crow (38:56.424): So once we perfected the design, then everything went through that staging floor and was built and configured in that environment before it went out. So we weren't sending raw devices. We were actually able to consolidate and build it in a factory. And then once it went out, it had the labels. It was already pretty good figured out the right IP address. had all the stuff. It got put on a pallet and shipped to wherever it was going, substation, generation site, whatever, and it was installed.

Aaron Crow (39:25.992): I didn't need all of my intelligence at all of these locations. Most of my intelligence could be in a central location and then they could support remotely as, cause again, this was part of this was during COVID. So we were, we were working remote. So we were having to do this. But the second piece to this, there really three, the second piece of this was we were able to test what you just talked about. Like, Hey, I want to bring in this product and I'm not gonna name any product names particularly, but

Aaron Crow (39:52.552): we could test whatever system or whatever product or capability we wanted to and see directly at least to some 80 % validation that it's not going to break the system. And I know what it's gonna get. I know what good looks like. I know what to steer away from. I know that I can't ping this PLC, I can't run in map, I can't do these things, but can I go active? What does it respond with if I do this? And I could have different versions of software. And then the third piece that it did is,

Aaron Crow (40:21.692): It was a good training and walkthrough. So executives or IT people that had never been in a substation, know, yes, Duke Energy is a large company. Everybody that's working there is not an operator. Like I grew up working in power plants. Many of those people, they came out of college and they're IT people. They've never been in a power plant. Yeah, they work for Duke Energy. That doesn't mean they've ever been to a power plant. Maybe they went on a tour one time, but...

Aaron Crow (40:48.008): You know, they never worked in an outage. They've never worn the hard hat and steel toe boots and been there, right? So, so seeing that and seeing the equipment, seeing the PLC, seeing how everything connects together was extremely powerful and letting them understand why this is different. And, and, and, know, you can see the IT stuff and you can see the OT stuff. And even though, you know, a switch, an industrial switch and a, and an IT switch underneath that they may have the same operating system, but they're different. And there's,

Jori VanAntwerp (41:17.379): Yeah.

Aaron Crow (41:17.780): there's obvious differences that you could see in the form factor and the power plug in and how they're mounted, you know, on a den as opposed to, you know, 19 inch rack and all these different things are just a little different that you can tell. And, and they're, for reasons like all of the equipment we put in, and you talked about it earlier, had to go into a place with no air conditioning in, you know, Georgia heat or Florida heat or Texas heat and, and be able to sustain. we couldn't just take something.

Aaron Crow (41:46.322): you know, that would go into a, you know, air conditioned space in a data center. Like most of the IT stuff goes, it had to be able to have no fans and be able to sustain when the air temperature is 145 degrees. And then it's going to add its own heat on top of that. Like all of these factors that you don't necessarily think about. And when you see it in person, it's just like, like it was like a light switch for some.

Aaron Crow (42:11.604): for a lot of folks as they started seeing it and why it's so complex and why it's different and why it takes all the steps and processes and became that training environment. So it was hugely expensive, but if you really look at it at the total cost of ownership of, we avoided one plant from our facility from tripping or causing an issue, it more than paid for it. let's say it was, I have no idea. I don't remember what the cost was. Let's say it was a million dollars. Let's throw a big number.

Aaron Crow (42:42.078): how quickly would it take to have that be a cost? If I take a plant down for a day, it's gonna be way more than a million dollars of lost revenue from all that type of stuff. Not counting if there's any safety or anything like that. It's very easy to understand how this is a value add. And there's a lot of ways that you can do this without building a physical million dollar lab now. There's with technology today, cloud ranges and all sorts of things like that. But still it's a different value statement to help people understand that environment.

Aaron Crow (43:11.636): It's different.

Jori VanAntwerp (43:13.014): I completely agree. actually I'm going to give some call outs here, shout outs. And I know I'm going to miss some folks, so I apologize if I miss you, you know, I'll talk about some of the bigger folks that we see, Duke, OG &E and SoCo are all doing similar things and they share that information. And I think sometimes that's forgotten. You know, they show those information with organizations and groups and associations, but they're also very happy to share it with other energy providers. And it's something that I see

Jori VanAntwerp (43:45.059): I'm sure that things like that go on in oil and gas, and I'm sure the things that like that go on in, know, manufacturing pharma, et cetera. But energy is becoming very good, good about it. And there's, there's some unsung heroes in there, which at first energy is one of them. First energy does a very similar thing and they, they bring people in to that lab and they teach them what they do. It's an amazing, it's, amazing. And they will openly invite you in and share that with you.

Jori VanAntwerp (44:12.546): And the same can be said for salt river project here in Arizona. Very big on not only having that lab, understanding what's going on in that operational environment, but sharing that information so that folks that, you know, that co-op that's got 12 people can actually come in and see how they might be able to stair step their way into being more resilient and protect the environment as well. And one thing that SRP does that I'm sure other people do, but it's something that I haven't seen much elsewhere is.

Jori VanAntwerp (44:41.944): two of the folks over at, at SRP actually go in and for years would work from a location. They would go to a power plant and that these are sock guys would go and sit and work from that location and get to know the operators. And it built a trust over many years between the operators and the sock to know that the sock guys were truly interested in what was going on in the operation and there to protect them. And on the op, the sock side, they learned.

Jori VanAntwerp (45:12.224): What goes on in these environments all the way down to things that, if you haven't been, you know, a little bit of, of crawling through these plants, you don't understand how forecasting happens. Or you, you talked about startup processes. If you haven't seen a startup process and phasing a generator into a grid, like it is an amazing thing to see. You can't just flip a switch that doesn't just happen. Right. Especially when you, you have, you know, things that are like combined combustion.

Jori VanAntwerp (45:40.470): Right. There's, there's, there's complexity to this. and I really love that these organizations are out there and why I'm naming them more than anything is so that co-opting municipalities reach out to these folks. Honestly, I mean, Duke, OG &E, SoCo, SRP, NYPA is in there, Nextera, First Energy. There are, there are a lot of folks that do really good work around this. We'll be happy to share their time and their knowledge with you and they're in your industry.

Aaron Crow (46:8.090): Absolutely. Excel, Exelon, you know, all the big players are there happy to reach out to me. You know, I built the lab for Duke and know the folks that run it. It's amazing. And I've done work at Excel and Exelon and Nextera and Southern Company and a lot of these different environments. My hands have been in a lot of those places and I'm not saying that to toot my horn. I'm saying I'm impressed with the work that they're doing in these spaces. And it's really awesome to see.

Aaron Crow (46:34.676): you when you look at the the 17 critical infrastructures, I do believe that, and I think I said this earlier, but that, you know, power generation or at least the power grid environment, utility industry is probably the most advanced, but still needs a lot of help, right? And we all, it's, the thing with cyber and all of this is it's never done. It's like, you you go to the gym, you don't go to the gym once and say, okay.

Aaron Crow (47:1.864): I did that now I'm done. I don't have to do that anymore. It's like you constantly have to maintain that and then you know you get older and you have a knee injury and you get you know all this different stuff thrown at you. It's going to be constant or you know like we talk about shooting all the time. I can't go to the range one time whether it's guns or or or golf or tennis or whatever your sport is. You have to maintain that. You know there's one reason why I don't play golf is it's because I don't want to spend the time that it's going to take to be decent at it consistently.

Aaron Crow (47:30.952): I'd much rather go do that, you know, throwing lead down range. Cause to me, that's just more fun. But you know, I used to play golf, but now I'd just rather go shoot.

Jori VanAntwerp (47:39.446): It's all these things are, I always say martial. It's a martial skill. It doesn't matter if it's baseball. It doesn't matter if you're using your brain and you're doing penetration testing. It's a martial skill. And if you don't flex it and dust it off, mean, I talk about being technical, but I've been on the business side for a long time now. So I go back and I try to dust off those skills when I can, because it not only does that skill atrophy, new things come out, new ways of doing things come out, a different approach to working out.

Aaron Crow (47:44.326): Yep. Right.

Jori VanAntwerp (48:8.374): a different approach to your stance, a different approach to your golf swing. All of those things, you know, end up changing and we can learn more. And if you're interested and you want to be involved in it, you have to stay involved in it.

Aaron Crow (48:19.954): Yeah, absolutely. And it just goes to show like, you know, we're the people that are successful and have built teams and, you know, we talk about how there's a resource problem and we don't have people that have the knowledge that we're looking for in OT cyber security. And there's not that many people that do this type of stuff. And, but I built a successful team of people that had zero experience in OT. They'd never been to a power plant before they'd, they'd been an IT person. They'd never been done cyber.

Aaron Crow (48:47.676): Like they'd done these other things. And I'm like, I don't care. I need to build a team that I can, I know I can work with and I can trust. So like I brought a guy that had worked for me twice before he was a Marine. He'd worked in the legal field and law firms and supported, you know, their technology. And I brought him in and when I, when I sent him the job wreck, because I didn't write it, of course, HR wrote the job requirement. He's like, I don't, I'm not applicable. Like there's so many things on this list that I don't check the box on. I'm like, dude, I'm the one that's

Aaron Crow (49:15.482): hiring this thing. I know you would be a good fit. Just freaking fill the thing out. And if you're listening, he knows who I'm talking to. I won't call him out by name, but he knows who he is. But he was the perfect person, but it wasn't in the job rack. He didn't align with the things that HR signed up. I've said for a hundred years, it's more important that the people are somebody that is a go-getter, that can

Jori VanAntwerp (49:16.974): you

Aaron Crow (49:45.032): you know, deal with ambiguity that can, can, you know, work well under pressure that is a team player. Like those, those, those skills are things that are harder to teach in my opinion. And if I, if I have somebody that I know I can work with, I'll teach you the technology. We'll send you to training. We'll, we'll, put you at the plant. Like we'll figure those things out. Those things are easier to teach than it is the other, right? If I can build a team of people that I know will work well and that will

Aaron Crow (50:9.776): ask questions and have a questioning attitude and, you know, we'll, we'll approach problems and, and, you know, be inquisitive, and, and driven. I can do, I can conquer the world with that. Right. It doesn't matter on the flip side. Sometimes you get the smart guy that's super smart and thinks they're smarter than everybody else, but nobody wants to work with them. It's like, I don't need that guy. I'd rather have somebody that's not nearly as capable as that person because nobody wants to work with that person. Like, yes, you're smart.

Jori VanAntwerp (50:31.982): Hmm.

Aaron Crow (50:39.218): Congratulations. Nobody likes you. And I could say that firsthand because I've been that guy in my past.

Jori VanAntwerp (50:42.296): Yep.

Jori VanAntwerp (50:46.742): I think, I think we all have, right? You got to be like the smartest guy in the room. And honestly at this, you've hit the nail on the head at this point in life, it's, about staying humble, but staying ambitious and being the dumbest guy in the room. I, I would rather be surrounded by people that know much more than me so that I can learn from them. then, you know, being that, that all knowing person in the room. And I think you're absolutely right about. OTE in general, it's something that we can teach, especially, if you've already been in technology.

Jori VanAntwerp (51:14.166): Yes, it's a paradigm shift. If you're humble enough, you're ambitious enough. It's not a hard paradigm shift. This is not a black box. It's not magic, right? It's just a different way of thinking. Our consequences are different and tend to be physical.

Aaron Crow (51:28.786): Right. And as long as you're open to looking at a problem from a different perspective, because ultimately the problems are the same. I have vulnerabilities. I need to patch. I need to have secure mode access. I need to have availability. I need to like all these things, but the answer is not always the same. IT, I'm just going to patch. As soon as it patch Tuesday comes out, I just push all those patches down to all my systems and I reboot them because I have a maintenance window on Friday night between midnight and 3 a.m. on Saturday morning. Like that's just my window.

Aaron Crow (51:56.584): I think we all want our power to work on Friday night. We don't, we're not rebooting the power plant to patch it. So we don't have an outage window like that. So yeah, I can't do that. So what else can I do? That's not that.

Jori VanAntwerp (52:0.878): You're spanking.

Jori VanAntwerp (52:12.044): yeah. And we could, we could have a, I think that we could have a whole nother podcast, honestly, just on, on vulnerabilities and risk quantification in itself and how difficult it is in, an OT environment, because it changes from location location, like substation to substation. I'm not talking company to company substation to substation. You know, a threat that's, that's a, Hey, I need to do this right now at this substation might be something you do next at the following substation.

Aaron Crow (52:27.348): Correct.

Aaron Crow (52:35.828): Correct. And that's one of the things to your point. And you're right. think we could, I know we could have an hour long or probably longer conversation about this. and, but you know, one of things I always, always preached as I was pushing this out is patching and vulnerabilities. And I've talked about it at DEF CON on the stage and I've talked about it at multiple conferences and in-person and to executives is you may not, you may not patch this thing until an outage.

Aaron Crow (53:5.336): In some place like nuclear power plant we talked about, those are 18 month refueling outages. It's the only time you take the plant down is when I'm refueling it. So the only time I'm gonna do maintenance on that thing is every 18 months. So if there's something, if I just had a refueling outage last month, it's another 17 months before I'm getting back in there to make a change. Unless there's something critical, right?

Jori VanAntwerp (53:25.932): Yeah, how do I? Yeah, absolutely. And you get into this thing too, where I think all of us in IT are trying to let's eliminate that vulnerability, right? I want to eliminate this exposure. I want it done. But the fact is that in many of these environments, it's about mitigation. How do I mitigate that risk to a point where it's an acceptable risk at this point? Because I don't have a way to eliminate it or I can't eliminate it for 18 months, et cetera.

Aaron Crow (53:55.016): Yeah, well, and some prime examples of that are, you you walk into an into a control room, there's no passwords. Machines are wide open. And why is that like? Well, that sounds dangerous. Why would you do that? That's risky. Yeah, it is. And what is more risky? If I'm at a nuclear power plant and the operator forgets his password or it takes him 10 more seconds to log into his machine and he can't control the reaction and then you have a reaction that's out of control or

Aaron Crow (54:22.836): had you rather just make sure have other mitigating factors that to get in that room, you've passed so many barricades of security and validation that if you're in that room, you can sit down at the computer screen and do what you need to do. Cause nobody's in that room that isn't authorized to be in that room. Like I've been in a nuclear power plant, I supported one and you don't get in the control room by accident. Like nobody wanders in. You've gone through a missile door.

Jori VanAntwerp (54:43.010): Yeah.

Aaron Crow (54:49.476): and through armed guards and scanners and all this stuff. If you're in that room, you deserve to be there. And that room is 24 seven monitored and it's got cameras and the people that are in there, me, even if I have authorization to be in the control room, if I were to walk up and sit down at a computer screen without asking an operator if I could sit at their station, they are going to very quickly move me and escort me out.

Aaron Crow (55:16.904): Right? Even, even as a badge person, got in the room, I'm supposed to be there. They, they, there's, there's other mitigating factors that says, Aaron is allowed to be in the room, but he is not an operator and he has no right to be at this computer screen. And you are a hundred percent authorized to forcibly move him out of the way.

Jori VanAntwerp (55:34.882): Yeah, we've, we've, mean, we've come full circle. It is the physical and cyber and cyber side combining in these OT environments that you cannot think of risk in one dimension. You have to think of it in both. And when you have the, the, and that's business as well. Like you were just saying that's policy, that's training, right? That's not even something that you would, you would define in a system somewhere that's frankly, it's culture. You've, you've actually, you know, instilled this sense of purpose and

Jori VanAntwerp (56:3.990): Empowered people to actually go out and do those things and it's incredibly important in our OT environments It is not just cyber. It is physical and we you know, I think It was a horrible time. But another example of that is, you know, we had all of those substations shot up like I could chuck a Chain over this the fence of this substation and caused just as much damage as a cyber attack So am I if I'm physically able to get there?

Jori VanAntwerp (56:32.538): There's a lot of things that come into play first that I personally worry about over someone spending the time to actually breach that location remotely, understanding what it is, doing all that reconnaissance and then figuring out how to attack it. So they have a lot of concerns that they need to balance and budget goes where the biggest hole is. You you don't have a fence around your substation. Probably going to up a fence first.

Aaron Crow (56:57.266): Well, it just goes to show like you need to be having it all comes back to the we're one team, right? And the way I say this a lot of times is you're on the same team. We're in the same Jersey. We have the same goal. We want to win the game. When in the game obviously looks different for every team, but for our team, we need to understand what that looks like. And if I'm in, if I'm on the OT team or the IT team or the, the, the operations team, we're all one team. We're all wearing the same Jersey. We all have the same goal. So we should come together and talk about it. Like,

Aaron Crow (57:26.386): You know, yes, offense has a plan and defense has a plan, but overall the head coach is driving the whole thing. The CEO is the head coach, right? And, and when I'm, when I'm going to a place and I'm saying, Hey, these are the risks that I'm concerned about. Secure mode access and vulnerabilities and old systems and you know, blah, blah, blah, blah, blah. I want to also not just force down my controls on how I think they should be fixed. Cause I probably don't know all the, where all the dead bodies are hidden in this system.

Aaron Crow (57:54.856): My question is always to those operators and the plant managers and the control system engineers and those guys, like, if you were going to take this plant down and you were a bad actor or you were, you know, somebody that made a stupid mistake, not even a bad guy, just somebody that made a stupid mistake, how would you do it? And they'll tell you and like, okay, which is more likely? Is it somebody shooting a transformer with a deer rifle from, you know, a hundred yards away?

Aaron Crow (58:21.584): Or is it that they hack in and do, you know, all this other stuff? That doesn't mean we don't care about both risks. Which one is more likely? Because that's what it really comes down to. If I'm going to spend a dollar or an hour of time, where is it most valuable for me to focus that time? Is it more likely that a bad actor is going to hack in remotely and go through all these things? Or, you know, a bad actor is going to get into the control room at a nuclear power plant and sit down at this computer and do something at the keyboard?

Aaron Crow (58:50.516): That's really unlikely. Like if they've gotten that far, you're having a bad day because they've taken out physical security stuff and they've been shot at and like they've gone way far into your environment. Like there's all sorts of other things that are going on beyond just somebody accidentally being in this room sitting at this computer. So it's really around approaching the problem differently and making sure you have the right people that you can ask the right questions and be willing to say,

Aaron Crow (59:18.492): Yes, I understand that your risk is a risk, but it's not as important as my risk. So it's not going to get done maybe ever. and come back to the table with another way that we can solve this. Cause we're not going to patch this thing or we're not going to, we're not going to lock the machines or whatever the thing is that you're trying to push through.

Jori VanAntwerp (59:35.406): Absolutely. And, you know, I had a professor actually tell me this a long time ago and it's something I actually speak with cyber professionals about all the time, which is whether you're headed into a discussion or debate, you should never start a discussion or a debate unless you're willing to change your mind. And that's the truth. So always enter that conversation with the door open so that you can understand where they're coming from and reach a middle point. If you can't change your mind, you're never going to meet anybody halfway. Frankly, you're never even going to convince them.

Jori VanAntwerp (60:4.980): of meeting you have.

Aaron Crow (60:8.372): Yeah, ego is the enemy.

Jori VanAntwerp (60:11.342): 100%.

Aaron Crow (60:12.510): So we've talked around a lot of topics today, so excited about that. in the next five to 10 years, always ask folks this, five to 10 years, what's one thing that you see coming up over the rise and that maybe is concerning in cyber? And maybe one thing that is exciting that you see coming up over?

Jori VanAntwerp (60:29.550): So I think concerning, and we've kind of danced around the topic in the discussion here, is really around the skills gap. So we talk about this a lot, and I'm 50-50 in here because we've got job recs that say, I need to have 10 years of experience, a master's degree, a CISP, and every other certification under the sun to be able to go and do a junior SOC job.

Jori VanAntwerp (60:57.260): That's not realistic. That's not a skills gap. That's that's inappropriately, you know, yeah, it's an expectation gap. but at the same time, the, thing that I've noticed and you and I started off this conversation, you know, before the podcast talking about some of our geekery, and, how we started is, the things that I learned fixing, breaking and building computers in, you know, the very early days of, I mean, weren't even really a career in my teens, were what

Aaron Crow (61:1.223): expectation.

Jori VanAntwerp (61:27.276): became the foundation for building robust systems and security. And one of the things I see coming for security folks, and we're already starting to feel it, is that we've rushed to fill that skills gap. And the way that we've rushed to fill that skills gap is a lot with tools and educational processes that do not go for the basics. So they don't start someone at level zero or level one and talk about how a system works and why it works.

Jori VanAntwerp (61:56.756): why that's important so that you understand how to protect it. They start at level five. And that's where we get into situations where we're now creating a different skill gap, where someone will come in with a fantastic understanding of security theory, security methodology, and compliance, but have no understanding how the business or frankly the system itself, just the computer works. And I see that coming to bite us.

Aaron Crow (62:24.947): Mm-hmm.

Jori VanAntwerp (62:25.210): You know in the future and it's something that I'm trying to get ahead of by helping Really push for some of those more fundamental things. We've made things more approachable like coding right you have scratch you have blockly you have things that you can do that you teach your kids to code but Are you teaching them to code? Are you teaching them to you of colored blocks? And it's something we need to to kind of get around and I think there's there's for us to do that and on the other side of this is is frankly just

Jori VanAntwerp (62:52.066): the jumps in technology. I'm sure you hear this a lot on this podcast. AI is always very interesting to me, but even more so around cybersecurity is what AI is doing for hardware. So we're, you know, we're seeing machines that are small, affordable and dedicated to doing these amazing computational models, which is more important to me than AI because those computational models are things that we can use to defend environments and they're doing it on a budget.

Jori VanAntwerp (63:22.350): And I mean, the things that you can do on in some of these environments, just running a graphics card, for example, blows, blows things that we were doing at a speed. That's incredible. Those things that we were doing just two years ago out of the water. And I see that exponentially growing. Now, of course that's a, that's a double-edged sword. So as, as our hardware gets better, right? That the Threat Actors hardware is to get better. But I think it's going to open up a different realm of how we look at things, because no matter if we talk about

Jori VanAntwerp (63:51.552): AI, machine learning, et cetera, it all comes down to static and dynamic or algorithmic detection. It's what we do. And until we can get really good about building models and understanding how people interact with the environment, it's going to be harder and harder for us to detect the unknown. So I see a lot of really interesting things coming with that.

Aaron Crow (64:11.548): Yeah, and that, for me, that's exciting because I've preached for a long time. It's really hard to protect these environments at that level to know what good, to know what bad or to search for bad until I really truly understand what good looks like. And that's one of the things that's a translation problem where in IT, we know it really well because we know what RDP looks like and we know what a secure HTTPS looks like. we have so many examples of that data.

Aaron Crow (64:40.766): that I can know what good looks like and I can search for those bad things, right? So blacklists, I can do all those things and I know what bad packets look like. I know what bad program behavior looks like and memory and all that kind of stuff and processor and something's out of whack. In an OT environment, we're so far behind, nobody's done that. And it's not like to your point you said earlier, site A and site B in the same company that maybe were designed the same thing 20 years ago, they're not the same anymore.

Aaron Crow (65:10.260): So I can't even compare site A and site B because there's so many different components. They're so vastly different. AI can really, I believe that's one of the things I'm most excited about with this is using AI for things like that to really truly understand and map out what good looks like. So once I truly understand what normal and good looks like, which is gonna take a while with existing systems, but maybe AI can fast track that. Then I can really start enhancing me. Okay, now that I know what good looks like, anything outside of these thresholds and parameters,

Aaron Crow (65:40.050): Those are things we need not necessarily bad, but there are things that we need to look for. They're the needle in the haystack or the needle in the stack of needles is more accurate that we can actually start focusing on and really expedite our detection and monitoring in these O.T. spaces. And you talked about that level one stuff. That's where the real meat and potatoes is going to come down to. And we can get to that level and truly understand again, this is what good looks like at this substation.

Jori VanAntwerp (66:2.446): I

Aaron Crow (66:8.774): anything outside of this we need to pay attention to. Again, not assuming that it's bad, but I'm just gonna assume that it's not normal and I need to look at it as opposed to right now, if I send all that data, which is what happens, I send all that data to my SOC analysts, they're gonna be like, I don't know, is it supposed to do that? Is that good? Is it bad? I don't know what you're showing me here. Like, it's just not sure what to do with my hands.

Jori VanAntwerp (66:24.599): Yeah.

Jori VanAntwerp (66:30.299): It's where active blocking, for example, and anomaly detection fall flat on their face because in an environment where we're baselining, right? I'm going to baseline this environment and understand what's going on. Hopefully you never ever see a safety system fire. What happens when that safety system fires? Are you going to block that because it's an anomalous activity? And this is where, as you said, as we get technologically more advanced and we get into AI and building models and understanding every possible command that safety system can send.

Jori VanAntwerp (66:58.274): we get into actually knowing what good is.

Aaron Crow (67:1.172): Exactly. That's exciting. Well, cool man. Hey, so, it's your call to action. Like what do you, how do people get ahold of you? What do want people to know about you, your company, all that kind of good stuff. Where are you going to be at those, those fun things?

Jori VanAntwerp (67:12.540): so the next event that I'm planning to attend, of course, is S4. Hopefully I'll see you there. As far as where to get a hold of me, emberOT.com is the easiest way to go and grab, you know, go and grab some information about what I do and what my company does, and also seek any contact that you'd like is there. And EmberOT, just as a quick overview, is a cybersecurity company that's focused on OT, and it encompasses a lot of things that we've actually been discussing, where it is a...

Jori VanAntwerp (67:41.930): It is a sensor that can actually deploy all the way down to level one. And it provides information not only around cyber, but around OT detections as well. So we do that asset inventory. We understand how devices are talking to each other. And we do detections in those particular environments. We do this passively. And we do it at a form factor that allows us to be installed directly on switches or existing hardware to make things a little bit more easy and flexible for operators to actually get into the environment and begin a

Jori VanAntwerp (68:9.718): in getting insights and actionable data out.

Aaron Crow (68:12.850): Yeah, that's exciting. And that's, one of the problems that we have in these spaces is we need all these tools and capabilities. Then I got to pull another piece of box and I got to make sure it fits in, you know, all the power and all the things that are the problem and the expense and support and all that type of stuff. If, I can deploy stuff on existing equipment, that makes it so much easier to deploy and speed and all that type of stuff. And I start getting value really quickly from that. So that that's exciting.

Jori VanAntwerp (68:36.282): I say this a lot and I really, really mean it. We need to meet operators where they are today because some of their environments are 25 years old, maybe even older. And we need to meet them where they are today and help them show us where we both need to go in the future. It's a joint exercise. And that's what the Embro-OT's aim is, is to actually start helping them today and then for us to journey forward in the future.

Aaron Crow (68:40.852): Exactly. Maybe hold her.

Aaron Crow (68:50.260): Correct. Absolutely.

Aaron Crow (68:59.302): Awesome. We'll definitely put all that in the show notes folks. If you want more information, reach out to Jory. If you have an OT environment and you're looking for that level of capability, reach out. there's opportunity for us to dive further into that at another time, but just reach out and they'll have more than enough information on your environment and all that kind of stuff. So thank you for your time today, Jory. I appreciate the time and digging into these fun topics that we love geeking out on.

Aaron Crow (69:27.567): and hopefully the audience enjoyed a little bit of that. Until next time, have a good day.

Jori VanAntwerp (69:32.600): Thank you.

Transcript lightly edited for readability.