Troubleshooting OT Environments with Next-Gen Firewalls with Adam Robbie

Aaron Crow (0:3.870): And there we go. All right. So it took a second. was, it was kind of frozen. Thank you for joining me today. Protected All podcast. I'm your host Aaron Crowe. Today I'm excited about having Adam on. Adam, I've worked together for, quite a while now. He just happens to work at one of my quote unquote favorite firewall companies. I have a lot of experience actually using Palos in, in, in OT.

Aaron Crow (0:30.242): and actually transitioning from other non next gen firewalls. And again, this is, I'm dating myself. I haven't been a firewall admin for a long time, but you know, doing that transition back in the day, we rolled it off. I think, I think we supported, I don't know, 150, 200 firewalls across, you know, power generation sites. So it was really a breath of fresh air to be able to use some of the capabilities in the firewalls. And honestly, I'm diving a little bit deep before I even let you talk, but

Aaron Crow (0:59.254): You know, some of the things that the Palo really did that was that was so different for us is I had a team of six and they none of us were necessarily deep level firewall people like we had to support the computers and the virtualization and all of the tech stack and a lot of the problems that we were having was just troubleshooting like, hey, this thing isn't working and we don't know why and it would go into the firewall and we wouldn't know the other firewall, not the Palo.

Aaron Crow (1:24.876): So whenever I was able to put in the Palos, we were able to track the traffic and actually see, the Palo did this with the thing. Like it accepted it on this rule and it went out this way. So we could very easily troubleshoot the packet and see it come in and go out. Okay, it's using this rule and all that. So it really made us beneficial just from a troubleshooting perspective, even beyond the next gen firewall capabilities and application aware rules and all that other cool stuff that it does. But just that basic level capability was.

Aaron Crow (1:54.208): was night and day difference in an OT world where we didn't have that capability before. So with all of that to say, Adam, why don't you introduce yourself and tell us about you.

Adam Robbie (2:4.916): Hey, Aaron. Yes, I am Adam. I am the head of the OT Threat Research at Palo Alto Networks. And for disclaimer that what I will say here, it's my personal opinion, not company opinion. I know you love Palo and I love Palo as well. And we did really have a good time working together. And I also I agree with you. There is a lots of things that I like about the firewall.

Adam Robbie (2:33.770): and how the firewalls can be in the OT environment and bring, like it's a game changer when it can give you more details and more packet analysis. And I'm more than happy to talk more about these, some of these capabilities or how we actually used it also for threat research as well.

Aaron Crow (2:55.970): Sure. So what is it? What is it that you do in your role at a 50,000 foot view? If you're somebody, if you're talking to somebody in an elevator and they say, Hey, what do you do for Palo? What, is it? Like it's a cool title, but what is it that you do?

Adam Robbie (3:10.660): So this is a really good question because every day I wake up, I ask myself the same question. What that means, what do I do, what do I do today? And basically, I'm trying to help identify threats that impacting OT environments. How can we do that? There's so many different ways we can do that. And that's why every day when I wake up, was like, how can I do this today?

Adam Robbie (3:37.326): How can I help finding new threats? How can I help inform our industry about new threats or evolving threats or even old threats? Like what are the information they need? And I usually like to start my presentations, my classes, any meeting that we are the cybersecurity professionals, we are cost, we are not revenue. And that's why the CFO don't like us.

Adam Robbie (4:7.398): Like, yep, been in lots of meetings and I know the CFOs very well. Like we are friends outside of the meeting, but inside the meetings like, nope. So when I go to meetings with that mind, it helped me also to understand what information are important for our industry to know about these threats, because these threats help them to make decisions, right? So.

Aaron Crow (4:8.300): Very true.

Adam Robbie (4:36.898): What information that can help make these decisions is what I focus on every day. And that day by day today can change. Sometimes there's a new malware. OK, so we need to focus on new malwares, malware analysis, reverse engineering, new vulnerabilities. Then I'm focusing on CVEs and signatures and alerts and so on and so forth.

Adam Robbie (4:59.746): New technology, then I'm looking for new attack vectors at attack surface and how that can impact and do some validation and use cases, so on and so forth. More information or insights, like they want to know like what happened in the past year or what's the new trend. like publishing white paper, collect threat telemetries and doing analysis and correlation between data to get more insight and more meaningful.

Adam Robbie (5:28.975): information. So as you can see it's a very different angle to do some threat analysis. It depends on when you wake up what's gonna happen. What comes to my email?

Aaron Crow (5:44.226): Sure. Well, what I love about that though is that you are focused on OT and, and, know, Palo Alto obviously is a very big company. you know, they have firewalls and everything. I'm sure from Amazon to Google and Facebook and any, any big name company you can think of and the IT problems and vulnerabilities and threats and how I attack things. talk about this all the time, but it's different. Yes. The tech stack is the same. A firewall is a firewall, but how I use it and what

Aaron Crow (6:13.758): I'm looking for and the attack vectors are different. Like so it's different enough. And that's what I love about Palo is that you guys have a dedicated environment and you're not the only ones, but I love the fact that you guys are dedicating and investing because every company has OT. Obviously the Duke Energy of the world's OT is a little bit different than Amazon, but it's all OT. And if you own a building and like OT is everywhere, but it's enough of a difference that dedicating

Aaron Crow (6:42.410): time and resources to looking at what are these problems? How do I solve it in IT? It can be the same problem, but I solve it differently in OT instead of how I do it in IT.

Adam Robbie (6:55.094): Yeah, I totally agree with you. And when I originally joined, I came in just to focus on OT and then I start, my rules start to grow and then start to hire people in my team. And I start to work cross-functionals also to impact other teams as well. Like, okay, I may not, I cannot do everything by myself or like I have a very small team and we are still growing. But I'm taking advantage that it's a huge company.

Adam Robbie (7:24.192): And we have lots of other experts in different fields. so I tapped with those experts to say, like, hey, let's collaborate and work together. I will bring the OT side, you bring the IT side, and we can work together on building that model or building that research and so on and so forth. So it's definitely an advantage to see a dedicated team just to talk about OT. And it comes with its own challenges as well.

Adam Robbie (7:53.816): Like it's a good thing to have and it's also challenged too that you need more flexibility. Sometimes you can feel isolated because you talk OT while everybody's talking IT. And then you take that rule of I need to keep educating everybody around me about why it's different, how it works and making like, you know, any decision making. So.

Adam Robbie (8:21.878): It is a challenge that I like to have every day. So I'm full for this protein.

Aaron Crow (8:28.172): Well, you know, some of the some of the very simple differences that, again, from my, you know, implementing firewalls like like the Palos into these spaces, the difference may be as simple as, know, when I see a threat or a vulnerability and the firewall detects it in an OT world, I may just acknowledge or notify as opposed to on the IT world. I'm going to block kick out, you know, be a lot more aggressive on my response.

Aaron Crow (8:55.254): Whereas in an OT world, I'm probably just going to raise my hand and say, hey, look over here. There's something going on, right? To be really clear. And that's because of the availability and safety and all the things that go along in an OT world and why it's, it's a little bit different.

Adam Robbie (9:0.826): Mm.

Adam Robbie (9:9.826): Yes. And also, like we recently built actually also an OT lab, like a specific environment and a rail and OT environment, like the full Purdue model stack, like from level zero, full automated factory system and all the way to level four. And just by putting myself on someone else's shoes, like, okay, I'm trying to install this firewall here in a real environment. What are the challenges?

Adam Robbie (9:40.062): And I had an automated engine, like an automation engineer, like an electrical automation engineer. have a lab assistant, have, know, all my team and there are IT, there's OT, so on and so forth. And doing commissioning for the lab and something is not working. The first thing the automation engineer will say, it's the firewall.

Aaron Crow (10:9.858): Hahaha!

Adam Robbie (10:12.634): Exactly.

Aaron Crow (10:13.044): heard that before. You have to prove it's not the firewall before they believe you, which goes into my very first story. But go ahead.

Adam Robbie (10:20.920): I was like, dude, we are working the same team. So it was really like, it's like, OK, now I get it. Also, to prove that's not the firewall, we put any, any, any, any, just here, prove it. And it will be surprising if it worked. It may have nothing to do about the firewall, but it just, incidentally,

Adam Robbie (10:49.890): And then we want to run, like finish this project quickly. We need to operate it so we can, you know, don't our research. So we don't want to like, okay, the firewall rules, maybe something is challenging. Just put any, any, any, so we can get the traffic and get things done to move on. Right. So these challenges, like between the time and the resources and like all that pressure. It's so understandable why there's a lots of misconfigurations.

Adam Robbie (11:16.960): in the OT environment. So understandable why someone would choose not to have a firewall or even to not to enable the right rules in the firewall to get their factory running. It's cost benefit analysis. Is it more beneficial for me to worry about security or just get the production line going so I can make money? So even just like

Aaron Crow (11:27.692): Yep. Yep.

Adam Robbie (11:46.510): the details of putting the cables on the PLC or connecting the I's to the PLC's and how long that takes or labeling. There's so many details that from an IT perspective that someone may not understand that at all. And that also every PLC work completely different.

Aaron Crow (11:57.301): Mm-hmm.

Adam Robbie (12:13.546): And may need different configuration me like even like something like in a hardware like when a PLC to move it from a remote to local Sometime like you may do with this do you configure it on the project file or you need to go on the hardware and manually Push the button to from remote to manual. It's a small detail, but it makes different it's small detail, but someone like for like

Adam Robbie (12:41.784): Well, would remote access to this PLC and be more challenging than other PLC? Well, it's the mechanism is different. Well, that tech surface will be different and the operation also will be different. and giving also like during COVID and a loss challenge that happened during COVID, like when people needed to work remote, that also opened the loss of solution. Okay. We need to make sure like,

Adam Robbie (13:8.332): our engineers or technicians can access remote to troubleshoot and fix and all of that. And that's just open the remote access like a huge door. So I'm sure you know all of that. I am preaching in

Aaron Crow (13:17.122): Sure. Yeah.

Aaron Crow (13:24.010): No, but it's good to go over, As, you know, as we talked before this, you know, there's people that listen to this podcast. Some of them are getting into OT. Maybe they've recently become responsible for OT and they come from an IT background. You've got leaders that are, you know, they're responsible for, OT, maybe recently, or even if they've had it, but they didn't realize how big of an issue it is, you know, and, and little things like, you know, again, going back to basic stuff, you know,

Aaron Crow (13:52.514): a lot of the problems that we would see when we're rolling these things out is as I'm transitioning from a, you know, a stateful firewall to a, to a next gen is the rules were so vast. They weren't quite any, any, but almost like they would have, you know, 44,000 all the way to 65,000. She's like, do you, do you really need all those? Well, it changes. Okay. But when you get to an application aware, then you don't have to do that. I don't have to have that stateful rule in there. So we would have to put in the old rule.

Aaron Crow (14:22.050): And then the new rule, the application where right above it to with the goal of it should never get to that original rule, which is the old way so that we can decommission it. But we had to do that. And you hit it on it before we had to prove that the firewall wasn't the thing that was going to break it. And we made the firewall be the hub and spoke. So it was the router and all traffic went through that firewall, which again, I'm dating myself. It's more commonplace now, but you know, 10, 12 years ago.

Aaron Crow (14:52.086): The last thing they wanted to do in an OT world was depend on technology they didn't understand and the vendor didn't install. Like the vendor being the control vendor. They put all of the weight in that control vendor basket and anything you did outside of the configuration that the control vendor put in from factory, they were very hesitant and leery on and you had to prove that it wasn't going to break something before they would let you install anything and change anything. It's an uphill battle sometimes.

Adam Robbie (15:1.665): Mm-hmm.

Adam Robbie (15:22.722): Yep. I totally understand that. I also, I think my career gave me a really good benefit that I experienced both like life of OT and IT. believe it or not, I started it as an OT, became a senior, went back to grad school, and then I started IT from like, from an entry level.

Adam Robbie (15:51.106): So I have two cycle of career, like 10 years and another 10 years. So, and it was a challenge, like to be honest, because when I started IT from, from the beginning as like inter-level, I started as like a sys admin in the high school or help desk high school. And just, I was a manager in OT and now I'm changing batteries for kids. That's the, that is not working.

Aaron Crow (16:9.142): Mm-hmm. Sure.

Adam Robbie (16:20.950): or the projector in the class is not working, I need just to restart it, or a student laptop is not working, just restart it. It makes me humble to say, OK, I need to start from beginning. But I get exposed to the server. I think this was the first time I was exposed to Active Directory. I've been in OT for 10 years. We don't need

Aaron Crow (16:28.672): Right.

Adam Robbie (16:48.890): We don't know what Active Directory is. It's a thing. And also, even I remember from an OT life, when we even have servers, I don't remember we worked with lots of Microsoft servers back then. There was lots of Linux or even a proprietary operating system, which used command lines to configure it.

Aaron Crow (17:10.338): Mm-hmm.

Adam Robbie (17:18.434): I would go with our IT guys to one day configuring the server and it's like, is boring. just like, I don't want to see that. Then I go there, I see operating like active servers and active directory and see like how creating policies and rules and security policies and how creating like a user names and domain controllers and just get involved in the entire environment. It just.

Adam Robbie (17:46.954): opened my mind like, wow, now I understand like, it is messy. Like there's tons of stuff happening in it and way faster. OT is like, we are very structured. we like things like, you know, we have to be very precise. Like you, you, you work on like, in a microsecond, like to avoid delay and like how long the, like the signals comes in, how like the counter and send it back. So

Aaron Crow (18:15.276): Mm-hmm.

Adam Robbie (18:16.418): Like we get very, very structured and detailed and organized and you you go to it. They just, it's just find a sport open, just put the cable and yeah.

Aaron Crow (18:30.998): That's right. Well, there's, there's something to that. And, know, I think my, my career is very similar to that. And I had, you know, some, some lateral moves. had a few that I took steps backwards. and, and I did it, you know, I, I was, I grew up in the, you know, MCSE and, know, going after CCNA and all the different certifications. And I would go.

Adam Robbie (18:52.196): Mm-hmm.

Aaron Crow (18:54.998): get a job that I had zero experience in and I would get paid less and it was a title bump and down in the wrong direction. But I would do it because I wanted that experience. And it's kind of like, you know, I get this question all the time of how do I break into cybersecurity? And it's not rocket science, but it's not easy either. Like sometimes you have to take a job where you were a manager and now you're the janitor or you're the entry level person.

Adam Robbie (19:5.626): Mm-hmm.

Aaron Crow (19:24.450): But the good thing is, is like when I did that, I remember I did that and right after the internet burst of the dot com era back in like 2000, right? And I got laid off like so many people did during that time. I was like a technical architect and I was making big money and then all of a sudden you couldn't get a job. And I found a job working at AT &T wireless, I think it was at the time, as a systems administrator.

Aaron Crow (19:53.184): And I think I cut my salary in half. Like I went from like a six figure salary to not a six figure salary. I could barely pay my bills, but I was only in that role because I got the job. And then I was in that role for like six months and I was, I did a great job. I didn't complain. was, you know, knocking it out and doing everything they asked me to do. And very quickly I got moved to the active directory and domain team and I got.

Aaron Crow (20:20.490): a big promotion and all this kind of stuff. But I had to, you know, eat crow and humble myself to say, I know this is going to go somewhere. And I learned a lot during that time and it jumped me to the next place, but I had to go through that. And there were so many during that time that that wouldn't take that job because well, I was a technical architect. I'm not going to take a step down. and I'm not saying my decision was right, but for me was the right decision and it helped me get experience. So now in an OT role, I've been the ad, the act.

Aaron Crow (20:50.324): Active Directory admin on an enterprise level. But I've also been the OT guy at a power plant, right? And I've had to have both those experiences so I can put myself in both of those avatars perspective and understand that, which helps me to be able to communicate both of their concerns and kind of be that mediator. Because that's really what we need a lot of the times in the OT IT is someone that can translate even more. You're using the same words, but they don't mean the same things in both in the different environments.

Adam Robbie (21:21.147): they don't mean the same thing at all. It's like the same word like an automation engineer. I try to make an industrial automation engineer so people know. There's so many kind of automation engineer. I'm talking about the guy who do the code for the BLC. That's one. And I need to remind myself when I go to meetings, like, okay, I need

Adam Robbie (21:48.772): to understand who is in the room and what language do I need to bring up and how can I interpret that easily to everyone. And also the more that I know, the more I know that I don't know. There is tons of knowledge. And that makes me realize, okay, this is something we cannot do it alone. We need to work together to

Adam Robbie (22:18.286): be able to accomplish any mission or any vision that we have. But the good thing is we did it. I hope that I don't need to go through this again.

Aaron Crow (22:34.323): Me too. Me too.

Adam Robbie (22:37.872): It's like once or maybe twice a lifetime thing, but there is a moment where, okay, I learned as much as I can. I know I need to lean on other people that learn something else that we work together.

Aaron Crow (22:53.814): Yeah. Well, hitting on that, that it's, ever changing. And that's something to, you know, again, I'm dating myself, but you know, when I, when I was deploying all these firewalls and in an OT space and I've done it since, but I wasn't the hands on, but that started back in 2012. mean, that was a long time ago. Technologies, even on the Palos, like they've, they've leapfrog they've, they're so much more capable than this was the PA.

Aaron Crow (23:21.686): God, two 20s, I think, way back in the day. they were, you you'd hit commit on them and it would take, you could go get a coffee and come back and they'd be done running. They were great. But anytime you made to change, you're like, man, I forgot to make a change. Well, come back in 15 minutes. It'll be ready.

Adam Robbie (23:40.930): Yep, yep. I have some experience, of just not the firewall, lots of devices that just takes long time to reboot or some. And it's impressive how technology evolved very quickly and convergence between technology is like a real issue now that.

Adam Robbie (24:8.142): not just it increases on its own, but it converges with others as well. that's the, I think that's one of the evolving challenges that we're going to see.

Aaron Crow (24:22.806): Yeah. Yeah. So, so what are some of the things that you, you guys are, are, really fighting for and, know, kind of building new capabilities. And like I said, obviously OT is, is, is different, but similar, the skill sets and, and some of the, the, the, the use cases will be similar, but again, the response may be different. What are some of the things that you guys are focusing on really solving in the OT space that is unique and that you guys are excited about?

Adam Robbie (24:50.810): So there are a couple of things that my team is working on from Threat Research. One is a new OT white paper specific. And it was a very interesting journey because there's millions of firewalls out there for Palo, right? And a lot in, they could be in an OT or non-OT. And...

Adam Robbie (25:20.678): we only have access to limited information. So we don't have a full visibility to say, this firewall in the OT or it's not in the OT, like how can we do that? So we had to be very creative on how we can identify firewalls that's involved in the OT network. And one of the first things we used is, I think you may...

Adam Robbie (25:49.218): reference to it is like app ID, which is the capability of analyzing the packets on the application layer, not just the port number or the IP address. We have a list for application that categorize or tagged as an OT. And said, OK, let's utilize that. If any firewall trigger traffic that tagged OT, most likely that

Aaron Crow (25:51.372): Mm-hmm.

Adam Robbie (26:18.926): that firewall either touching or like on the edge of the network of OT or inside the OT. Either or, I don't have enough visibility to distinguish, but I think that's more than enough to say I need data from these firewalls. So we started from there and then we start to collect threat telemetries. And we looked at the signature fired on those firewalls.

Adam Robbie (26:48.882): And usually when a signature is fired, there's a high confidence there's a malicious traffic that fired these signatures. And we took those signatures and we did our analysis. Basically, two major outcomes of this analysis was interesting. The one, we mapped these signatures to MITRE ICS.

Aaron Crow (26:57.634): Mm-hmm.

Aaron Crow (27:14.476): Mm-hmm.

Adam Robbie (27:15.482): And we were able to identify what are the top TTPs are impacting this environment. And we're not so surprised, but remote access was number one on the list. Exactly. So that was the rec really interesting. We dive deep more on the white paper about the other TTPs and also like what other, like what a threat actor have used this technique in the past.

Aaron Crow (27:26.274): Sure. Yeah.

Adam Robbie (27:45.146): as well, right? So it's just to give you a more holistic view of that threat. Then the second piece of information, which is if anybody listening, I want your attention to that particular part specifically. So we talk a lot about CVEs. We talk a lot about the vulnerabilities. We talk a lot about, oh, there's a new CVE that's critical.

Adam Robbie (28:12.218): Or we did a vulnerability scan and we find old CVEs. I am not talking about any of that. Okay. I am talking about CVEs that has been exploited. That's like the worst of the worst, right? Like I'm telling you, this CVE, we've seen it and it's been exploited. That's it. So we looked at the ages of these CVEs that has been exploited.

Adam Robbie (28:42.106): Can you guess the age of more than 60 % of these CVEs?

Aaron Crow (28:47.850): I would guess 10 plus years.

Adam Robbie (28:50.326): It's from five to 10 years. That's correct. Yeah. And that was just like mind blowing that we are not talking about CVEs in the systems. We are not talking about there's a CVE and critical, but the chances of being exploited is low. Right now I'm telling you these CVEs are being exploited. So if you have it, you must.

Adam Robbie (29:19.256): Do something about it. I'm not going to say back. There's something you have. need to do something about it, right? Because yeah, so so this is this is the piece of information like one like when you get access to the white paper, I strongly recommend go to some of these CV is just use them. See if this if any of these CDs are in your environment and remodeled immediately. About.

Aaron Crow (29:20.534): You gotta remediate in some way. Yeah.

Aaron Crow (29:25.612): Sure. Yeah.

Adam Robbie (29:49.306): So those were kind of like the top. and then the third piece, which was also interesting. So usually any traffic have source and destination, like where it's coming from, where it's going to. We looked at the traffic from an internal network to an internal network. We are not looking for inbound or outbound. So it's from internal to internal, meaning it's a lateral movement most likely.

Adam Robbie (30:18.414): Like it's something like, you know, exploited or something compromised that's going to somewhere else. And we looked at what our number one industry for like internal to internal traffic we have seen and manufacturing industry was number one by a significant ratio. So that's just gives us like a holistic view, a little like if you're in the manufacturing environment or an OT,

Aaron Crow (30:37.922): Sure. Yeah.

Adam Robbie (30:49.208): What's going on? Remote access, old CVEs, lateral movement. Start from there. That's a starting point. Those are kind of like the top three pieces from the IPS threat limit. We also looked at the malware.

Aaron Crow (31:12.480): Yeah. Yeah.

Adam Robbie (31:18.810): So we have a wildfire, any file that transfer over the firewall, we can capture them and analyze and see if they are known malwares or not. One of the also interesting information we have seen that I think over 60 to 70 % of these malware categorized as unknown. So with a high confidence, we know they are malicious. But have we seen them before? Most likely not.

Aaron Crow (31:20.684): Mm-hmm.

Adam Robbie (31:49.050): So what this tells us tells us that a loss of new variants of malware are out there. So our best detection of like, oh, you have an antivirus or firewall, like you turn it on and you're to do your job, most likely to do like 30 % of the job. There are 70 % of effort that need to be done, which is proactive defense, whether this is a SOC or monitoring or using like more advanced

Adam Robbie (32:18.598): solution, so on and so forth. So this is like what we have been seeing so far. Yeah, any questions?

Aaron Crow (32:29.708): So all that is all that is awesome in that. And you and I've talked about this extensively a lot, right? Is, is you, you, it has to be an ecosystem. There is no silver bullet. You have to work with your vendors because to your point, I've got, I've got antivirus. think I'm good. But when you detect, when it's not a detectable, it's not on the blacklist, then your antivirus isn't going to detect it because it's not looking, it's not in the signature. So it doesn't know that that thing is bad.

Adam Robbie (32:33.156): you

Aaron Crow (32:59.168): Whereas you need other products and segmentation, right? We talked about that briefly, but why do I need to segment so that one of those things that gets off in the wild in theory would be isolated to a smaller environment. wouldn't spread to all of my environments, to my other sites and all that type of stuff. Right. So that defense in depth and having, you know, the other piece that you said earlier, and I wanted to circle back to is we're all in the same team.

Aaron Crow (33:26.530): And I hear this a lot of, I'm the IOT guy and I'm the IT guy. He doesn't understand what we're doing. Well, you need to make them understand, right? We need to be on the same team. We're wearing the same Jersey. We're trying to protect the same assets. We both, we all get our paycheck from the same company or you're my customer and we have the same end goal of protecting your environment. We need to stop being adversaries and start being allies and utilizing the tools. And that's what I did in my space is because I had a small team.

Aaron Crow (33:56.384): Yes, we manage the firewalls, but I would reach out to my IT team that had an entire firewall team and they had hundreds of people and all this stuff. And I would use them, hey, these are our firewall. Would you mind doing a review? Am I missing something or my rules too broad? Or, you know, do I have things turned on? And I would do an internal audit almost with, still held the control, but I would get somebody else with a different set of eyes and more, potentially more.

Aaron Crow (34:26.190): usually more experience than I or my team had. And I was okay with that. I was okay with them giving me suggestions. It was still my responsibility and it was my choice to implement what I wanted. But I was stupid to not reach out to and lean on expertise that was in-house on my team. It didn't cost me. I didn't have to go hire a consultancy or anything else. I could just start with my internal teams and get their feedback.

Adam Robbie (34:52.654): Yep, that's true indeed.

Aaron Crow (34:55.797): Yeah. Well, awesome. So dude, I know we did some cool stuff with MTU. I don't know if you want to talk about that a little bit and what that is and kind of, you kind of briefly talked about the, the lab that you guys have and, know, kind of the purpose behind building it. But I think you guys have a, kind of a, a multi-use, for your labs and, kind of what you're doing, both internal and, know, kind of customer facing as well.

Adam Robbie (35:22.562): Yes, so this is an invitation for anyone that's going to S4. I will have a talk about risk assessment, basically, more about how to apply game theory in OT, which in summary, we all get like all this cool solution. You get all this security solutions.

Adam Robbie (35:52.046): The question is how strategically you can apply them. How can you avoid overspending or underspending for the threat that you are facing? And so I will go more in deep in this presentation from a little bit mathematical, but more simplified. How can you make a strategic decision complying the threat that you are facing and analyze it or map it to MITRE's

Adam Robbie (36:22.328): MITRE framework and also use the game theory model to identify your solution. The cool part of that, if mass is not your cup of tea and you want to experience this in a really like a game format, like as a tabletop exercise, we will have also surprise there. So you come over, like join the talk, and then also you can sign up for this.

Adam Robbie (36:52.338): like tabletop exercise format that will help you to understand these concepts. And it's a really, really interesting, and I really was curious to see people from different backgrounds, professional backgrounds, how they run this tabletop kind of exercise. it's a very fascinating thing when you see like,

Aaron Crow (37:14.945): Mm-hmm.

Adam Robbie (37:18.266): how IT personnel responds versus an OT versus someone knows both of them. So this is an invitation for anyone to come join us and I'm looking forward to see you there as well.

Aaron Crow (37:29.814): Yeah, absolutely. I'm excited to see it. You know, it's been a long time, but you you hit on something right there too, right? your perspective and your experiences go into how you answer things or how you view things. So when I've done tabletops, hundreds of them, if not more, with different audiences from, you know, plant people in manufacturing or power generation or you name it.

Aaron Crow (37:57.856): All the way up to IT people. I've done them individually and I've done them with, all the parties in the room. And I've done them with people that are outside of cyber and are wanting to get into. And it's amazing the perspectives. And sometimes you're like, well you have no experience. What do you know? Sometimes the most.

Aaron Crow (38:16.844): just off the wall, amazing ideas or thoughts come from people that have zero experience because they look at a problem differently. They're not jaded by previous experiences or past failures or all of the things. Well, we've done it this way for 40 years. Any of that stuff, sometimes having those interns in the room or whatever they are to ask the dumb question that people like you and I may not ask, because we're afraid we may look stupid.

Aaron Crow (38:45.536): Those questions sometimes make you be like, you know, I didn't like, just assumed we couldn't do that. Like, and you look, you see the head looking around, like I've never, I never voiced that out loud and you just did. And sometimes those are more powerful than the smartest people that have 30 years experience because they're, they have this bubble or the, these, these guard rails that they inherently bring to the conversation.

Adam Robbie (38:59.097): Yeah.

Aaron Crow (39:11.434): Especially when their leadership or when something, their mentor or their boss or whatever, they're afraid to answer or look stupid. We've all been that kid in the back of the room that doesn't want to raise their hand, even though I don't get it. Like, I don't want to raise my hand because people will look at me and then have, everybody's going to know I'm stupid. At least that's how I used to feel.

Adam Robbie (39:29.924): Well, I can relate to this as well.

Aaron Crow (39:35.276): So it's just fun to see the differences in there really is no stupid question, right? And getting that in front of all of these people, there's value add. And you talked about, you know, the white paper and the fact that it was 10 years, it didn't surprise me at all that it was five to 10 years of the age of those because of the age of the equipment, the environments, and the way that things are rolled out and implemented and as slow as they are to change in these OT spaces, they didn't surprise me at all.

Aaron Crow (40:4.386): It also didn't surprise me about secure mode access because, or the lack thereof, because that is one of the common problems that you have. A lot of these environments are in the middle of nowhere where I don't have my staff. So I have to make it where an engineer can get to it. So what do I do? A lot of the times I just SSL VPN into the environment and put my network on it, right? It makes it easy, but it also brings all of the issues. just bypassed all of my controls, not all of them, but many of my controls.

Aaron Crow (40:31.030): by segmenting these environments, putting in these great firewalls with all these rules, and then even worse in an OT environment, which I was gonna bring up before, but now that we're talking about it, you also have these weird architectures where I've got a machine that's dual and triple-homed, and they've got a 3G card that goes straight to the vendor and bypasses all of your security things, and you don't even know it's there. And it's impossible to understand that, and even down to an asset level.

Aaron Crow (41:0.034): I don't even know what the equipment is in the corner and it's not on this asset inventory and nobody in the room really understands it. And they say, oh, well that's vendor X's and what does it do? We're not really sure, but it has to be there. Does it, does it bypass it? Does it go the internet? Does it go through our stuff? Like many times I'm walking in these environments and I'll get an asset. just doing a, an assessment, right? And I'm, and I'm walking down using the asset inventory they give me. And many times like, there's, there's 15 devices and I'm walking around and there's 300.

Aaron Crow (41:29.194): Where are the rest of them? They're not on here. Do you know about them?

Adam Robbie (41:33.880): yeah, I have seen those kinds of stuff as well. Like, yeah, it's not a surprise, but I was like, wow, well, we need to deal with the reality. And this is what we're dealing with and how can we start from there and improve. And I think one of the good things is like someone like you or the protected all it's, we, we, start these conversations. We talk.

Adam Robbie (41:59.098): how to bring it up to the surface so people can listen and hear. Because also one of the challenges that some, an expert may have the right information, but they may have a challenge to educate up. And hopefully like, you know, now like people start to listen and when you go to your managers or your C-suite, they can align quickly and they get, you know, react or respond to the requests faster.

Adam Robbie (42:28.794): I think sometimes it's just some people may have the right request and just get stuck on the process or the leadership or the C level like, don't do that. Why do we do that? Or specifically if there's a CFO there, don't hate me, but you guys are usually a big walker. We love you, but you sometimes make our life.

Aaron Crow (42:48.672): The No Man.

Adam Robbie (42:56.696): which is reasonable, I understand that. So I hope that this knowledge and information is helping to facilitate this kind of conversation to make things move faster and in the right direction.

Aaron Crow (43:10.358): Yeah. I mean, that's, that's one of, as you know, with me, that that's one of the reasons why I have this podcast is I want information to get out there. I want you to be able to take this to your boss and say, Hey, just listen to this. Like, this is a problem we're having right now. We're not the only ones, right? one, one of the things that I, I wanted to ask you, especially because Palo is so big and y'all are global, like what, what problems do you see or how are people, I guess it's kind of a multifaceted question.

Aaron Crow (43:38.054): What differences are you seeing in geographies and different countries and whether it be the UK or Asia or wherever, Australia handles things differently. Europe has regulations that are different than NIS2 and US has NARC-CIP and CMMC and all these different regulations. But in addition to that, people in other countries handle it obviously China. The government controls the things that they do.

Aaron Crow (44:7.062): Like there's just all of these different problems and solutions and each, you know, geographically area, country, et cetera. You have to handle them. And some companies you look at FedEx is of the world or whatever, and they're in all of those places. So they have to have a different policy and capability, maybe a firewall rule difference in China as they do in Mexico.

Adam Robbie (44:31.190): Mm-hmm. Well, this is really good question because we did came across a geographical analysis in our white paper and that was a debate actually should we include it should we not because There are some data biases that sometime you run into it and we have to normalize the data to get Keep it integrity as much as we can because maybe we have a thousand firewall in one country and we have ten firewalls in another country that's like you cannot compare this but

Adam Robbie (45:1.006): We can normalize the data and get average payer one firewall, and then we can try to compare the averages to get that picture. The other thing also beside the biases, it doesn't mean we don't see much malicious traffic from one country, that they are more secure. It most likely could be because they don't have enough.

Adam Robbie (45:29.484): information or they are not enabling all like the right security measurement or they bypass it right so geographical as much as it's interesting piece of information to bring up but it's it's it's really hard to make a story like or getting precise information what's really that that means right that's that's from just like a very very high level

Adam Robbie (45:58.560): If we want to go like, what are the top threats impacting per country? We did not go that granular. I think like global infrastructures, like it's, mostly face this some, some similar issues. The, the compliance you mentioned just add a different dynamic. I think the threats.

Adam Robbie (46:28.844): are the same, even in different regions. I think different regions, the only thing I would say was interesting, or I think it makes sense to see different, the volume. Some country that has more manufacturing environment makes sense the volume of attack will be more over there. So that's something I will pay attention to it in specific.

Adam Robbie (46:57.556): geographical location, A country that's known for mining. That industry will be the top one we see there because they are known very well. I think like in Australia, it's very known for mining. So that's what we see a lot for manufacturing like US and Europe, something like that. Yeah, we maybe we don't get a lot from like, like from China just

Adam Robbie (47:27.598): because different regulations or different, but those are like from a geographical, we give a high level, but because of the data biases, we try to, know, I don't, yeah, like how true the story or how much we confident, it's a tricky.

Aaron Crow (47:49.952): Yeah, it just really gets to the complexity of these larger entities. And again, how I do things and it's funny because I can go to a US company that has, again, use my example or my experience, a power plant. I can go to a power plant or a company that has power plants and they've got one in this city and one in that city in the same state. And they may have the same vendors, but I go in there and the architecture is different.

Aaron Crow (48:18.454): the implementation's different, how it's done is different, like where the firewall sits, the rules are different, like everything about it is different. Like, again, I'm dating myself, but when we rolled out Palos, we weren't even using Panorama. And that was because that was a conscious decision because of NERC SIP and the way that we read the NERC SIP as if I had Panorama that can control assets that are providing access control to these environments. And that would have been a NERC SIP asset. We didn't want that. So

Aaron Crow (48:45.888): we were individually controlling all of these 200 firewalls manually. So when I wanted to make a change, I had to go to all 200 of them and implement those changes locally on those firewalls. Now we had HA and backups and all that kind of stuff, but I didn't have the benefit at that time, or again, at least right or wrong, our compliance attorneys thought that that was our response to compliance, so we did it differently. Segmentation, IT, you're gonna...

Aaron Crow (49:14.870): connect everything and you're going to block people off. Whereas an OT, you're going to segment and put as many barricades and, and, know, layers of protection as you can. So I'm going to hyper segment zero trust, all the, all the newer capabilities, but you have to understand the environments that you're going into before you roll out these new technologies. So it just, adds complexities, but, it's just awesome to see, you know, big companies like Palo and others that haven't forgotten about OT. Cause you know, one of the frustrations in

Aaron Crow (49:44.448): You know, again, working at a power company is the power plant is the thing that brings the revenue. But, you know, if I looked at my team, I had six people. We supported all the power plants, every bit of technology from the firewall to the control system. My team was responsible for, had six people that supported 45 power plants. IT had hundreds of people.

Aaron Crow (50:7.724): They had an entire dedicated firewall team. had a virtualization team. They had an application team. They had a server team. They had a backup team, a networking team, cloud team, like all of these people. And I'm not saying they shouldn't have. I'm just saying that my team is only six because again, this was a while ago and it was a struggle to get budget and team because it was new. It was different, right? And used to, they leaned on the control vendors to do all of that work. And it just really showcased the

Aaron Crow (50:36.938): the need and the tech stack. Like again, my team of six supported 45 sites and hundreds of firewalls because we had firewalls at multiple levels and the virtualization and the servers and the applications and the networking gear and the switches and the, the, the, logging and monitoring and antivirus and secure mode access and all of the things where the team of six. So there was no way we could be great at any of that stuff. We were just, you know, whack a mole.

Adam Robbie (51:3.128): Yeah. Yep. Yep. I totally see that. And when you add compliance on top of all of this, it's just like you're losing 50 % of your resource adjust for comp. That's it's. Yeah. I understand. I feel your pain.

Aaron Crow (51:14.336): Absolutely. Yeah.

Aaron Crow (51:20.054): Yeah. Well, in that, that's why it was so important for me to leverage our internal teams. And, and I knew my people were good at, I mean, I was, I was good at Palo. Like I could deploy them and all that kind of stuff. But was I the same level as the team that dedicated and that's all they did? Of course not. Like they were way better than me. Like I actually went to, and got a job offer to go work at Palo for a while, way back in the day before I went to work someplace else.

Aaron Crow (51:47.394): Um, and, and, know, it was, I went in on the interview and I thought I was a great Palo person and they were just asking me questions and I was like, I have no idea. I've never done that with a fellow, but I was using it in a very specific use case. So I didn't do some of the things that they were asking me about because that wasn't applicable in how I use the firewalls in our space. Even though I had years of experience and I deployed hundreds of them and support them in production, but I just didn't do the, some of the things that, that you guys were asking for and rightfully so. Cause most of your customers do use them that

Aaron Crow (52:17.026): Bye.

Adam Robbie (52:17.860): Yep. it's, it's, yeah, it's, it's, it's a ongoing challenges and the more we see, the more we learn. And I remember one of the things that was interesting. I saw reviewing some one network architecture or like IP planning. And then I saw a public IP address assigned to a local device. And it's like,

Adam Robbie (52:47.502): that is doing here. Originally, like, it took me off guard and it took me a while to analyze what's going on here, like why we have this public IP address, what's doing blah, blah, blah. And after a long investigation, we found that it was a misconfiguration. Someone just know they needed IP, they just put random IP so they get things done. And it's like, okay, that's good to know, but...

Adam Robbie (53:17.209): We need to fix that.

Aaron Crow (53:18.846): And unfortunately I've seen that in many places where they're just like, well, our stuff doesn't touch the internet. So I can use whatever IP address I want. So they take routable IPs on the public space and put them into the private space. But then when you see them now, when we see them in the firewall space and we start seeing these public IPs showing up in our OT space, you're like, why is this here? Is this trying to get out? Like what is going on? Sometimes it's just stupidity or not stupidity, but ignorance.

Aaron Crow (53:48.352): and they didn't realize and it was a, shouldn't have chosen those, but they didn't know that they shouldn't choose them. And others, it's just like they know, but they did it anyways.

Adam Robbie (53:53.358): Mm.

Adam Robbie (53:56.878): Well, I was one of those people like in early in my career. I wasn't, yeah, I was in a position where I was just like fresh graduate and my manager went out for vacation and it's like, just do your best. And the commissioning team were going down to the site to commission the device and they said, we need an IP address. I was like, here's one, maybe it worked, maybe it's not, I don't know.

Aaron Crow (54:0.290): Sure, yeah, I was too.

Aaron Crow (54:22.944): Right.

Aaron Crow (54:28.394): And especially in OT, many times those band-aids get put in. And the reason they're still there is because it was put in 10 years ago and it'd be so hard to change it, to rip that out and re-IP it and rename it and all of the things, interdependencies that are tied to it would be really difficult. So sometimes it's easier to just leave it in place and document it of why it's that way. And eventually you'll upgrade it or roll it out, but it...

Adam Robbie (54:36.366): Mm-hmm.

Aaron Crow (54:56.458): you can't just go out and that's another, you know, vast difference in my experience in OT is you're not just gonna go rip and replace. It's too complex. There's too many dependencies and problems that that can bring. So many times I may just document it, you know, I've got XP, I've got Windows XP running in a critical system in my environment. I see that all the time. You would never allow that in an IT world, but in OT it's extremely common or running IP addresses that are publicly routed in a private space. It's not ideal. I don't recommend it.

Adam Robbie (55:15.332): Mm-hmm.

Aaron Crow (55:26.198): But if it's there and it's working, there's other ways to mitigate those problems other than just re-IPing. It's not the popular answer, but it is reality.

Adam Robbie (55:33.818): Yeah.

Adam Robbie (55:39.170): Yeah, yeah, it does require time to fix and change and just the cascaded issues of trying to this band-aid off.

Aaron Crow (55:47.958): Yeah, absolutely. All right. So I asked this question to everyone. I've given you a warning on it, so it shouldn't be too much of a surprise. But over the next five to 10 years, what's one thing that you see that's exciting coming up over the horizon and maybe one thing that's concerning that you really think that we need to we is the collective. We need to really do something about to make sure it doesn't impact us in a negative way.

Adam Robbie (56:14.522): So one thing I think like exciting, which is I was predicting, and I think AI came and took all the trend for now, but I think after that, augmented reality. I have a strong belief that virtual reality and augmented reality will grow exponentially, given

Adam Robbie (56:40.728): the lots of challenges that we see in the OT environment for in terms of personnel and expert experts having this easy way to operate a factory that's just like amazing and cost efficient and helpful for troubleshooting and so on and so forth. Which actually just heads up like I have done this years ago when I predicted this is coming, I was like, well,

Adam Robbie (57:11.286): I need to start to use them now because I need even physically to be trained. Like my eye needed to build a seminar to use that. It's like when like BCs was generated like 10, like 15 years ago, you will see someone who have not grew up with computers, they're just struggling using BCs or workshops, right? So I didn't want to be like struggling when

Adam Robbie (57:39.288): like all the virtual reality and augmented reality, come out and become more prominent out there. So that's something, I know it's not security per se, if that's what you were expecting, but it's technology that I will keep eyes on. And the other thing that's concerning is the conversion of technologies.

Aaron Crow (57:54.358): now. Yeah.

Adam Robbie (58:8.238): We are not just talking about IT and OT. We are talking about IoT. We are talking about AI. We are talking about so many technologies that gets converged and just increase the tech surface of the impacts that can happen in our OT environment. So that's a real challenge, specifically something for IoT. That's something I have done research on a couple of years ago, and I had discussion around it with other people.

Adam Robbie (58:38.106): But it wasn't reality until like a few months ago, with the Ion controller came out and they show how they use the MQTT to do a command control and so on and so forth, right? So it is, yeah, it's like that. The conversion is one of the things that's a nightmare that it's a really, really important. You cannot stop it, but also open the challenges.

Aaron Crow (59:4.288): Well, and I think that's the point is, and the key is it's coming. So you can't just, and unfortunately I've seen this too much in OT is, and just, you know, controls and operations and all that kind of stuff is, well, we're not going to do that here. We need to start planning for it. How would you do it? Like we need to start thinking about it, getting our minds around it. So to your point, we're not 10 years down the road and now it's getting crammed down my throat and I don't know how to do it in a safe, good way. It's going to come.

Aaron Crow (59:34.034): And I used augmented reality, you know, to roll out all that stuff at power plants across 800 sites. And, you know, it was really good because I could have one really smart person centrally located and they didn't have to go to every single implementation because they could sit in a war room and see all the things going on because somebody was wearing glasses and they could see, no, it doesn't go there. It goes over here or, I know what that thing is. You plug it into this.

Adam Robbie (59:48.942): Mm-hmm.

Aaron Crow (60:2.086): And they could get that expertise by sitting in a war room, almost like, you know, the control room. and they're controlling and seeing all that data. I'm kind of like the engineers and, you know, NASA, you know, sending it up to Apollo 13 to help them solve the problem. I don't need all of those engineers on the spaceship. I just need them to be able to understand it enough to be able to help them solve the problem. And augmented reality is way better than using words. You know, we've all done the telephone game where I whisper in your ear and you whisper in the next person's ear. When you see it on.

Aaron Crow (60:31.404): camera at high definition in real time, makes it a lot easier to describe and, and, you know, troubleshoot.

Adam Robbie (60:41.038): Yeah, even like, like Mark, like change this one. Like you don't even have to say like this, cable, the red cable to from bottom like you pointed. And it's too late. pulled it out.

Aaron Crow (60:52.214): Right. Nope, nope, not that one, not that one.

Aaron Crow (60:59.198): Exactly. Exactly. Well, I mean, I've done that in virtualization again, way back in my career, not to dive too far down that, but you know, you think you're on server a and your own server B. So you reboot it. You're like, why did you re what happened? I don't know. I just reboot the server. Well, did you reboot the right server? I don't know. I'm on the same KVM. crap. I reboot the wrong server. It's, it happens to all of us. Right? So, so having that ability to, to have somebody pure

Adam Robbie (61:18.778): the app.

Aaron Crow (61:25.356): peer check you watch over your shoulder. Am I on the right thing? Three person communication. Like a lot of these things that we use in an O.T. space, having tools like, you know, augmented reality would make those things so much vastly better. And my hiring's better. I can get, you know, resource allocation. I can hire really good people that and they don't necessarily have to live in the small town where the plant or the facility is. They could be in a more geographical location and I still get their expertise on site without having to put them on a plane or car.

Aaron Crow (61:53.526): or whatever to get them there. that's, I could see that being a huge impact in industry 4.0.

Adam Robbie (62:0.366): Yep, I totally agree.

Aaron Crow (62:1.324): So what's the call to action for y'all? know you already mentioned S4 and your speaking thing there, anything else that you want people to know. You also mentioned the white paper and we'll definitely put that in the show notes as well.

Adam Robbie (62:13.626): Yeah, like the white paper, read the CVEs in the white paper because those are exploited ones and come see us at S4 for the talk. And also we will have the Cypher wall as well as S4 if you haven't seen it. It's also a good time to come and see it. So come play with the tabletop exercise for MTU and I'm looking forward to see you at S4 and I'm very happy that I was able to join you today.

Aaron Crow (62:43.968): Yeah, man. Thank you. I enjoyed it. It's, been a long time. We've had a lot of these conversations offline, so I think it's beneficial to have these conversations so others can hear it. and it's just getting different perspectives and, and, seeing, you know, asking questions differently or seeing things a little differently. And you know, who knows who this could help, you know, to, to level that up to their boss or, think about a problem differently.

Aaron Crow (63:6.286): Definitely don't hesitate to reach out to myself. I know Adam is very active on on LinkedIn. Reach out to him and the Palo team. They're a great team to work with and they have some awesome, awesome stuff. So definitely check that out. So excited to see you in at S4 in Tampa. Not too long from now, actually until then, sir. Have a good day and glad you glad you were able to spend time with me today. I appreciate it.

Adam Robbie (63:29.402): Thank you, Aaron. Same to you.

Transcript lightly edited for readability.