Regulation Is the Budget Unlock OT Has Been Waiting For: NIS2, the CRA, and Getting the Basics Right with Tobias Nitzsche

Cold Open: Back From DEF CON

Aaron Crow (0:1.302) Hey, everybody. Thank you for joining me on another episode of the Protect It All podcast. I am just off of coming back from DEF CON in Vegas, where I'm actually glad that I'm in Texas, where it's a lot cooler, believe it or not, than it was in Vegas. but Tobias, I I I've got my friend here, Tobias, from across the world. so why don't you introduce yourself, tell us who you are, background, all that kind of fun stuff that share with the audience.

Tobias Nitzsche (ABB) (0:31.674) Yeah, sure. hey, thank you, Aaron, for having me. I was not able to make it to Defcon or Black Hat, so but there will be a new Europe version of it, so maybe then. Let's see. But you yeah, you get all of that information now condensed after the in the next weeks I would say. Maybe something from you today I see. yeah, to myself, I'm Tobias Nietzsche. I'm

Aaron Crow (0:51.470) Mm-hmm.

Tobias's Path: Practice Lead to Legislation & Technology

Tobias Nitzsche (ABB) (1:1.894) yeah the head of legislation and technology for cybersecurity at ABB energy industries. And in my former role I was the practice lead, global practice lead for cybersecurity. And therefore now shifting a little bit besides from the before practical part now more into the organizational structural part I would say into the organization and also

Tobias Nitzsche (ABB) (1:31.216) into our customers, let's say different angle, different viewpoints. But the ba the foundation is that I have is is on the technical side. So that's interesting. It's a new challenge for me also because I started the new role beginning of this year in ABB. so yeah stay hands-on let's say attitude still still there that's

OT Is Everywhere (and DEF CON Still Misses It)

Aaron Crow (1:56.973) Yeah. Yeah. I mean, I I just came off what one of the things I I get the privilege to do is is work with the ICS Village advise and support and help them teach classes and hands on things and and and just you know work the booth at at different conferences. So, you know, at coming off of DEF CON and and all you know, the largest hacker or or cybersecurity conference in the world that week with Black Hat and DEF CON back to back. you know, it it's surprising, not surprising.

Aaron Crow (2:27.338) it it's still, I don't know, a little disappointing sometimes that there's so many that still just don't really have a grasp of what OT is and and why it's so different. So, you know, luckily we we have a lot of hands-on examples of things from PLCs and and and control processors and and different things that that folks get to see.

Aaron Crow (2:51.714) But you know, the the world of the ABBs and the Rockwells and and all of that, like it's not something they necessarily see on a daily basis, or at least they don't recognize. Because as we know, OT is everywhere. It's it's in the elevator that you you took up to the second floor. It's it's in the parking garage that opens the you know the the gates. Like it it's everywhere. We just don't, you know, the uninitiated don't necessarily recognize it everywhere we go.

From Server Rooms to Boardrooms: Who Owns OT Security Now

Tobias Nitzsche (ABB) (3:19.310) Yeah, yeah, I d that's true. some sometimes we forget about that we are so deep in the top topic that we take things for granted. then if you talk I mean I I sat together with a lot of I and this is maybe a shift and it it it's a good, let's say, I would say a good in a good way the the IT is taking more accountability for the O T parts and ha there's more I mean

Tobias Nitzsche (ABB) (3:48.861) We're talking about in since years about ITOT convergence, but I see this now with also with a lot of regulations which shift, for example, the responsibility from the server rooms into the boardrooms or from the let's say operator also to the manufacturer. There there's a there are different teams now that will start con yeah talking about the the security of the

Tobias Nitzsche (ABB) (4:17.968) let's say OT space. And this can just be I would say it is there are challenges, but overall it's always good. If you if you have people talking about cybersecurity in O T, if they are from IT or not, that it is always yeah fruitful I would say.

Why 30-Year-Old Gear and Old Protocols Persist

Aaron Crow (4:36.118) Yeah. So so what in OT, obviously you guys are huge and and and we're we're we're still approaching problems and and I and I have this conversation all the time in OT because you know people are coming up to me and they're like, hey, well, why are these protocols insecure? And you know, all of these different questions. And they just don't understand, you know, the the the ways that things are architected and also the fact that

Aaron Crow (5:4.502) You, you know, ABB, you guys have equipment at plants. I mean, I literally was at a plant the other day that had an ABB equipment at it that was probably 30 years old and was running the system, right? You know, and it it it does its job, it's perfect. Why am I going to replace that with a newer version that has some bell, you widget? Not saying that you shouldn't, but all organizations are not able to just rip and replace. It's not like IT where I replace my laptop every three years.

Aaron Crow (5:32.482) You know, your gear is designed to sit in an environment and run for thirty years and continue to do its job.

Tobias Nitzsche (ABB) (5:39.525) Yeah. No, you see, I I started my career in IT and when I started in IT there were also a lot of protocols that were not secured. if not the the most of them. So yeah this th that's the point. You you have equipments from that time still in in in in the in in customer systems or in in the operational area.

Aaron Crow (5:50.542) Yeah.

Aaron Crow (5:53.613) Right.

Tobias Nitzsche (ABB) (6:7.406) Not to say that we couldn't do better today, maybe sure always. but this is what we all and also our our competitors and NABB what we all work on and this is also through standardization. So everybody wants interoperability, everybody wants that things and plans that are running today are still running in ten years and and can be extended, upgraded or interconnected and so on. So there are

Tobias Nitzsche (ABB) (6:36.816) A lot of different reasons I would say why you still see these old protocols, and many of them are because of long term yeah supportability requests, not only from the vendors but also from our customers. So they're the newer protocols, the newer standards, the newer solutions, the newer systems have much more and better security controls in in place.

Legislation as the Modernization Unlock

Tobias Nitzsche (ABB) (7:6.520) That's also why we as a vendor for sure try to educate that don't do the thing that you did maybe the last thirty years and just keep it as it is, don't touch it, as long as it works, keep it as an island or a silo more. These kind of security moves need to be busted because I I had it just j just some days back. I had a yeah, I I talked about

Tobias Nitzsche (ABB) (7:36.583) What we can do today, w why is the push in legislations also let's say a chance for operators to get attention to modernize their systems because of cybersecurity requirements coming through legislations. That this is y y you can do it stepwise, you can do it w as you said, rip and replace, but rip and replace always is yeah, it costs more money than doing

Aaron Crow (8:4.110) Yeah.

Tobias Nitzsche (ABB) (8:5.104) and continuous or yeah, a continuous approach. Yeah, I think that is that is one of the stories that we should bring up more often.

Aaron Crow (8:16.739) Well, and and to your point, like it's not that there aren't, there are definitely secure protocols, and you guys have brand new gear that you can buy and drop in that have all the latest bells and whistles. And, you know, we're not using Telnet anymore. We're not doing a lot of these things. But to your point, like when I change those things, I it's not just a technology change. It's also changing the processes and how I have to train my staff on how to use them and I have to look at the logic and and the actual things.

Aaron Crow (8:46.435) the the things that control those devices are are div th those things are controlling things. I have to change how they interact with all the other systems downstream that they're they're touching, right? You know, how do I read the sensors? How do I, you know, open this gate and this valve and and and it's more than a simple product. And and and again, I I think that's a a thing that IT folks don't always think about if they haven't been in this OT space is it's not just changing out one box. And yes, you can, but to really get the the the impact,

Aaron Crow (9:16.171) Even if I put in a new, you know, A B B device in the middle, I also have to update the protocol, which means all the other downstream things have to be, you know, updated at that same time. And and it's a bigger problem than just replacing a device, right? It's it's people, process and technology.

NIS2 Widened the Net; the CRA Pulls In Manufacturers

Tobias Nitzsche (ABB) (9:32.115) For sure. And for example, when we when when we see in the in in Europe as an example, in the last years the NIST two regulation was a lot of focus because this has ch changed who who needs to do cyber security from a critical infrastructure perspective.

Tobias Nitzsche (ABB) (9:59.739) much much more companies are are now in scope and th it was always on the operator side. You but now with Cyber Resilience Act coming, it is also the manufacturers. So it is the down the supply chain. So yes you will be able to buy secure OT devices systems because that's the only ones that you will need will you that you will be able to buy because you you we are mandate we are mandated to to provide

Aaron Crow (10:25.241) Yeah.

Tobias Nitzsche (ABB) (10:29.699) you with secure products.

Aaron Crow (10:32.919) And and we're seeing we're seeing more and more of that with, you know, obviously I I came from Power Utility. So you've got NERC SIP and Power Utility, right? So obviously that's done really, really well at pushing that envelope and saying thou shalt have secure blah blah blah blah blah. Like and there's lots of different standards from firewalls to segmentation and changing passwords and you know, just basic foundational things. And and I absolutely see regulation being a big factor. NIS2 in the UK.

Aaron Crow (10:59.753) being a being a good standard in that space. and and we're starting to see that, you know, in the US, TSA Pipeline and and and lots of others, right?

What Gets Regulated Are the Basics: You Can't Wing 24-Hour Reporting

Tobias Nitzsche (ABB) (11:8.166) And and to be honest, i if you look into what these regulations do, that's and what they are talking about. Primarily it's what you I have looked in some of your re reviewed some of your talks that you had before, and you're always coming back to the point, yeah, you need to do your foundations right, you you need to do the basics, you start with that. I mean w what gets regulated are fundamentally the basics, right? You d

Tobias Nitzsche (ABB) (11:36.189) Do your risk assessment, do your asset inventory, do s basic cyber hygiene. Yes, you need to be able to detect something so you can be able to report the the things that come up now in the US, which are let's say postponed again to September. Now we will see with the regulation about reporting obligations for critical infrastructure.

Tobias Nitzsche (ABB) (12:0.167) Yeah, okay. You maybe you need to be able to report within twenty-four or seventy-two hours, whatever it is. I mean first you need to be able to to detect something. You need to know what what do I have? What has happened? Does it make any sense? Is it really an incident? And then with that collected information, you need to report within twenty four hours. You can't wing this. So you you need to have your basics in place, right? So I and it's not it's not like it's the it's

Tobias Nitzsche (ABB) (12:29.030) the burden that maybe some reg regulation is just a burden. No, it's it just tells you about your basics in in cybersecurity, I would say. In the most cases. There are there are there we we can debate on on other things if they are really necessary and yes for sure especially in Europe we we we intend to have a little bit too much regulation, but I would agree to that too. But yeah, there are reasons also.

NERC CIP, Fines & Why Power Leads in Maturity

Aaron Crow (12:54.851) Yeah, but like you said, right? If you don't have any regulation, then then, you know, you you unfortunately many times organizations choose choose to do nothing and and they they they don't do the right things. you know, Nurk SIP is not perfect. and and some some say it's it's pushing too much and and and others don't think it it's going far enough. I I can I can argue both sides of that coin. but but I I think it's obvious that power utility as far as critical infrastructure goes

Aaron Crow (13:24.697) Power utility is one of the most mature, if not the most mature, in the United States at least, because of NERC SIP. because they have pushed and forced that envelope and there's a compliance and there's a there's a there's a a negative impact. There are fines and they will fine you. I know a lot of organizations have gotten some really, really, really big fines because of those things. And and you know, organizations don't like that, right? So they want to do the right thing. The board wants to do the right thing. And I and the other side of the coin is.

Aaron Crow (13:54.361) Nobody wants to be in the news from a from a an attack, right? You know, obviously it's going to happen. You're gonna get, you know, it it it's like anything else. It's not if it's when, and and being able to recover and respond is huge. But if you don't have those foundational things, part of those foundational things that I scream about all the time is recovery. Like, yeah, we need to focus on left of bang, trying to stop and monitor for things before they happen. But I also need to assume something bad can happen. How can I recover from it?

Be Prepared to Restore: Exercise the Plan, Not Just Write It

Tobias Nitzsche (ABB) (14:22.106) Abs absolutely. I yeah, yeah, I I just released let's say in an an article on Water Online about it and it was released before the latest news in the US happened, so some some days before. So so b but that's basically what I did. what I did say. You yeah, you need to be prepared. W you don't know when it happens, but the potentials are high that it will happen and then okay, well

Tobias Nitzsche (ABB) (14:49.274) What can I do? I can be prepared to restore. And this is something that you can always do. So you you you can be prepared and test and be sure. And it makes you much more resilient if you have trained, if you are aware where you yeah, don't don't just have a disaster recovery plan, but also exercise it if it's working. your instant response plan, is it there or is it also executable? And these are things you you should validate frequently.

Combine Cyber Tabletops With Operator Simulator Training

Tobias Nitzsche (ABB) (15:18.470) I wouldn't I would say n at least once a year sh somebody should should have g gone through and see if all the contacts in your communications plan are still the right ones. Have you covered this and that scenario? Tabletop exercises are things that you could do. Another good topic. Everybody talks about tabletop exercises. In IT, I I think the IT security teams do that much more frequent than the OT teams. But OT teams also do that, but from a different perspective. They have their

Tobias Nitzsche (ABB) (15:47.397) simulator software. They have let's say areas where they train their OT operators on how to react on problems with their system. This is very well the same thing, but not from a cyber security perspective, but from an operational perspective. Why not combine these teams? Why not combine these exercises, right? That would be perfect. Yeah, for sure the the reason for a problem could be cyber. So how how can I detect it? If it is there, can you com

Tobias Nitzsche (ABB) (16:14.970) Yeah, do together with the team that runs the simulator and how to react on it. That would be also one of the advices I would give. So you have s you have

The Vendor-Install Backup Problem

Aaron Crow (16:23.568) Hundred percent. Yeah. And you know, the the other piece to this and and and you know, as I do assessments on places, I'm always making recommendations on, you know, we go in and we're looking at, hey, your firewall rules and we're looking at, you know, vulnerability and your asset lists and you know, your backup and your instant response plans. the bigger piece that we have is is NOT. they you many times don't have a backup or or their backup, when I say they don't have a backup, it's not that they don't have a backup.

Aaron Crow (16:53.516) It's that they their their their vendor installed their system and that that's the last backup that they have. Like maybe they have a USB stick sitting in some engineer's drawer, but when's the last time they ran that? And also, have they ever actually tested a backup on their OT system?

RPO, RTO & What Recovery Really Takes in a Plant

Tobias Nitzsche (ABB) (17:14.086) Yeah. Yeah, no, absolutely. So yeah I from the past when I did practical system engineering and consultations with customers, I heard a lot of let's say interesting responses. So yeah, for sure we the vendor delivers the initial after SAT, you get your system together with the latest version of the backups.

Tobias Nitzsche (ABB) (17:42.931) Potentially you also get a backend recovery procedure and documentation and some kind of a solution to to do it. But then let's say, when do you do it? When do you need to do it? These are things you need to understand. So does your recovery kind objective fit your requirements? So can I can I restore to a point which is fine for myself for the system?

Tobias Nitzsche (ABB) (18:12.346) And can I restore it in the time that I need to? And that in the time for OT systems, I would say in the most scenarios, for the most plants, it might be okay if you have a complete shutdown really a a really disaster if you can be back online within one, two days, maybe three days is fine. But what does that mean? if you if you start from scratch.

Aaron Crow (18:31.845) Right. Yeah.

Tobias Nitzsche (ABB) (18:38.354) you need spare parts not only it and they should be potentially not not cold spare but you need to have somehow a warm standby or something because after you have restored the system starting up the plant doing it that all t takes time so it's not only a computing should be restored right so

Redundant Controllers Are Not Cyber Resilience

Aaron Crow (18:56.953) Yeah, and and knowing those critical systems and knowing your critical sites and understanding all of those things, right? And and that's where, you know, we we design this all of these, and you said it just a second ago, right? The the all of these OT systems, we design this redundancy into them. Like we expect that a controller is going to fail. So usually we have redundant controllers, especially in critical processes. So if one if if one controller fails, it automatically fails over to its backup.

Aaron Crow (19:23.887) What we don't always plan from is from a cyber perspective of okay, now I just infiltrated and and and both of those controllers are now bad. Right. Do I have another one that's on the shelf? And where is my backup? Is my backup safe? And you know, the other piece to this is as as you just said, if I'm starting over from scratch, if I have to back it up from six months ago, I'm gonna have to take everything in this line back to six months ago. I can't just upgrade one and and do a backup to it because all of the logic changes are gonna.

Aaron Crow (19:52.624) be incompatible with each other until I take them all back to that same version. and I'm not necessarily just talking firmware. I just mean even the lot the logic changes that they're making on a daily, weekly, monthly basis, I'm gonna have to take all that stuff back because it's gonna be incompatible with the version that I restored from a month ago or a year ago or two years ago.

Tobias Nitzsche (ABB) (20:9.638) Yeah, and then you then potentially you have tuned the the operators maybe tuned while the system is running specific things which have never or which are not the let's say the values that you might have in your starting program or in your in your default, but they have tuned it because they have worked with the with the system. So these kind of cold retain values, do you have a do you have a secure you have a mechanism to capture them potentially yes or no? I mean A V B systems can do that.

Aaron Crow (20:13.787) Correct.

Aaron Crow (20:37.201) Sure. Yeah.

Retain Values, Vendor SLAs & Who Actually Shows Up

Tobias Nitzsche (ABB) (20:39.622) Others also can do that, but you need to secure them, you need to store them somewhere, backup them. So yeah, it's a different story than in in IT systems, but it is manageable if you think it through, potentially with your preferred vendors, and then because they have the experts around that. Same for and then if a disaster happens, if you need somebody on site within a specific time frame to help you with restoration or r restoring the system.

Tobias Nitzsche (ABB) (21:11.263) Is that everything that you need the vendor for? we have that in the in the past too. So yeah, A B B system engineer comes in to restore the system, but it's the system engineer, it's not the cyber A B B cybersecurity consultant expert whatsoever. And if the system then wants to investigate or want to find out how to close the pro close the the gaps in the system or find the problem, I mean yeah

Tobias Nitzsche (ABB) (21:35.453) That's may wasn't maybe part of the let's say SLA or of the of the call the guy you called. So things like that should be also considered in the beginning in your disaster recovery plan, instant response plan, in your communication plan. So again, but these are normal exercises I would say that we do as customers. but yeah, s it should be thought through.

Tobias Nitzsche (ABB) (22:0.783) Then it depends on where you are. If you are in the energy industries, many of the customers have expert teams, have these expertise. If you are in other, let's say, I would say, industries where maybe the they work with different budgets. So they have maybe no dedicated OT cybersecurity guy there. It's the operations team that is an IT guy, and then somebody needs to do it in between. So

Tobias Nitzsche (ABB) (22:30.022) I don't know if it is for water in many cases, the smaller ones I would expect like that. They still can do something, but it's a different exercise. And again, we are starting with w what what are the foundations and the basics they they should follow.

Water Utilities: One Person, Fifty Jobs

Aaron Crow (22:44.601) Yeah, I mean, obviously you you you talked about water. I think it ev it should be relevant for everyone right now. you know, go go look at Tobias' article. But, you know, obviously we we also just saw FBI release around Minnesota and and now others. I I literally just gave a talk at DEF CON, myself and Tom Van Norman and Dylan Lee on on, you know, the basics and foundation of OT and and and we talked about those water issues. And to your point, right.

Aaron Crow (23:10.117) You know, you these water municipalities that they've got they've got one guy and and that person, guy or gal, that person is responsible for, you know, the operations of the site. They're probably mowing the yard. They're they're doing all of the I'm I'm not trying to be rude to anything, but they just have multiple roles and responsibilities at that place because they don't have a large team. So they're literally taking the trash out and and doing anything that is needed to do for that, including the cyber responsibility.

Aaron Crow (23:38.311) Right. And they may or may not have a a a deep knowledge of those cyber. They're very capable. They're very intelligent. None of that's the question. Even just the most intelligent cyber person, but if I have 50 other responsibilities, I just don't have time to focus on looking at logs and all of that type of stuff. So that's an issue that we have in these water utility districts for sure.

Tobias Nitzsche (ABB) (24:0.313) Yeah, we we we see that in multiple areas. And this is also one of the focus areas that we had with our some of our solutions that we have, if it is the cybersecurity workplace, that we want to make it easier for the operator staff to maintain their cybersecurity. foundational controls. So yeah as you said, th this is one of the the things leverage the OT vendors' security capabilities to make it easier for the operation stuff. I think that is that is a good

Tobias Nitzsche (ABB) (24:31.048) Yeah.

A Cyber Incident Does Not Need a Nation State

Aaron Crow (24:31.142) Yeah. Yeah. And and you know, go going back to you mentioned we we we do tabletops, we do disaster recovery planning on the operations side. We need to be thinking and and having the right people in the room when we're doing those. So we're thinking about a cyber attack. And and a cyber attack does not have to be a nation state attacker, Iran, whatever n comp country coming at you. It can just be something bad happened and I bricked my my

Aaron Crow (25:0.476) Controllers because I got a bad piece of firmware or it failed or whatever that may look like. A cyber incident does not necessarily mean, you know, nation-state attacker, right? I still need to be able to recover from firmware that failed for whatever the reason is, or a bad configuration, or whatever those digital sides of things are. I I need to think through those things. I need to have a backup. I need to be able to test a backup. And that's the other piece, is a lot of these environments they don't know how and that have never actually tested.

Aaron Crow (25:30.350) restoration of a controller in the in this process because they're running all the time, right? They're running 100% of the time. They don't have an outage window on Friday night like it we do in IT that they can just reboot and test restore. They're not just testing a file. It's it's different.

Restore in a Bubble: Test and Scan the Copy, Not Production

Tobias Nitzsche (ABB) (25:48.017) Yeah, I I I worked in a project with a customer who yeah in the in the oil and gas sector and yeah they it's twenty four seven for sure. y you there is no shutdown. Even if there is a shutdown, it's just par partially, right? So it's never everything shut down or these I at least never to not typically. So you need to find ways around that. So you you put

Tobias Nitzsche (ABB) (26:16.754) Potentially you need to have a separate system where you can from you you take the snapshot or the backups or whatever you could get from the original system and then you need to restore it in in the bubble, wherever it is. And then also did this is also by the way, an interesting way to beat be beat everything out of it with whatever kind of tool you want without harming your actual production system and still get all the insights of the copy, so to speak.

Tobias Nitzsche (ABB) (26:46.054) So y you can get you can get very, let's say, intrusive on your copy of the system, but you you shouldn't do it potentially on your production system. But it is a way yeah, it's a s suitable way to get more out of the system that you would never do on a production. So also that is

Aaron Crow (27:4.816) Yeah, and I and I see I see more and more organizations. I I've built a lot of really, really robust hardware replication labs for environments where we have, you know, we duplicate hardware and we put them in there. For that, that's one of the use cases is we're we're testing restoration. We're backing up and restoring and trying, you know.

Aaron Crow (27:23.114) And we're testing logic changes. We test there before we do a firmware update to the latest version and make sure that the logic works and that, you know, it doesn't change, you know, it doesn't open ports and services. You know, again, some of these requ are required in Power Utility that I have to do if I'm going to update firmware, I then have to do test it against firmware or or ports and services. Like I'm not necessarily doing that in r industries that I'm not regulated to do so.

Aaron Crow (27:48.647) I'm gonna test, I'm gonna update a firmware and then I'm just gonna validate that it works and I'm gonna move on down the road. But I don't necessarily know, hey, now this this device has got Telnet open. did I did I know that it was that telnet was open on this system before? Was it open before? Like those are things that it's kind of that next level step. And and it sounds stupid because we do this in IT all the all the time because we have Nessa scanners and a device shows up on your network and you're scanning every port on it. You can't do that in OT. So you have to do it more intentionally.

Aaron Crow (28:18.635) that doesn't mean you can't do it. It just means you have to do it differently and be intentional about it.

Tobias Nitzsche (ABB) (28:23.614) Exactly. So y you mentioned Nessos or whatever kind of vulnerability scanner you might run against a system. This this some of the scans are intrusive and I wouldn't never r run them and the same as for Wiresh. Whatever kind of tool you want to use, yes you can yes, there are ways to do it if you know how to do it. and what is the thing that you want to scan, but if you blindly do stuff in a network that you don't know,

Tobias Nitzsche (ABB) (28:50.534) The chances are high that you harm something. So that said, if you copy your DCS nodes to another place, run the scanner against the copy, you still get the same information out of this at least if it is the the software layer, you can still get the same information out of it as if you would have scanned the original system, at least from from that perspective. So I would say it's it's a doable doable task.

Aaron Crow (29:16.796) Percent.

Backups as the Last Defense Against Ransomware

Tobias Nitzsche (ABB) (29:21.555) well what was one example. yeah, I I would I would agree. So backup and recovery is one of the mechanisms which should be utilized more for different security regions. And it is also one it is also one of your, let's say, last defenses that you have, for example against ransomware. So y if you got encrypted and you know you have your backups right and you can restore within a time frame.

Tobias Nitzsche (ABB) (29:49.447) You're in a different kind of let's say situation than if if everything got encrypted and they extracted the information and then you get into discussions with these guys. So

Aaron's Exchange 5.5 Story: When the Executives Lost a Week of Email

Aaron Crow (30:1.723) It it it's funny because, you know, I I I also came from IT. So I've you know, about half my career I was spent in enterprise and in IT and I worked for a very large company. and I was the active directory and exchange administrator. So I managed all the exchange email servers and all all the domain controllers. And and this was like a, you know, large hundred plus thousand employee company. and we were this this tells you how old it was, but this was back in exchange.

Aaron Crow (30:29.969) 5.5 to 2000 time frame. And we were doing a an upgrade from exchange five five to two thousand. So again, it tells you that's why there's gray in my beard. but yeah, the at the time we didn't have high availability on our exchange servers. And it was decided because it was too expensive. We had way too many users, it was way too much expense. So we had backups, but that backup was a full backup. I wanna say it was like once a week.

Aaron Crow (30:58.759) And then we have incrementals every you know, every day, every night from that. but you know, if you know anything about backups, audience, like when I take that full backup, if I'm gonna do a restore, I have to restore the full backup, then I have to restore every incremental backup. And then the dip the delta between the last incremental backup and and when the server failed, I'm I'm just losing all of that data because there's just it it it wasn't there. Anyways, long story short,

Aaron Crow (31:28.507) We got hit and and and it wasn't a cyber attack, but we had physical failures on on an exchange server. And it happened to be the the one for whatever reason that all of the executives were on. So that w for whatever reason all the the C suite executives were on this one exchange server. That exchange server failed. So all of a sudden, when I when I was recovering it and we couldn't get their email back and they lost like a week worth of data because the backup.

Aaron Crow (31:58.548) We had multiple backups, the incrementals didn't work. They thought we thought they did, but they didn't because it was back on DLT tapes and the tapes were bad. Like all of these problems went through. And when we lost a week of the of the executives email, all of a sudden it was like, huh, I think we're gonna go with high availability, at least on the executive's email server, because now it's important.

Tobias Nitzsche (ABB) (32:23.149) I mean yeah.

NIS2 Article 20: Personal Liability in the C-Suite

Tobias Nitzsche (ABB) (32:26.064) Yeah, that that's also a common pattern and I I guess that's normal. If if you got hit, you you think through why and what can we do better. I mean that's a it's a normal behavior. I would say it's it's good, but it shouldn't be the the only trigger. Say and and I mean but yeah, if if it shifts let's say the focus for the management team to that specific problem, yeah.

Tobias Nitzsche (ABB) (32:51.836) I mean, there are other ways. Say for example, again I can I can say about what what NIST two is article twenty does. I mean that it introduces personal liability for the executives. So yeah. It is for sure no longer only the IT p department's problem, the cybersecurity. So no it's not. So they they need to get trained, they need to understand the risks and they need to sign it off also the risk assessments and and the risks that they accept. And it's not only the CIO's tasks anymore. So

Water Is Waking Up: Put the Focus on Recovery

Aaron Crow (33:20.039) Right. Yeah. You know, and that that's that's the point is unfortunately there's gonna be a lot of organizations that that think I'm gonna be able to keep everybody out. They're not spending as much effort on recovery. and or that they think it's not gonna happen to me. And until a bad thing happens to either me or you know, my friend, my my my competitor, whatever down the street, it's not real. so you know, when I I think water utilities are waking up this week and thinking

Aaron Crow (33:49.916) What can we do? Because we don't want to be like all these others that are impacted. And we need to think about and and to your point, it's not all about firewalls and cyber tools. A lot of their focus needs to be on recovery. Like how can I make sure that I can back up and recover in these environments in case something bad does happen? how do I get back up and going as fast as possible?

Tobias Nitzsche (ABB) (34:12.518) Yeah, that's true. And and it it is I wouldn't even say this is a cyber security control. And the reason for that can be anything b or cyber, but yeah, i I would say that's that's why I I would say one of the fundamentals, absolutely. Yeah. what I

Aaron Crow (34:21.908) A hundred percent.

Aaron Crow (34:35.070) Sure. Yeah. A hundred percent.

What Was Hot at DEF CON: AI on Both Sides

Tobias Nitzsche (ABB) (34:40.966) What I would like to maybe ask you when you were at DEF CON there, w what was the there's always the shiny thing that is i is it is it was it still AI and in and if so or is it something else already what was discussed there?

Aaron Crow (35:0.540) There there was a lot of focus on AI, for sure. you know, lots of folks were coming over. you know, again, I I spent a lot of my time in the ICS village. I I I actually took my 17 year old son with me to to DEF CON this year. So that was that was fun. It was his first trip to to Vegas, but also just kind of seeing all the things he's not really a cyber person, but still it was really cool to see and and be part of. But yeah, it's to answer question.

Aaron Crow (35:28.548) A lot of the focus was AI and and you know, everything from mythos and vulnerabilities and zero days and and how can I use AI in critical infrastructure and and how can I use it as a tool and a and a and a cr and a and a capability, but also how can I defend against it? And and that's the piece that I think that I was excited to hear is it wasn't just the latest tools and processes.

Aaron Crow (35:53.211) It was also, hey, we know that bad actors are gonna going to be using AI to try to attack us. How can we use AI to protect us? Right. Those are conversations we need to be having. and we need to be having them early, not like we've done in OT in the past, where we just wait until 10 years down the road when everybody else is already using it and now we're just trying to catch up.

Where AI Belongs in the Purdue Model (and Where It Does Not)

Tobias Nitzsche (ABB) (36:15.695) Yeah. I I I would I would also say that this is one of the one of the things where we need to be faster this time, right? So we we we can't cannot wait. But the other thing is also where would you do it? I I wouldn't do it if we if we take the typical architectures or the IC six two four four three kind of zones and conduits approach. So where where would I see AI first? I I would see it not in la

Tobias Nitzsche (ABB) (36:43.666) in level zero or one I would say. also I wouldn't see it at the beginning in level two potentially, but at in three. Sure, why not? And and here they the ri the I would say the the risk is lower than the possibility. So yeah there's a lot of there's a lot of opportunities that we have enriching data, detecting things faster, giving more contextualized information.

Tobias Nitzsche (ABB) (37:13.606) I think that there it's helps. So I can give you an example out of my work where where I use it at the moment or where I have let's say pre pip prepared an agent that can be used in our in within our team. So especially on this legislation p kind of thing, what do I need to know? What I what what is what I what I need to do? How does it compare to my situation in my country, in this region? Wha w what I what kind of things do I need to look after?

Tobias Nitzsche (ABB) (37:43.301) it's super good. So you you you have a you have a you have something that helps you very fast with information, which you before might need to have go with local experts. you might have never got the information because and nobody knows. and and then and with that you you really have it fast and easy. So having a legislation landscape all over the world i is doable for from a from a central point of view and updating that.

Tobias Nitzsche (ABB) (38:11.772) Things of that. So I think there there's a y great potential in the their telops.

AI as the Daily Brief for the Understaffed Operator

Aaron Crow (38:16.338) Yeah. Agreed. Yeah, I agree. I mean, I I don't want my I don't want an AI agent, you know, doing control in in an OT world, especially not today. Maybe we'll get there. We probably will get there, honestly. we're we're not there today. but there's a lot of things that you can do. we just talked about water utility and and how they are understaffed and and how they don't have time to look at things. Imagine if we were just sending logs from

Aaron Crow (38:43.048) firewalls and alerts and things like that, and allowing AI not to take action, to just say, Hey, Bob, these are the things that came through. These are the needles in the haystack you should go take a look at and decide what you want to do on them. Right. So that morning daily recap of things that happened last week, that today, and and also, you know, the the the daily brief of, hey, here's the things in the news. There's seven or thirty plus

Aaron Crow (39:9.970) Water utility districts in Minnesota that were impacted. This is how they were impacted. You also have this problem because you have these same types of controllers. Let's make sure that your controllers are not on the internet, right? Like these are these are things that we can use AI for that doesn't add a vulnerability to my OT environment, but adds huge value on supporting that environment, right?

Tobias Nitzsche (ABB) (39:30.800) Makes it faster.

Tobias Nitzsche (ABB) (39:34.557) Yeah, it makes you faster in in in in your decision and this is where we need to be, right? We we need to get a lot faster. Also with these reporting obligations. So you if you need to report in twenty four seven, that means you need to detect, make sense of it, and all of that before. So having that said, of you have it you named it event monitoring, network monitoring for lock collection mechanisms.

Tobias Nitzsche (ABB) (40:4.644) And all of that, if if if you have that in in the old way, like manually with passive with rules that somebody engineers th this and that correlated, there's alarm, please send it out to someone, they need to investigate. There's still a lot of things that we can automate. And I mean, automation is where we come from. So there's a lot there's a lot wor knowledge that we have on on the way how to automate things and therefore

Tobias Nitzsche (ABB) (40:33.488) I guess here with with with AI that's just another tool to automate and be faster.

The Wireless Pineapple on the Plane

Aaron Crow (40:39.539) The the good thing is is that I I see I see people having the right conversations. the good news from from our perspective and people like me, you know, I don't see any lack of work coming up over the horizon of there we're we're we're we're definitely in demand and need. you know, and and lots of folks coming from DEF CON asking, How do I get into OT? How do I how do I learn more about OT? and yeah, I'm sure some of them are looking at, you know, trying to hack

Aaron Crow (41:9.011) I literally saw this morning a Reddit thread where somebody grab from DEF CON flying back from Vegas to wherever they were flying. They bought a you know, a wireless pineapple at DEF CON and while they were on the plane, tried to take over the wireless hot the the Wi Fi and and and the pl and the the airpo the pilot detected it or they saw it and then you know obviously called in support, landed.

Aaron Crow (41:38.806) They didn't leave anybody off the plane until they did like it's just like, come on. That's just the dumbest, dumbest. Like, you can't go anywhere. Like, how can you you you don't hack when you're stuck on the plane and you don't hack the plane that you're on? Like, that's just you shouldn't be doing that anyways. But my gosh, why why are you doing that? I'm guessing it's somebody that had watched some talk of somebody on how to do wire, you know, the wireless pineapple and decided, hey, let me try it right now while I'm 30,000 feet in the air.

Aaron Crow (42:7.389) Not intelligent. Let's not do that.

Tobias Nitzsche (ABB) (42:7.810) And yeah, and and also thank God the engineers of the airplanes they they know at least today that you should not mix up entertaining wireless lands or entertaining networking with the with the control ones. So yeah.

Aaron Crow (42:24.093) With control.

The Unsexy Foundations, Again

Aaron Crow (42:28.797) Yeah, for sure. You know, and and and unfortunately, and you know, again, I I know we we talked about this multiple times and and I feel like I'm like we're on repeat constantly, but it it's that foundational thing, right? To your point, the fact that, yes, maybe they they could impact the the the wireless so that you can't, you know, stream YouTube, but it wasn't impacting the actual flight navigation systems, right? You know, having that segmentation is is huge. It's foundational.

Aaron Crow (42:57.749) And and we should be looking at those foundational things across our organizations. Anytime I talk at OT, most of the things that we need to do are not sexy. They're not super expensive. I don't have to buy a lot of tools to do them. I just need to do foundational things, back up, make sure I have segmentation. I need to have secure remote access. I need to have secure file transfer. I need to monitor my environment. I need to understand my assets and more than just an asset list of of types of devices. What do they do?

Aaron Crow (43:23.987) And what's their function and how critical are they to my process? Like those are the basic things that I need to know at an OT site.

Borrow the Safety Engineers: Hazard Studies as Risk Input

Tobias Nitzsche (ABB) (43:30.482) Yeah, and involve the people that you already have. for example, you you I mean, it it's a safety kind of topic that you mentioned here. You say, Yes, you have safety engineers, they might be already doing hazard studies and they they know how how what could go wrong, what would be the impact. The the same people could help you with the risk assessment and the input for that because they they they do that for it

Aaron Crow (43:59.318) Hundred percent.

Tobias Nitzsche (ABB) (44:0.295) Yeah I mean yes, you you maybe need to bring in the cybersecurity part, but with the information of the hazard documents you y you have a good understanding what could be the impact on your risk assessment, specific risks of threats.

Aaron Crow (44:13.020) Exactly.

Horizon: The AI Trust Problem, Both Sides of the Coin

Aaron Crow (44:15.689) For sure. Well, awesome. So what what is I I I usually ask folks this question. I kind of stopped it for a while, but I really like it, especially after our conversation today. Like what's what's one thing coming up over the horizon that's concerning in this space and maybe one thing that's exciting that you see that you're you're hopeful for?

Tobias Nitzsche (ABB) (44:36.774) I mean we we touched it I would say w what what is concerning and what what is coming. I think but it's w with with AI. For me concerning is to it give too much trust and too much okay, we have the thing, it will detect, it it's running, it may detect if something happens and you you always need to do your due diligence and it doesn't matter what kind of tool you use, and therefore this is for me one of the risks to

Tobias Nitzsche (ABB) (45:6.962) To leave too much trust in these things and let it run. And the and the the but on the other hand, this is also this the the same tool can be a a great yeah utilizer or multiplicator if you if you do it right. So I would say it's both sides of the coin the same.

Aaron Crow (45:28.777) Yeah, a hundred percent. So what what's the what's the call to action for folks to to learn more about what you guys do at A B B, learn more about you, find your water article about the water utility district stuff, all that kind of stuff. Give give everybody the the the call to action.

CTA: Legislation Is a Utility, Not a Paper Exercise

Tobias Nitzsche (ABB) (45:49.710) I I think way w where where we started, I mean don't take legislations as a paper exercise, take it as a utility, get broad attention. If you have it now, then make use of it and get the foundations and the basics right. Start with risk assessments, but not only as a paper exercise, not only as a as a questionnaire.

Tobias Nitzsche (ABB) (46:16.380) checklist or something like that. And don't auto-generate it with with AI because you need to understand the risks that you have. If you auto generate or if you use AI for it, then at least you need to understand if makes sense if it is true. And yeah, and then yeah, we nay B we we are there to support you. If you're a small or a medium or a large customer, depending on the size and the challenges you have.

Tobias Nitzsche (ABB) (46:44.050) potentially involving the OT vendors can can can be on one hand on the SLA part, but also on the modernization part, yeah, the right partner for these topics. We we know the OT stuff at best, for sure.

Aaron Crow (46:56.574) Excellent. Yeah, for sure.

Aaron Crow (47:0.094) You you guys have been around a long time and been doing this. Like I said, I I was literating a plant the other day that again, they the equipment's probably 30 years old if it was a day old. and it still runs. Everybody's happy with it. Like still, still great system. you know, and I and it and it's everywhere, right? So and that's that's the the the the thing I love about the OE inventors like you guys is, you know, you support your equipment. You you support your your processes and your systems and your people.

Aaron Crow (47:25.700) instead of well, that system's 10 years old. We don't support that anymore. Buy a new one. Like that, that's never something you see in an OT world. And that's what I love about it. So we'll definitely include all the all the links, anything you guys want to have in the show notes, definitely look there. There'll be the link to the the water utility article, how to get a hold of of A B B and and Tobias specifically. and then, you know, kind of where where what's next with you guys. So

Aaron Crow (47:51.338) Man, thanks, thanks so much. I I love I love digging into all all things O T and and and especially working with with you guys. It it it it's always a pleasure.

Tobias Nitzsche (ABB) (48:0.914) Thank you for having me.

Aaron Crow (48:2.742) Yeah, absolutely.

Tobias Nitzsche (ABB) (48:5.147) Bye bye.

Transcript lightly edited for readability.