Walk the Floor: Why Your Asset Count Is Wrong

Source: Riverside aligned transcript of clean-body edit 6ae228ef83d9190e48996434 rev 3 (pauses over 1.5 s shortened, fillers Smart). Times are clean-body playable M:SS. Backchannel (yep, mm-hmm, right) dropped, stutters collapsed, names corrected (Kasm, Avangrid, Cyolo, OPSWAT, Barry-Wehmiller, Cybereason, Forescout, Labshock, OSINT). Still unresolved by ear: 'company coup' (15:34), 'steel dome operating system' (51:5x), 'Open ZD' (51:3x), 'GM Hyde Clark' (47:43), 'AFAR' (56:29), 'goalie light ... off the post' (49:3x). Not for publication as-is.

Aaron Crow (0:00) Thank you for joining me on another episode of Protect It All, where again, I say this all the time. I love this job. even though it's not a job, it's it's something I do on the side. But I love getting to have these conversations with people that, you know, I'm working with. It's a small community in a small world of OT cybersecurity. And those of us that have been in it for a long time, we see each other at conferences and at client engagements and all the things. And today's guest is no no different in that way. So Kevin, introduce yourself. First of all, thank you for taking time. Thank you for being here with me. Thank you for wearing your AI socks. Those are the best socks. and then introduce yourself. Who are you? All the all the things that people want to know that may not know you.

Kevin Kumpf (0:44) I'm Kevin Kumpf. currently I'm consulting with Kasm, Barry-Wehmiller, Xeed, you know, a whole bunch of different companies out there. I've been the Johnny Cash of OT Cybersecurity. I've been everywhere. literally director of OT security for Avangrid, worked at Case New Holland, United Healthcare, and then just on the vendor side alone, helped launch the OT Vision for Cyolo, OPSWAT. Reversing Labs, Forescout, Cybereason, SSH, you name it. just a whole plethora of organizations out there that now most of them are winding up in the OT realm.

Aaron Crow (1:25) Yeah. Yeah. So how did you what is your origin story? How did you get into OT all however many years ago that has been?

Kevin Kumpf (1:36) I actually started out in a steel plant, hence where I got the blue hard hat from many, many moons ago, where I was going to college for IT technology. And they said, Hey, we've got this great quote unquote data entry job for you for the summer. And I went to a local steel plant that was basically strapping steel, and they said, yeah, well, you'll do that about 10 hours a month, and the rest of it you'll be out here strapping steel. with the giant gantry trains going over the top. And I'll never forget the guy that came up to me on my first day of work when I'm sitting there going, Wait, this isn't I wore a dress shirt here and he's like, don't worry about it. The oil comes out. and he handed me a hell this hard hat and I said, What's this for? Looking up and he goes, he goes, It makes it easier if something lands on you to scoop up your brains and put him in the helmet. And I was like, Well, at least you're honest. And I wound up getting into their systems that were controlling the cranes and things and they weren't working right. And then I said, Well, did you guys ever check for a software update? And they were like, We don't update the software here. And just rolled from there. and it was interesting. I didn't expect it, but then again, what's the line nobody expects the Spanish Inquisition? So so I wound up in the industry.

Aaron Crow (2:54) Yeah. No, it and it's it's so funny for folks that have never been in OT. we're used to you can't see this right out of camera, but I've got my hard hats up there as well. you know, we're we're so used to it. And in an IT world, your uniform or your dress code, you may have to wear business casual and slacks, and maybe you've got, you know, casual Friday. we don't have that in OT. Like every day is, you know, we're wearing arc flash rated clothing and steel toe boots and hard hats and you know I can't wear metal rings or watches as I'm getting into cabinets and you know all this stuff and we're doing pre-job briefs and talking safety and safety zero and all of those things. It's just a different culture and mindset. And unfortunately I think any of us that have worked in the industry long enough, we probably know of people that have either been Severely injured and or lost life. we don't take it's not just theater, right? You know, all of this stuff matters. And that's that's one of the big differences I see in IT and OT is just that understanding, realizing that the things I do when I'm reconfiguring or rebooting or configuring a firewall or whatever the hell I'm doing in my widgets online, there's a real impact. in the world and when you're when you're in those plant environments, those manufacturing environments, those real when there's steel hanging over your head and you've got a piece of plastic protecting your head, that if one of those pieces of steel fell, that hard hat's not gonna do anything, as But it's still part of the safety culture. But it at least makes you think about all those and that's a it's definitely a different mindset as you're looking at problems from an OT lens versus an IT lens and how you How you solve and solution for those things and what the impact of those things can be.

Kevin Kumpf (4:51) It is. And it's so funny. You know, I've had to fill out the injury and the death reports at Avangrid and validate them when people were injured or trespassed in the substations. stealing ground wire, which is a huge one. It's like, well, this is the ground, you don't need this until all of a sudden you become the ground and then well you need this. it's clearly a problem. you know, and I've climbed the, you know, the high towers and having all your wrenches and things on cables to make sure they don't drop down and kill somebody. and in fact, I was talking to my friend last week. There was three of us talking, he goes, I picked up this beryllium wrench online and it was almost only 900 bucks. And the other friend looked and goes, Who the F spends $900 on a wrench? And was like, Well, when you want the anti spark and you don't want to go somewhere quickly in a million pieces, this is what you spend the money on. It's funny, you know, you speak at a lot of conferences just like I do. You will see the people come to the conferences in their safety shirts with the tag on it, a year of safety or X number of years on a patch. And they wear those because it's a pride thing. In IT, I tell people you'd never put up a sign in an IT, you know, soccer data center that says X number of days since your last incident, because it would never get past the single digits. W when you're an OT, if you get below like the three digit realm of a hundred, they're like, Man, that sucks. Single digits is like just don't look up there. That's that's a huge difference. I tell people like, wait, what's that sign about? I'm like, it's everything. It's what you work for,

Aaron Crow (6:35) What what you just said, I didn't think about this. So I worked at a power utility for 10 years. and every year on my anniversary of being hired, most companies have, hey, Aaron's been here a year. Everybody congratulate Aaron. Yay. And my current company does that, right? and I'm not bashing any corporate company that does that. There's no wr there's nothing wrong with recognition. At a power company, it was different though. Every year I got a new sticker. That would go on my hard hat or wherever I wanted to put it. And it was not a year of service. It was, I've been one year without a safety incident. I've been two years without a safety incident. I've been three years without a safety incident, et cetera, et cetera, et cetera, because they wanted to, yes, Aaron's been here for 10 years, but I'm going off of how long have I been able to do it safely. Right. And it's not because we want to shame people for making for getting injured. That's not the point. We want everybody focused on safety first because there's an impact there, right? And so every year I had when I left the company, it was 10 years. So I had been at the company for 10 years, but it wasn't a 10 years of service. It was a 10 years of safely working, right? And every year, there's guys that had been there for 40 years and they had those badges. There's also guys that have been there for 40 years and only had a 20 year sticker. Because they had an incident at twenty years and they only had twenty years of safe working, right? And it wasn't their fault, but it's just part of the part of the part of the day, right?

Kevin Kumpf (8:06) Yeah, and I learned that real quick. When I was at that steel company as a strapper, we had a machine that was that would cut the scrap steel into small pieces. And they had a guy that was on that machine and he kept overfilling basically the cutters. And they kept telling him, Don't do this. You're gonna basically kill yourself or kill somebody else or break the machine. And he kept doing it. It was a Friday afternoon. I'll never forget it. And He was jamming as much metal in there as he could to get out of there quickly. And me and another guy were standing there, and he brought the jaws down, and the hydraulic line basically snapped off, hit the side of his leg. And me and the other guy, the other guy basically tackled him. I wound up yanking him out of the way, but he lost his leg. And that you don't realize how high of a pressure that hose is. And basically it's like those air needles that when you're in the military, you walk through the door, they shoot the shots in your arm. It's and that to me was a my God moment where this guy had basically lost his leg. just because you wanted to get out of there on a Friday and he kept jamming those jaws full of basically that, that extra steel we had. So you see it and then you realizes that I shouldn't do this. There's there's no shortcuts to safety.

Aaron Crow (9:32) Well, yesterday my wife I so she was putting up stuff in the top of our cabinets and the cabinets are really tall. So she had a stepladder and then I needed the stepladder. So I took the stepladder and then I came back and she was standing in a chair. and my mindset still it because I've been beat over the head so many times over these decades of working in these spaces, that's not safe. And she's like, Well, I was I was fine. I'm like, yeah, you were, but it wasn't safe. That wasn't well. And then she was like, well, but you took my step ladder. I'm like, I get it. And I'm sorry for that. But at the same time, you should have just waited or went and got a different ladder or something else. But that's the mindset. And I saw that a lot as I was building teams to work in this OT space. I would in I would hire people that had an IT background, right? And they were super smart and understood firewalls and networking and servers and all that type of stuff. But when I had them working in my environment, it was hard to get them to understand safety. why do we do this safety, you know, safety moment before every meeting? And what until you actually take them out. So usually what I would do when somebody came to my team the first like month, they never came to corporate headquarters. They were at a plant. And it's like, you know, drink from a fire hose of what it's like and what we're supporting. So that way you don't ask these. stupid questions in the future of why is this different and why can I not just reboot an operator workstation at two o'clock on a Friday because there's a patch update, right? And why what the impact of that can actually be.

Kevin Kumpf (11:09) Well, it's funny because I approached one of the major concert sponsors out there when I was speaking at a conference two years ago and said, Why before the conference don't we do a safety moment each day? And he said, Why would we do that? And I was like, And yet you bill yourself as an OT conference, you know, expert, like you understand the industry. And I said, We should do a safety moment. And he was like, Why? And I said, even pointing out the fire exits or something, or before the audience saying, Hey, what's a safety thing that you do in your job day to day that's important? And you never hear people do that. and I've done it a couple of times and people look at me odd, but then you can tell the people in the room that they're the OT people. so you know it comes to and the funny thing is we're not trying to shame the IT people. There's IT people coming in the space. We want them to understand. And the only way you do that when you're out there speaking is to interact with the audience. And so many people just get up there, want to blow through content and say, Hey, I have a book over here that's signed if you'd like to pick up a copy, or come see us at the booth for the product over here, or you know, who wants to go to dinner on my tab, you know, as a potential customer. That's that's not why I speak and I know it's not why you speak.

Aaron Crow (12:36) Not at all. Even though I think you're jabbing me for my book, but that's okay. That's not

Kevin Kumpf (12:40) No, no, not no, believe me. I'm sure we can both think of somebody else out there. We know that's on book twenty-eight of a four hundred and fifty book series. which you know, but I mean that's the funny part is you know, when I spoke last week or two weeks ago, I asked people in the audience how many people know your baseline? One guy put his hand halfway up and I said, Is it halfway up because you're not sure? Or because you don't want to be called upon to talk about it because nobody else is raising their hand. And I said to the other people, I'm like, why didn't anybody else raise their hand in knowing a baseline? And that the and it's like they don't want to be embarrassed. And I basically laid in two guys, we're here to share information to learn from each other. This is a this is a no-judgment zone. This is an enablement zone of I'm here giving my time to present some material for you that hopefully is important. And let's talk about it and let's talk how it impacts you.

Aaron Crow (13:39) Yeah. Well, it and to me it gets to that, what so I know we've been we haven't talked technology at all, guys, and that's okay. It's so important the things that we're talking about. When you go to these OT spaces, they have this safety culture, right? This the entire culture is focused around that is the driving factor. Nothing else matters, not how much money they make. Not that it doesn't matter, but nothing is more important than safety. I would never like Any of the places that I've worked, and I've worked at some of the largest, manufacturing and power utilities and all those kind of things in the world, and every one of them has the same mindset. Stop when unsure. Like nothing trumps doing things safe. It is not worth risking life limb injury just to get this thing done. I don't care if you're shutting the entire plant down, shut it down. Right. And that's how they're trained, right? And that's that's a culture that has it. IT doesn't have that. if you make a mistake in IT, people shame you. If you're if you do that in corporate America, they're like, look at Aaron. He made a mistake. He configured something incorrectly, or he said something stupid, which is why people are hesitant to raise their hands instead of saying, hey, I'm not sure. I'm here because I don't know and I need help. And that's the culture. That's the biggest thing I think we need to change in this ITOT conversation, is we want you to be unsure. We want you to know. I don't know what my environment is and I need help because we want to help. People like Kevin and I are out there trying to help you, but we can only help you if you're willing to admit you need help.

Kevin Kumpf (15:17) Yeah, and the other part is that when they take things up to management, this is where the C suite has to be really involved. If you look at the sales cycle in OT, people are like, well, if I get to the CISO, he'll do this and he'll buy this and he's my buddy. But then it doesn't go anywhere. We've seen so many products out there. everybody says know your asset, and yet I can guarantee I was just the company coup last week. And one of the people said, you know, one of the vendors I was with said, How many control cabinets do you have? And they said, one guy said fifty in that plant. And the other guy goes, Well, I think it's closer to two hundred. And yet I'm like

Aaron Crow (15:53) It's a big difference.

Kevin Kumpf (15:55) And then talking to just reviewing a spec that was done for a customer a couple of weeks back, they were off on their assets 153% because they figured a tool would solve it. And so everybody's selling this bill of goods of hey, you buy this tool and you're good. The problem is, is there's not enough people that understand what that tool's supposed to do. And now that I'm working building a platform for the industry, you know, which is where we saw each other last in Houston. Not my managers, look, I want you to focus on the people and process. And we'll bring in the technology to help match that. but it's it's difficult because when you sit down and you say, what's your network architecture? What do your maps look like? What does this look like? They're like, hmm, is this multiple choice? And that is where IT can help us is IT generally knows what their assets and their resources and their infrastructure is. OT is controlled by the vendors that show up and put something in a cabinet and walk out the door, and the plant manager's like, hey, it's running, I'm good. Because they don't want to know. And that, I mean, let's let you know, me and you talked before we started this about the refrigerator hack that went over in France. And I just sat there and said, everyone is to blame. And by everyone, I mean is that. The vendor said, if you read, the vendor of the product said, well, they had devices out there that had old firmware in them and they shouldn't be using those devices. But then the next thing I read was that they were using the old firmware and newer devices. Then you look at the supermarket and they weren't monitoring for a simper tent simple temperature thing, which is what I showed you, what the OT community issue we built can do. And in fact, I've already got a call scheduled with our buddy and Mr. Farnum in December to show, look, we can already do this in a lab setting that you can do at home in 15 minutes. How hard is it to do for these people? The problem is that nobody knows what a baseline is of what that number should be, and nobody has a clue of what happens till the systems catch fire, supermarkets catch fire, all there's lost, and all of a sudden it's like, hey, that sensor really does matter. Well, it's always mattered. It's just to who and why.

Aaron Crow (18:24) Well, and like you said too, it's like a couple things there, right? Is the foundational things that we're doing or not doing are the big things. I started posting some funny memes for OT security, and one of them is the guy that walks by, he's with his girlfriend, he walks by, and there's, you know, another attractive girl walking the other direction, and the meme has her as, you know, AI tool, and he's looking back at her, and his girlfriend is asset inventory. Right. So basically it's saying, hey, he's focused on the AI capabilities that's coming out, and he doesn't even have an asset inventory or an understanding of what his environment is. Most of these attacks are not sophisticated. They're getting in because they're using the, the default passwords. they didn't secure them. They're running old firmware, they're directly on the internet. it's a combination. If any one of those things were changed, we're not having this conversation, or at least the impact wouldn't have been the same. But because they're running old firmware, they didn't change the passwords, and they had them directly on the internet, they got what they got, right? It's

Kevin Kumpf (19:30) Yeah, and it's funny, at a conference recently, and I won't mention names, somebody was up there promoting, goes, and the cloud, he goes, You guys gotta stop worrying about it. He goes, that's just somebody else's computer. And I was like, okay. And my bank, don't worry about it. That's just somebody else's vault in their house somewhere. It's like and and these people are sitting there, there's like 60 people in the room, and it was like none of them raise their hand and goes, Well, exactly what does that mean? That's somebody else's computer. And now, if you take that hack that happened to the supermarkets and you look at phase two of it, there was another supermarket that was hit, but they were monitoring the temperature. One of the premier understanding things of a door sensor, temperature, humidity, power, all these things here, you know, valve flow. I had a guy tell me that who cares how much. Water is flowing through a valve and who knows it. And I was like, Okay. And he's a chief OT strategist for one of the major vendors out there. And was like, guys, seriously?

Aaron Crow (20:38) Probably the pipeline that's putting a bunch of product down a pipe and wants to know how much is flowing through those valves.

Kevin Kumpf (20:44) Yeah, or you've got basically a utility that has to cool things and all of a sudden you shut the water down, I'm like, and it turns to a trickle. I'm like, you've got problems. And

Aaron Crow (20:55) and it's what it goes back to the other the other thing that you said there a minute ago, right? Is you have to know what good looks like to be able to know what bad is, right? If you don't know what your baseline is and you don't know what normal, good, everyday operations are, and not just it seems fine, but like documented, like these thresholds, it should be between this temperature and this and this pressure and this and this flow rate and this. If you don't know what good looks like, and same thing on the network layer, who talks to whom? how many devices do I have? those are all your baseline. And if you understand what your baseline is, then it's the Sun Tzu art of war. I say all the time is like things don't change very often in OT. We're not patching, we're not doing a lot of those more advanced things that you do in IT. But because things don't change very often, I should be able to see when something changes. If I know what it's supposed to look like and then something changes that's outside of the norm, I should be able to say, Hey, why is why is that doing that? Or why did this device show up here? Or why is the why are all that the refrigerator temperature sensors going off? This is not normal. This doesn't ever happen. Somebody should go take a look at that and see, is the sensor bad or has is there a problem? And then all of our refrigerators are in defrost mode, which is a problem.

Kevin Kumpf (22:19) Yeah, and this gets back to an inherent problem in the industry. And I actually ran into that a bunch of years ago and I was working doing a security audit and this cruise ship line had put out ten new engines on these ships. And I was looking at the baseline on the screen in this operations center. I said, Why is that one running at different RPMs? 'Cause I'm a car guy and stuff and I understand things. And one of the and one guy looked up from the screen at me and goes, turned his head and goes, Aren't you a security guy? Like stay out of my space basically, right? Well, I wrote it up in my report, and my boss ran back to me, why are you telling him this? I said, because there's nine others that are like this. And one of these is not quite the same. that whole childhood thing, one of these things is not like the others. Well it turns out they were getting their oil from a place different than everybody else. And it was reconstituted oil. It had been used once, screened and now it was thinner. The viscosity wasn't the same. And that engine was going to burn out three years to four years earlier. And I was like, that's important. I mean, because you know, but that's a baseline. And the problem is, is you go into these companies and you say, what is your most critical system in this facility or production line or infrastructure or process? And they don't know. And the scary part is they will protect. The most expensive asset coming out. Like Case New Holland, they were protecting combines. But yet the little five cent piece, the impeller blade that has to go in here to make something work, that plant gets no budget. You have to protect and you saw this in the utility space. They will protect the large basically capacity substations that are putting out for the general populace around hospitals. But Bob down here with two farms and a street light. He's got a chain link fence around it with a lock. Well, guess what? I break into that little shack. I go in and plug in. I'm on the same network. I'm literally so as I tell people, would you break into Fort Knox directly? Or would you go to the little bank in your town here that you could walk into and be on that same network and shuffle the money out in another way? Which way would you do it? And that is what's happening

Aaron Crow (24:41) Well, it's it makes me think back to one of the first projects I did when I was at the power company is they had something called they called it BTFR bail boiler tube failure report. And it was a really, really complex spreadsheet, literal spreadsheet. And it got data from a Bunch of sensors throughout the control system at the plant, at each individual plant. So there was one of these reports for every individual plant, every boiler. Let me put it that way, not plant boiler. So every some plants would have three or four boilers, depending on how many units they had, et cetera. and they had a report for each one. And the whole point of it was it was looking at data from chemistry. So the chemical make down of the water that's going through it and pressures and temperatures and ramp rates of the of the of the of the of the turbines and all these things that all these factors that go into. And it was this extremely complicated engineers working on this and like it was like, you know, beautiful mind of all of the data points that went into and all the calculations that went into. And the whole point of it was to anticipate when the boiler tubes were going to fail and would they make it to the next outage or would they fail before the next outage so that we should do it we should replace the boiler tubes in this outage. So like we did outages every spring and fall at these, you know, old school fossil power plants. And would we need to do the boiler tube replacements in this boiler in this current like spring or fall outage? Or can we wait another six months to a year, depending on these numbers? And they got really good at estimating those. Because the problem is, and for those that don't necessarily are not following me, if these boiler tubes go down and they get leaks, then they have to shut down the unit in an unexplained or an unplanned outage to go in, cool the boiler down, get in there scaffolding. Cause these boilers, we're not talking about Something small. We're talking about an eight-story

Kevin Kumpf (26:51) They're they're massive.

Aaron Crow (26:52) tall freaking boiler. So they got to build scaffolding inside of it to get to the boiler tube safely and be able to replace all the piping because it's all a closed system. Anyways, all this to say, and that was not critical path. That was not, that was somebody's spreadsheet that was sitting on SharePoint and an engineer, and I was working on it because it was having some connectivity problems and things, but that was not even part of the control system. But it was making decisions on Could the plant shut down out of band? And if it did, that's a million dollars a day if the plant's in peak system. So that's a big deal. And it probably wasn't and still today, probably isn't even on somebody's list of most important assets or devices or part of the process of what is the riskiest thing in my business? That'll shut it down.

Kevin Kumpf (27:42) And you know it's interesting because first off, I never thought that the day would come where Excel would be the most used cybersecurity tool. but it's true. it just reminded me to think of that, you know, that meme that you were talking about. And here's somebody with his arm around Excel and he's looking the other way and now here's AI. But what they don't realize

Aaron Crow (28:02) That's right.

Kevin Kumpf (28:05) is that all that data has to come from Excel for the history right now to figure out what your baseline is for the past ten years. But there's a second part to that. And this is what now, you know, what I've been, you know, what I'm doing now with Kasm stuff is I've said all these point products are useless because they don't tie together. So we've built an actually AI that runs on premise. And now when we're meeting with customers, it's like you've got 10, 20 years of data here. Historians, mom mez, all these other crazy systems, ERP, all these things. And the funny part about it is they're already doing some of this. if you walk through a plant, they'll have an extra motor somewhere. They'll have extra valves, they'll have extra things that they can pull off the shelf quickly. When you're talking about this boiler shutdown, it's like, do you even have the tubing? Do you have the skilled people that are available at that time? Because it's not like you can just go out and call, Bob's plumbing place and say, Hey, come in here. it's boilermaker stuff. You got to test this stuff. all these things. And so your shutdown window, may not align to somebody else's window. And my wife always says to her staff, poor planning on your part does not constitute an emergency on my part. Well, guess what? Now you're looking and you say, if I can take this data now and I can crunch it on premise, because data in the cloud is a whole other issue. But if I can crunch that data, now I can get meaning out of it. But guess what? You need the people that can understand that boiler data to make sense of it. Cause if you still don't know what the mean time between failure is or the deterministic rate of which something's gonna degrade, that data does nothing for you. And that's where AI is failing us right now is everybody's throwing their data to AI and going, Tell me what this means. And it's like, well, to AI it's like, Well, it's a bunch of numbers because it's it's you know, have to put context to it. Context is the

Aaron Crow (30:02) And

Kevin Kumpf (30:02) key.

Aaron Crow (30:03) and that's what it was for me supporting because I supported the because it I looked at it when we classified that boiler tube failure report as an application. So it was an application that I supported from a technology perspective, but it was just a freaking spreadsheet. So when I'm looking at the spreadsheet, I it's Greek to me. I have it's a bunch of numbers. I'm pretty damn good at math, but I have no idea what any of this stuff means. All I'm doing is when these engineers are saying, Hey, this isn't working, I'm trying to help them troubleshoot. But again, I don't know what good looks like because I could you could change a number and I'd be like, looks good to me. I and that's kind of what AI does now is you throw a bunch of numbers at it and AI's like, I don't know this. And you're like, if you don't know that what the egg what good looks like. So it reminds me of in engineering. So in my what freshman year of college, taking calculus, university level calculus in college. And they let us, we had like a TI, I think it was T I 92 graphing calculator. So it was about this freaking big, right? And you could that calculator, you could actually put calculus formulas and equations in it. So they could give you the equation on the test. And this was allowed with us, you know, my our professors are the ones that gave us these calculators. And they allowed us to put the calculate the formula in, the equation in. So they'd give us the question. They give it and we would have the calculator, and I would know that this question, the answer is I don't know, 42. But that didn't matter because I had to be able to show the work to be able to get it to there, right? So it's the same thing. if the AI shows you and says it's 42 and you don't know is that right or is it not right? You we need the human to be able like AI is a powerful tool, but we have to have the human that knows the knowledge. And the understanding to say, yes, that is right, or no. I don't know where the hell you made that shit up, or your math was wrong, or something. That's it's not forty two. It's seven hundred and forty eight thousand four hundred and twenty-three, right? It's couldn't be further apart. Like you said, I think we have fifty cabinets. Well, I'm pretty sure we have two hundred. Okay. You guys are way off from each other.

Kevin Kumpf (32:18) Exactly. And it's and it's the tribal knowledge, this human in the loop thing. And it's funny, when I was in college, they made us take calculus, which I'm really good at using tape measures and stuff on to the sixteenth, the thirty second, and the sixty-fourth. I can eye all that stuff, but they wouldn't let us use calculators on this one calculus test and everybody failed. And he hands them all back and everybody's like, I got a thirty, I got a forty, I got a twenty eight. And somebody raises their hand up and says, Will there be a curve? He goes, Well, let's let's graph that out. And so he goes, as X approaches this, and he goes to this 10 minute dissertation, and everybody's just looking at him. So I raise my hand and go, So is there going to be a curve? He goes, You can't curve a negative number, but of course I have to give you a curve. And it was just it was just that point of at that point I realized that a calculator is great, but check your work and know how to do it on your own. And now this data. So without the human in the loop or on the loop, this data is irrelevant. But last week or two weeks ago when I spoke in ICS Nashville, I did I'd had a presentation I'd held off on thinking people aren't going to find this important. And it was about data versus information. And so to your point, I even these supermarkets, the substations, the boilers, anything like that, they all collect this data. And now you're seeing the vendors out there, the big providers of the asset inventory, of the threat detection tools, everything else, they're all becoming smarter because they're taking your data and your customer or another customer's data and XYZ customer down the line, and they're pulling it all back to their cloud without anybody knowing about it. And all of a sudden, they're transposing that and working on that with AI, and now it becomes information. And nobody has any idea that your raw data, if I gave it if you had a file of social security numbers and somebody had a file of names and somebody else had a file of data bursts on separate environments. To you, it's just a bunch of numbers over here. From a sensitivity standpoint, there's no dashes or anything in it. Yeah, it's just digits. I got names over here. Great. I could have pulled them out of the phone book. But if you get that to AI, it's gonna come back and all of a sudden build information that is relevant and critical. So now I don't even have to go out and hack a customer. I just have to hack into one of those vendors that has now taken all that data, turned it into information, and weaponize it. And here's the other dirty little secret that people don't know is that once that data has had analytical processing to it, it becomes information. Your contract probably says you own your data. Great. Here's your numbers back, Aaron. Here's your names back. Here's your dates of birth back. Guess what? I can now sell this. As profit. And now I can go to other companies and say, hey, I can give you information on so and so if you want to buy it. And that is to me a wake up call where people are like, we've got all this information. So we have to take it down to the plant floor with the motors and stuff and show those people why it's important and why you have to have inventory. And then you got to take the to the people in the C suite and go, guys, do you know what's going out the door? Your business. And people at the end going, Are you sure this is true? I'm like, Really? Would you like me to stop by one day and show you what data's leaving your facility? I mean, five minutes. They don't know.

Aaron Crow (35:49) Well, and it's it goes back to, the person who said that, don't worry about the cloud, it's just the computer in somebody else's data center. It's just like okay. Technically you can be right. The difference is that when it's in the data center, I can go find the physical server that it's sitting on, who has access to that room, who has access to that server, and I know that. When the expanding problem with cloud. Is it goes into AWS? How many people have access to that cloud instance?

Kevin Kumpf (36:23) How many people trust Yeah,

Aaron Crow (36:23) AWS can't even tell you.

Kevin Kumpf (36:25) and how many people trust that they know what server it's on and it's not with other data on that platform? And it's like, guys, this is like saying, you know what, my accountant's got my taxes. That's wonderful. And he does them for me, but now you know what? Now he's gonna ship them out somewhere else to be stored, but don't worry about it, they're safe. It's like trust me.

Aaron Crow (36:44) Yeah. Trust me.

Kevin Kumpf (36:48) And this is the problem is that all these IT people now are coming into OT, which is awesome. But it's like they look at us like, man, you guys don't have your crap together because we knew this stuff and we know this, and you guys can't even tell us when something fails. Or you know, where's your asset inventory? Well, I got a spreadsheet over here. Great. That is now six months out of date. It was out of date the moment that you finished. basically creating it because other people have walked through the plant and done things in your cabinets. And we're sitting there going, trust us, we take safety as the prime factor here in our capabilities. And they're like, You haven't patched this since nineteen twenty.

Aaron Crow (37:33) Yeah, and that you know, one of my another one of my mentors, former boss, he always used to beat over my head. and I've carried that through and I'm you know, I say it like it's a negative. It was not a negative, but he would always say trust but verify. It's like, you know, I trust Kevin. Kevin's a nice guy. Kevin's never lied to me. I have no reason not to trust Kevin. I'm still gonna go verify the things that Kevin tells me. It's not because I don't trust Kevin. It's not because I think Kevin is a liar. It's because I still need to validate because it's my responsibility to confirm that the things that Kevin is telling me is true. Because it's my goal, it's my responsibility. It's my shit.

Kevin Kumpf (38:15) Well, yeah, and not only that, I've worked in I've worked in the, the power sector just like you have. But they're all run differently. Every single one of them, it's not like, you know, you can walk into, you know, a fast food restaurant and says, I've cooked the burger this way, and they say that's great, come do it here. It's every one of them has a different way they do things. So if I tell you to put the ketchup on the bun first and then put the patty on it here. Another place will be like, no, no, no, put it on top. It's it's trust. So you got to learn and say, okay, he gave me the basic knowledge, the building blocks. Now I need to go out and figure out how it applies to my situation. And the problem is that we've do not have enough practitioners that have done it. We a lot of people out there that just say, Hey, guess what? I saw somebody do it this way. I saw it on YouTube. And it goes. And I had a guy arguing with the other day going, I think VPNs are fine and I said, Based on what? He goes, Based on we use them and I haven't had an incident. And I was like, maybe in the IT world, but in OT they're not. He goes, Well, what's the difference? We still get there securely. And I was like, it's more physical device connectivity and things down at the, you know, L01 and 2 in the Purdue model. And I was like, okay, it's totally different. And it's like, Aaron, how long do you argue with somebody like this to say, you know, the spinal tap moment? Why not make 10 louder? But it goes to eleven. Why not make ten louder? But it goes to eleven. I'm like, well, why not make twelve? Because it goes to eleven. It's like eventually you sit there and go, Hey, you know what? They're they're they're giving away free booth basically pairs of socks over there. Go get a pair of socks and leave me alone. It's like I think.

Aaron Crow (39:58) unfortunately it's it's we continue to have this problem. we continue to have the lack of understanding. And I think part of it is because vendors have promised overpromised value from their products. Me being a former vendor and having a product, I get it. you have to sell your stuff. asset inventory is a great example. I love all the passive players out in the world. There, there's a great value for them. But most many, I won't say most, many, especially what, 10 years ago, when asset owners were buying these passive tools, it's kind of like before that it was the firewall. I have a firewall, I'm good. Now it's then it was I've got a passive monitoring product, I'm good. Like I've checked the box for OT security. Cause I have a firewall and I've got a passive monitor. I don't need anything else. It's gonna tell me everything I need to know. And we know that's just not the case. everything, it's it's gotta be. Yes. Should you have a firewall? Absolutely. Should you have a passive monitoring tool? Absolutely. Is that all you need to do? No.

Kevin Kumpf (41:06) Well, that's where I tell people to focus on the people process part now. When I go out and say, Okay, I found your asset. Who needs to get to that asset? What data does it provide? What process in your plant or facility does it provide? And how critical is it then this other asset here? And that's where now what you know, that's why this platform thing that, you know, we've put out there and people have come to us and said, my God, this is like Amazing. I was like, no, this is just IT tech and OT, but applied the right way. And it's you find your asset, you find the user or the system needs to connect to it. We now build the connective tissue in between here and we have a baseline of what look good looks like. And we do that for every single thing in your facility. And it's scary because people are like, Well, where do I get that information? You walk the floor, you ask the people that use it, what does that red light mean over there? I don't know. Well, who does? If not, why have the red light? We have too many people looking by go, wow, this is really cool here, but it's not cool, Aaron. It's it's life or death.

Aaron Crow (42:13) It is. Yeah. And, it's funny you say the IT stuff. Like when I first started doing this OT thing before it was called OT, I was rolling out enterprise IT products in an OT space. WhatsUp Gold? Splunk. I had Forescout counteract, not silent defense, not passive, active. We were doing SNMP community strings. We were doing, you know. Mac address, show MAC address tables and things like that from the switches. and then we were we were doing WSUS for patching using VMware. So like I had this full IT stack active directory. So I had a full active directory forest, independent, no trust relationships for each individual environment, dedicated firewalls, file, you know, secure file transfer server, secure remote access. So it was like a mini business. For each individual unit to segment them all away from each other. It was overkill-ish, if you think about now what you can do. but when you think about what I'm protecting and the risks of not doing that, I if the reason that we were that adamant about it is because if I had the same password going from you know one site to another, then if somebody steals my password, then they can just reuse it, right? So all of these things are pro like I don't want to have trust relationships because then I get in, and then I don't want to allow my laptop to connect into the control network. And I don't want to be providing patches. I don't want these Windows machines going to the internet to get updates. I want to provide those updates for them and I don't want them to get all the updates and all of those problems. But it become it came with that people process the technology side of things. Yes, I rolled out the technology and I did it in a smart way. And my first conversation was walking the plant and figuring out what those red buttons were and which devices I could scan and which devices I could not scan. and building the systems to match what the business built out and understanding what those baselines are. But dude, the funny thing is, is when the very first plant we rolled out this technology. And I've told this story. So if you've heard this before, guys, sorry, you're gonna hear it again. We rolled this out at a at a at a coal fired power plant in East Texas. And we rolled out all the monitoring and configured all the switch configurations and had Splunk and had all this stuff. We spin it up and it's first up and running. We got Splunk running, we got all this, counteract us running, all this kind of stuff. And we start looking at the logs of Splunk. This is day one, hour one of really having all the tools up. And there's this one device that's at the top of the list for Splunk, and it's just an alert. And it's just like hammering. It's like millions of events. And we look at it and it's a switch. It's one of the switches in the control room or electronics room. So we start looking at it and it was an error code and it was overheating. Huh? Okay, this switch is overheating. Well, they had redundant everything. Every switch was redundant. So they had dual path for everything. Every device had two paths, one to each switch. So complete redundant entire network is completely redundant. So we walk we go, okay. Well, where's the switch? So we start asking questions and we find out where the switch is. We go find the switch. It's in a cabinet. Sure enough, there's just a blinking red light. And then one of the operators is with me. I'm like, how long has that been like that? He's like, I don't know, months. I don't know. I don't remember it not being like that. So we look at it and it says overheating. It's in a cabinet, not really sure why. So I walk around to the back in the back side of the power of the cabinet, it's just Cisco switch, one use Cisco switch. It has two power supplies. One of the power supplies had a zip tie that was had fallen and was in the fan and was not allowing the fan to spin. So I removed the zip tie, the power supply fan spins up, the error code goes away in about 60 seconds, and the switch is actually online. So this whole long story, it's funny that I say it now, but they had a false sense of security because that switch was one of their core switches. And it was in a failed state, meaning they didn't know they had a problem because the primary switch was running and there was no issues. But if the primary switch had failed, the backup wasn't working and they the whole site would have gone down because of that one zip tie. And they didn't know about it because they weren't monitoring and they didn't know to look at the stupid blinking red light that's going on the fucking switch panel.

Kevin Kumpf (46:49) and it's funny because now people would walk by and go, Hey, where'd the red light go? it's true. And you know, what you've talked about building, we've now built in this platform that we're giving away in the community edition. Literally you can do everything there, but the problem is that it's like, you know, I saw a guy last week talking about, hey, let me teach you how to use OSINT. And he's talking about Nmap, and he goes, An Nmap you can use everywhere. And I was like, Should I just get that printed on a shirt? when I was at OT SecCon, it was or CyberSecCon as it's called now. There was a guy that had a shirt on called Human in the Loop. And I walked in and said, that is the coolest shirt here. And the guy I'm with goes, what's so cool about that shirt? And I was like, I can tell you come from IT.

Aaron Crow (47:33) Ha

Kevin Kumpf (47:34) Go sit down for a moment and let me explain it to you. but this is the part that people don't get. Is you have to understand what tools can be used, where, when, and how. And I'll never forget one of my first jobs. I was working for a consulting firm. We were doing things for GM Hyde Clark and all their industrial cleanup and stuff. And you had to go down and do every single down to the roll of paper towel you used. And worked with this guy that was always cold, and underneath his desk, he had a space heater. And right next to the space heater was basically this server. And I said, What is that? He goes, he goes, That's all our data backups. And I was like, Where else are they? He goes, nowhere. He goes, but don't worry. He goes, it's got he goes, it's got full rate on there. We're good. And I was like, What's near a heater? He goes, Yeah, he goes, it keeps me warm. He goes, and it keeps the unit warm. And I was like, Okay, that's that's really a twisted sense of humor, but okay. That weekend coming up two weeks later, he says, Hey. the servers are gonna be down this weekend. I'm basically gonna add more drives to it. He turned the things off. They never came back up because they had been so hot that when they went down, everything was like cool. And all of a sudden it just drives ceased. All of a sudden, everybody gets a call on Monday morning saying, we're gonna have to terminate your contract because we all the data we had for these contracts and all the billing and stuff is now gone. And this guy's gone, and everybody that's working with him is gone because We don't have any way to get money for this project. And I was like, all because you're stupid. You don't look and say that needs airflow. Or my favorite is the fan is noisy. Let's put tape over the back of it so it doesn't make any noise. Or the guys that are on the production line with the air nozzles where that fan has basically got the little dust and dirt, you know, monkeys hanging out the back of it, and they just take that. Air compressor, which now has moisture coming out of it because it's not basically an air-dried moisture to go pfft and they spray the fans out. And I was like, How many minutes are gonna that last for? In fact, I built a Halloween decoration that I can send you a picture of. It's a goalie light outside for a gravestone I named off the post, which is a goal hockey thing. And I put conformal silicone all up and down the LEDs and stuff, because I'm like, I want this to last a couple years. It even says it's for. Damp locations. It doesn't mean that it's gonna sit out in a rainstorm, but people are like, yeah, it's it's it's it meets this standard here, so I'm good. This IP whatever, and that's what they do. And so we're not just cybersecurity people, we are actually people that understand the physical items and how they need to be protected in relation to the humans, and many times, Aaron. We're protecting the device from the human. In other instances, we're protecting the human from the device. As you said, you could do that on a ladder. But if you sit on a s a stool, it's a little different. It both can get you there. Which one's safer?

Aaron Crow (50:47) Yep. Absolutely. Yeah. And it's it's never ending, right? And there's there's so many of these. So tell everybody, go into a little bit about what the Kasm is, what you guys are what the platform is, what people can do with it, what the use case is, and how people can get learn knowledge about it and start using it.

Kevin Kumpf (51:08) So, what we've done is about a year ago, I started working with Michael Farnum and said, we need a platform for OT. And I had previously built secure access and things like that for Cyolo and my history and understood what it was. Well, the benefit of the platform is it gives you, and you can go literally out to the Kasm website, kasm.com, and there's try this right on the page, and it will bring you up in an ephemeral workspace. So now you have zero persistence. With full DLP controls in it, with full SRA, with a full AI container, recording. But the other part of it is that it allows you for patch management. It has an incident command center, OSINT tools, all these things in the platform. And so we've put it together with Elasticsearch. We've put it together with Exceed, and there's the free Open ZD on there if you want to try that, if you want the unmanaged. And then we put Sakara's Labshock on it. And it's built on a steel dome operating system. So literally now you can spin up an entire platform and start testing, like we talked about the refrigeration system. You can literally use Labshock, go into a workspace that's already in the dashboard, launch that, build an Excel or build an Elasticsearch basically framework that says here's a dashboard I want to see, and start pumping temperatures to it. Start pumping valve controls to it. And now this is what would have saved this the refrigeration industry. And that supermarkets. So now in April, we're already talking about a day at you know the conference where people were going to be able to come in and say, hey, bring a web browser, because that's all you need. Connect to the server and you do things in a browser. And we even do malware detonation in it because forensically it's ephemeral, which means it's like your giant sandbox that you can test things, you can destroy things, you can screw things up without impacting your plant floor. And this is why it's important, Aaron, is because you now have a safe space to do stupid things, as I tell people. And believe me, I've done enough of those in my lifetime where it's like, well, you what? If I could have done that without breaking my dad's window or crashing the car or flattening the tire, I would have done that. But we didn't have the virtual world then.

Aaron Crow (53:23) That's true. Yep. I'm I'm I've got a 17 year old. He's my oldest. And I'm teaching him, changing oil and cars and things like that, right? And so he's done it multiple times. So he changed the oil in my wife's car. And she'd left the other day and I went out in the driveway and there was an oil spot in the driveway. And I'm like, Well, that's not good. Why is that there? and He's like, well, I don't know. Maybe I just spilled some oil when I changed the oil. I'm like, okay. So she comes back. and I'm like, go out, pull your mom's car back up on the ramps, and let's take a look. He's like, no, everything looks fine. I don't see any oil leaking. I'm like, oil's leaking. We got to figure out where it's coming from. Well, the oil drain plug going into the oil pan, he had just hand tightened. Because I had told him to only hand tighten the oil filter. He thought that also meant I should only oil hand tighten the oil bolt drain plug. so it was just it was on there, but there was oil everywhere. And it and it was a lesson learned. I didn't beat him up about it. I wasn't like, what the hell is wrong with you? What better way to learn? Luckily, there was no, big damage that could have happened because obviously oil goes out of an engine. That's a bad day. But that's the best way that he can learn is we use that as a lesson learned of this is why you don't do this. You have to check and make sure nothing's going on. Now imagine what would have happened if all the oil drained out of the out of the out of the engine and she was driving on the road at 70 miles an hour. maybe they have a sensor that's gonna notify her before that happens. Maybe not. I'm not sure. And I don't want to test it.

Kevin Kumpf (55:10) well I can I can affirmatively say they have a sensor, but people don't look at it and you ruin the engine because my friend back in high school decided to do his dad a favor, change the oil, drained the pan, poured five new quarts in, never put the plug in. His dad started to drive across the city, got on the throughway, and then seized the engine. because he had a gravel driveway that all drained into the gravel, so you didn't see it running all over the concrete or the asphalt. So I can affirmatively say that my friend seized his dad's engine because he was doing him a favor, put the oil filter back on, poured the quarts in, never put the plug in, and that result does happen. So your son got

Aaron Crow (55:50) And that goes back to people, process and technology, right?

Kevin Kumpf (55:55) goes back to the point of if you can't get the oil filter off, yeah, it's great to put a screwdriver through it and try to spin it off until you shred the oil filter and then you've got that fitting that you still can't get off. So and you can't take a screwdriver and start hammering that because now you nick the edges of that. But hey, those are more lessons learned, just like a hydraulic line near somebody's leg.

Aaron Crow (56:17) Yep. Exactly.

Kevin Kumpf (56:18) So well

Aaron Crow (56:22) Well, awesome dude. So where can people find you? Where are you gonna be? Speaking, conferences, all the things.

Kevin Kumpf (56:29) Actually, we're going to AFAR because we've built this platform now for Barry-Wehmiller and they're labeling it their DG three sixty platform. And the nice thing is that we can now take an entire facility, pull in every asset, tie every user down to an application, a device, and help them start parsing their data. And security doesn't sell as Aaron. But what now sells is if you can do operational efficiency mean time between failure and you can extract it out of the same platform.

Aaron Crow (56:58) Hundred percent. Yep. Every time.

Kevin Kumpf (56:58) That's the money right there. Cause now you can sell it through everywhere else.

Aaron Crow (57:02) Yep. And the I don't want to take too much time because we should just do another one of these because I think you and I could probably talk on these for ten hours and I don't want to bore everybody. But that was the thing that I used when I was pushing this in power plants. It was never the cybersecurity conversation. I was pushing availability, you know, visibility into their systems, troubleshooting, All of those, reliability. I was never telling them cybersecurity 'cause anytime I brought that term up, they would just roll their eyes. whatever. I don't care about that crap. Yep.

Kevin Kumpf (57:34) Yep, it's process improvement, it's performance, it's uptime, it's analytics. It's giving you a competitive advantage, not just and then doing it securely. That's all it is. Competitive advantage done securely.

Aaron Crow (57:46) And if you do those things, you're gonna get all the cyber tools and data that you want out the back end, but it's not the thing the asset owner cares about. Nobody signed up saying, I wish I had a firewall. No, they have to have a firewall. It's like having insurance. You don't want insurance, you just have to freaking have it. Well, awesome, man.

Kevin Kumpf (58:02) Yep. Yeah, 'cause then if you don't put Yep. Thanks, Aaron.

Aaron Crow (58:06) Dude, thanks for wearing the AI socks today. I think it definitely helped our conversation.

Kevin Kumpf (58:11) I Hey, you know what? It's it's my attire for the day. You know?

Aaron Crow (58:15) All right, man. Let's do this again. And I'm sure we'll cross paths many, many, many more times as this is such a small environment. So definitely everybody reach out. We'll have all Kevin's details in the show notes. definitely check out Kasm and all the things they're doing over there. check out Kevin, go hear one of his talks, ask questions. That's what we're here for. So thanks for taking time today, buddy. I appreciate it. And until next time.

Kevin Kumpf (58:41) Thank you, Aaron. You as well.

Transcript lightly edited for readability.