Bridging the Security Gap: How HERA Transforms Remote Access in Industrial Environments with Andrew Ginter

Aaron Crow (0:3.804): Thank you for joining me, Andrew, second time, at least on the, on the podcast. So I appreciate the opportunity. I love our conversation. So why don't you, for those that don't know you, which I don't know how they don't, if they're listening to this, but why don't you introduce yourself, and, and who you are, your background, all that kind of good

Andrew Ginter (WF) (0:17.298): Sure thing. Hello, Aaron, and thank you for having me. I'm Andrew Ginter, VP Industrial Security with Waterfall Security Solutions. It's a fancy, goofy title. What does it mean? I lead a small team of subject matter experts. are charged with, mean, Waterfall is a technology vendor. We produce unidirectional gateways and related technologies. And we've got a whole, a large team of people who are experts on the

Andrew Ginter (WF) (0:44.154): If you need to know which button to press in this really strange circumstance, know, they're the gurus on that. My team is charged with sort of a different mission, which is understand the world of industrial cybersecurity, understand standards and regulations, understand the guidance that's coming out and, you know, position the waterfall solutions within the bigger picture. So, you know, we're tasked with understanding the threat environment. We have an annual threat report. That's what my team does is sort of the big picture.

Aaron Crow (1:16.764): That's awesome. It's a, it's a needed environment because we're constantly having to look and pivot and our attackers are constantly looking, regulations are changing. yeah, it's, it's a, it's important and even fun place to be

Andrew Ginter (WF) (1:29.102): It's, I digress. We're going to talk about something different, but.

Andrew Ginter (WF) (1:34.564): It has been fun lately. I don't know if you're tracking it, but I strongly recommend to you and your listeners, the Cyber Informed Engineering Initiative out of Idaho National Laboratory. I'm an understated kind of guy. I don't use the word excited very often. The one thing I'm excited about in the last two years is that. think finally, when I explained CIE, so maybe that's a topic for a different podcast, or I can introduce you to the leaders of the CIE initiative. When I explained CIE to, know, owners and operators in the field,

Andrew Ginter (WF) (2:3.154): the feedback I get and I'm paraphrasing, but it's roughly, what a good idea. Why is this new? This shouldn't be new. Why have we not been looking at the problem this way for the last 20 years? It's like, you know, it's all the same puzzle pieces, but where we used to put them together this way with all sorts of space between them, now they fit and there's no space between them anymore. So yeah, it's been an exciting time the last two years and you

Andrew Ginter (WF) (2:31.314): things are changing surprisingly in a field that's 20 years old.

Aaron Crow (2:39.412): And in the right direction, right? We may be lagging a little bit behind, but I actually went to Idaho national labs and took their, their CIE training and actually going to be doing a talk around CIE at DEF CON for ICS village. kind of talking about how do we use it in critical infrastructure? What does it mean? know, cause it's really easy, know, cyber informed engineering. It's really easy to do a green field, easy being relative.

Aaron Crow (3:5.876): in a greenfield environment, but how do I use those those same things in and you know, existing space, legacy equipment, all that kind of stuff we see. It's just as relevant in those spaces as it is in greenfield, but many when they're looking at it, they think, well, I'm only I can only do this when I build new power plant. My old stuff doesn't work. I'll just have to accept what I have and hope for the best next time, right? And that's just not the

Andrew Ginter (WF) (3:25.702): Not true, not true. It's all about risk, it's all about consequence, but again, different conversation. I look forward to seeing what you have to say at DEF CON.

Aaron Crow (3:38.438): Yeah, absolutely. Well, let's dive into it. So secure remote access is something that that I have been working with and dealing with and fighting, especially in OT for a long time, right? We've done such a great job over over my career and seeing the difference of, know, in the beginning it was just putting in a firewall. When I first started, there was no segmentation from OT to IT, so we put in these firewalls, but then we need to have remote access and you've got vendors that have 3G cards and they can directly get in because I need to monitor and.

Aaron Crow (4:8.146): there's just all of these problems, we've seen multiple instances of vulnerabilities and changes, not even just cyber ones, right? But a vendor logs in and makes a change that I didn't know about. And now it impacted my system from an operational perspective, beyond just a cyber attack or a nation state or all these types of things. talk a little bit about, obviously Waterfall has a new product coming out, but really just high level, the remote access solute problem.

Aaron Crow (4:36.234): in OT and why it's such a big one that we haven't solved

Andrew Ginter (WF) (4:38.034): Absolutely. So where to begin? In the OT space, there is a spectrum of, let's call it consequence. And a lot of people don't understand this. So side track again, very briefly to 62443. A lot of people read the standard and say, I must do this, I should do that to be compliant with the standard. So I do all that. And I sit there, I'm good. And I have to ask, who are you? I'm a large power plant. And you did.

Andrew Ginter (WF) (5:6.392): everything the standard says you have to do. Yes, I did. You I didn't do everything I could do. I did everything I have to do. You realize that the standard applies equally to small shoe factories as it does large power plants. Yeah, I didn't really think about that. So you've just implemented the minimum that a small shoe factory needs. Because you did the minimum. and so people, you know, there's this spectrum.

Andrew Ginter (WF) (5:33.870): And it's mostly a spectrum of consequence. If you breach a large power plant, the consequence is different than if you breach a small shoe factory. And when we talk about remote access, it's in the of consequence. so the most consequential industrial enterprises on the planet, extreme example, nuclear generators, in my estimation, will never deploy remote access, any kind of remote access.

Andrew Ginter (WF) (6:3.868): Well, partly because they have extremely important information to protect and they can't leak that information out to the world. Partly because the other way around, these are militarily strategic targets, every one of them. If the military comes after them and they've got remote access enabled, what's to stop a spy from breaking into my house, putting a gun to my head and saying, log

Andrew Ginter (WF) (6:33.860): use your two factor, do it now. Type this. You know, I'm sorry, I'm gonna log in. So the most consequential are saying nuts to remote access ever. I don't care what kind of remote access you have. know, the least consequential are saying I've got, know, I've sprinkled a little VPN, I've sprinkled a little firewall, I've sprinkled a little antivirus. I'm good remote, you know. And in between is sort

Andrew Ginter (WF) (7:3.442): where it gets complicated. So, you know, we have been talking to a lot of owners and operators who are on the high end of the consequence spectrum, not the extreme end, but the high end. And they're looking at the remote access solutions and they're saying, and I'm not gonna name names. I mean, I could give you concrete examples. This vendor's firewall, 20 ,000 of them were breached. You know, that vendor's VPN.

Andrew Ginter (WF) (7:32.722): But in a sense, that serves no purpose. You name and shame, and you ignore the fact that all the other vendors have the same problem. So what I'll say is that people are looking at their remote access systems and saying, this vendor's VPN product had a zero day exploited and 20 ,000 of them by a nation state. The other one had an end day exploited in all of the instances that

Andrew Ginter (WF) (8:1.500): didn't install the patch within a week and a half. And so they're saying, if you breach the firewall, if you breach the VPN server, if you breach the remote access server through a vulnerability, through stolen credentials, if you didn't do two factor, through forged credentials, if your two factor is weak, if you get in any of those ways, it's a very bad thing. It's close to game over because the enemy, the adversary now has

Andrew Ginter (WF) (8:29.956): a foothold on the edge of your network and can launch whatever attacks they want deeper into the network. And the buzzword is pivot. Once they get a foothold in this, and why is the remote access server, why is that different? It's because it's exposed to the internet. Very little else on your OT network ought to be exposed to the internet, that, your remote access server by definition.

Andrew Ginter (WF) (8:57.636): is reachable from the internet because you're on the hotel network and you need to get to it. And so it's vulnerable in a way that you can't afford the rest of your system being vulnerable. And if you're breached, the bad guys pivot. They use the compromised systems to attack deeper. And this is what our customers are saying. We really need remote access. We're using remote access. We're not happy with it. We wish there was something

Andrew Ginter (WF) (9:25.330): And again, the small shoe factories don't, but a large automobile plant that produces a quarter billion dollars worth of automobiles per day, or a large power plant or a refinery, these sort of consequential outfits. In a large refinery, you've got 300 people walking through the front gate every day. And a large fraction of that coming in remotely as well. It's a huge environment, and it's a very sensitive environment.

Andrew Ginter (WF) (9:55.224): People are saying, can't we get something

Aaron Crow (10:2.480): Yeah, and a lot of times people aren't thinking about remote access the way that you and I do. I know I've fought this battle many times is the manager wants to be able to access the environment from his from his office that's across the street. It's in a different building instead of having to walk over to the factory floor or to the control room, etc. That's remote access. Well, I'm on site, so it's not the same, but it is like if you can access it from there, which is outside the protected network, it's north of the OTE environment.

Aaron Crow (10:31.716): north of the firewall, it's exposed. That means that I can do it from virtually anywhere. I don't have to be sitting in that chair. Now, obviously I can restrict some of that on on on, but most of the time in my experience that I've seen those, there's no more restriction whether I'm sitting at my office or I'm at Starbucks or I'm in China, right? There's really not much difference between any of those because it is all remote and I don't have to

Andrew Ginter (WF) (10:52.444): And you can try to restrict it. But again, if we're dealing with a software artifact, I'm sorry, all software has defects. Some defects are vulnerabilities in practice. These vulnerabilities, some of them we've discovered, the good guys have discovered and are madly trying to fix before the enemy gets to them. Some of them the adversaries discovered and is exploiting without us knowing. Worse, it's software. It can be misconfigured accidentally.

Andrew Ginter (WF) (11:20.594): It can be misconfigured deliberately. There's issues. so again, small shoe factory doesn't worry too much about it. They've got insurance. If they go down for 10 days, they restore from backup and off they go. High -speed passengers, rail switching, if it is compromised in the worst possible way, trains collide. You can't buy insurance for a mass casualty event. It's unacceptable.

Andrew Ginter (WF) (11:50.266): And so, still, know, mines are extremely safety conscious and there's no mine in my backyard. They're out in the middle of nowhere. And you know, do you want to fly people out to the middle of nowhere? You know, sometimes a two and a half day trip to make a small change. No, you need remote access into these safety critical sites. You mess with the airflow and people die underground. It's just, so, you know, there's a need out

Andrew Ginter (WF) (12:20.060): for something stronger than software exposed to the

Aaron Crow (12:29.274): Well, and today's pretty relevant. We've seen in the news today a lot of outages across our critical infrastructure from a software product, right? And we don't even have to name them. Everybody that's listening probably knows it's brought almost all the airlines down. Banks are offline. I logged in my bank account this morning and they're like, yeah, you can see your account, but you can't really do anything with it because all of our systems are having an outage. And it was because of software that was installed.

Aaron Crow (12:55.494): I'll just go and say it was the crowdstrike issue and that was not a cyber attack. That was because a, a patch was, was, was an issue and it got pushed out and they re removed it eventually, but it had already had this large impact. Right? So it wasn't even a nation state. wasn't a bad actor. It was just a mistake. And that shows, especially in OT, how critical you can't have critical systems, even a mistake, even if it's accidental, that that's, that's unacceptable in all of these spaces. Lives can be

Andrew Ginter (WF) (13:21.926): That's right. CrowdSight was, know, errors and omissions. It was not an attack. There have been cloud -based attacks. In the modern day, they're not called cloud -based attacks. They're called supply chain. There was SolarWinds with the headline recently, a couple of years ago. Before that, the first big one was not Petya. took hundreds of victims down simultaneously. And, you know, this is why when people talk about

Andrew Ginter (WF) (13:50.428): cloud systems or supply chain, this is why it's such a big deal. If there is an opportunity, mean, concrete example, a couple of years ago, the war in the Ukraine, the Russian invasion of the Ukraine, the front line of the battle was moving back and forth. When it moved towards Russia, apparently the Ukrainian farmers got it into their head to drive their million dollar John Deere tractors into the battle zone and drag dead Russian tanks out and sell them for scrap.

Andrew Ginter (WF) (14:20.370): This annoyed the Russians. So when the battle line moved back the other way, they stole a bunch of million dollar John Deere tractors and drove them 300, 700 kilometers into Russia. The Ukrainian farmers were naturally annoyed by this, called up John Deere and said, the Russians had just stolen our tractors. What are you gonna do about it? And John Deere said, oh, we'll turn them off. And they turned off the tractors and they don't work anymore. They're dead tractors.

Andrew Ginter (WF) (14:48.112): And you know, the good guys said, yay. And then they said, just a minute. What just happened? What if John Deere gets it into their head to turn off all the tractors in Europe at planting time? What if Russia gets it into their head to break into John Deere and turn off all the tractors? This is the supply chain problem. I think of it as a cloud problem. Any time where we have an important asset connected to the cloud.

Andrew Ginter (WF) (15:15.334): were vulnerable in new ways that a lot of people aren't considering. Now, I don't know about a lot of remote access systems and where the remote access is the topic here. I don't know that they're connected to the cloud, but if they are, there are cloud -based remote access. mean, you can do remote access through Teams if you want, and that's a cloud -based system. So anytime we're talking cloud -based, again, I'm not saying don't use it. I'm saying consider where you live.

Aaron Crow (15:37.138): Yep. Yep.

Andrew Ginter (WF) (15:43.758): in the spectrum of consequence. On the low end of the spectrum, know, go for it. Provided, you you've got a little bit of protection in place, provided your insurer is okay with it. On the high end of the spectrum, you can't buy insurance for mass casualty events, you can't buy insurance for threats to national security because critical infrastructure is failed. You can't do that. You need something else. And you need to be looking really hard at whether any kind of cloud connection is wise.

Andrew Ginter (WF) (16:13.026): And this is, if you're curious, my latest book, Engineering Grade OT Security has a chapter on network engineering, trying to provide some guidance on where in the spectrum, what kind of protections are appropriate.

Aaron Crow (16:33.958): Yeah, it leads right into what we were just talking about with CIE and really understanding the risks that you're doing, know, secure mode access, remote access, cloud -based, you you just need to understand you don't have to put platinum level, the most, you know, physical removing. Not every site is a nuclear facility, right? So like the shoe factory doesn't need to have the same requirements around it that the nuclear facility does.

Aaron Crow (16:58.290): That doesn't mean that it should just haphazardly say, well, we're only a shoe factory. So just whatever you want to do, right? It's just understanding what those risks are and being careful that you understand it. so talk a bit about what is the difference between the product that you guys have now and, you know, a software based, remote access

Andrew Ginter (WF) (17:13.034): So we've just announced hardware enforced remote access, HERA for short. And this is something that fits sort of in the spectrum of security. fits between the high end of software based remote access and the low end of hardware enforced remote access. What is that? You know,

Andrew Ginter (WF) (17:35.056): Waterfall for years has been selling unidirectional gateways, hardware that sends one way out and the other back. We have a software product that goes with the gateway that sends screen images out. So you can see the engineering workstation. The Microsoft support tech, the GE support tech can see the engineering workstation screen coming out through the one -way hardware, but nothing gets back. No keystrokes, no mouse, no nothing. So you need someone on the inside on the phone working with the remote tech. But you know,

Andrew Ginter (WF) (18:4.466): So that's extremely secure. Nothing gets back in. Doesn't matter if you've exploited a vulnerability, the hardware you can't get back through. But it's inconvenient because you need someone on the... And so customers have been asking us, look, we have scenarios where we need something stronger. We're higher on the consequence, high enough on the consequence spectrum that we need more than software. But you haven't got anything for us. All you've got for us is remote screen view, which works well when you have a comparatively small number of remote access sessions coming

Andrew Ginter (WF) (18:34.278): What if I have a manufacturing plant with a thousand machines in it and a hundred vendors and, know, at any given moment, three or four of my machines, my machines are out. Half of them, the technicians can figure out on their own. The other half, they have to call the vendor who's going to remote into the machine and help the technician figure out which piece needs, you know, repair. I've got constant remote access into my systems, a different three vendors into a different three machines every half hour. You know, remote screening doesn't work.

Andrew Ginter (WF) (19:3.960): software is all I've got, can you give me something stronger? And so we have this new product called Hera and under the hood, yes, it's remote screen view. It's a unidirectional gateway out so that the remote user can see the screen and we're letting keystrokes in. so the keystrokes come in not through a firewall where it's software, if you breach it all bets are off. They're coming in through an inbound gateway with

Andrew Ginter (WF) (19:33.490): hardware filtering.

Aaron Crow (19:39.954): What does that mean? Talk, talk through what is that hardware filtering look

Andrew Ginter (WF) (19:41.646): So two or three things. One is let's talk about the screen images first. When I say screen images are coming out, it's not TCP coming out. It's screen images being sent through the hardware in our own proprietary thing. And on the outside, they show up as a video on a web server that the remote support person can talk to. So there's no way to open a TCP connection even through the outbound to the internet.

Andrew Ginter (WF) (20:9.496): from the OT. It doesn't work that way. All we're doing is fetching screen images and sending out. Similarly, on the way in, it's not a TCP connection. We're not forwarding TCP packets. It's not a router. What we have is an inbound gateway, and the inbound gateway can only send stuff one way. It can't send anything the other way. That's the nature of the gateway. And in the gateway, there is what's called a Gatorade chip. Think of it as a poor man's custom AC.

Andrew Ginter (WF) (20:38.084): It's a million transistors that can be configured as needed. So we've configured them in such a way that what comes in is encrypted keystrokes and mouse movements. So, you know, let me back up. What does Hera feel like when you use it? And let's follow the connection path all the way through, if I may. A remote user, I'm sitting on my laptop at a conference and I've just made a change to the power plant network.

Andrew Ginter (WF) (21:7.804): two days ago and I need to log in and see if it's still working, see if there's anything. So I'm sitting in my hotel room in the evening, I connect to the hotel wifi, I'm on the internet and I fire up Hera. What does it do? It gives me a little screen, says which power plant do you wanna log into? I've got seven configured, so I pick one. And what it does is opens a standard TLS connection. Now, let me say the Hera gateway, the appliance, has got four CPUs in

Andrew Ginter (WF) (21:37.990): The outbound component has two CPUs, a source and a destination, and the strange one -way hardware. The inbound has two CPUs and the strange one -way hardware. Two of the CPUs are internet exposed, and two of the CPUs are OT exposed. So my laptop connects to both of the internet exposed CPUs, one to get the screen images and the other one to send my keystroke and mouse in. And it's a standard TLS connection.

Andrew Ginter (WF) (22:5.850): There's a trusted platform module, TPM hardware has to be on my laptop, but all modern laptops have this. And it negotiates and I wind up with a encrypted socket. And now it challenges me for a username and password. It challenges me for the encryption credentials that are in the TPM. I can only make this connection if I'm on the right laptop, because all that stuff is buried in the TPM hardware.

Andrew Ginter (WF) (22:34.374): So I authenticate and now I see an image of a virtual machine that's been created for me. I'm on the OT network. What do I wanna do? I wanna start moving the mouse. I move the mouse. The first thing I do, the mouse movement information, a nugget of it, is encrypted using a different key in the TPM. Okay, so there's two keys, one for the TLS session and one for the mouse movements.

Andrew Ginter (WF) (23:1.550): And the beauty of the TPM is you can't steal the keys without, you know, physically taking the thing apart and putting it on an electron microscope and nasty stuff like this. The TPMs are designed to hide key information. now, but I'm using the key information, I'm sending encrypted keystrokes through the TLS connection into the inbound gateway. Okay, it gets to the CPU in the inbound gateway. And now what? Well, the CPU decrypts the TLS.

Andrew Ginter (WF) (23:30.030): It has its TPM has the credentials to decrypt the TLS. And now it sees the encrypted mouse and keystroke, you know, information. Well, it can't do anything with that because it's encrypted. The internet exposed CPU does not have the key. So it sends that encrypted information through the one way hardware into the inside where it has a TPM. It's decrypted and sent to the virtual machine. And I see my mouse move. Clever.

Andrew Ginter (WF) (23:59.408): What's happening in the hardware? The hardware is one way in and that extra chip I talked about, that million transistor, you know, gate array is looking at every chunk of information. It's not even a message. not, you know, it's not TCP. Every chunk of information coming through the hardware and saying, the only thing I'm allowing in is encrypted keystrokes and encrypted mouse movements. That's all that's allowed in. I can't send even if, you know, even if I compromise both.

Andrew Ginter (WF) (24:29.584): of the Internet exposed CPUs, I can't send anything through that hardware. I can't send a TCP request. I can't send a UDP request. The only thing I can send through there is encrypted mouse movements. And that's the only way anything gets into the system. You can send what you want out of the system as long as that's all that's getting in. You cannot pivot and attack through the Internet exposed CPUs no matter

Andrew Ginter (WF) (24:57.020): how many zero days you take advantage of. So that's the claim to fame here is the encryption happens on the inside so that nothing on the outside can forge those keystroke and mouse movements.

Aaron Crow (25:15.024): Yeah, that's incredible. Obviously it's very built upon the idea and the engineering behind your gateways, right? So having that dual channel, only stuff goes out and the only thing that can come in is hardware, right? Where a lot of remote access solutions are doing that through software, they're saying, hey, I'm not going to, I'm going to block like a firewall, I'm going to block RDP or I'm going to block file transfer. I'm going to block these other protocols. I don't want to let those things through.

Aaron Crow (25:42.844): But as you know, with a zero day or anything else, there are possible ways to do that, right? It's better than nothing, but it's not, it's not, it's not full -proof, right? I can, I can take advantage and I could potentially pivot because I can, you know, obviously next gen, you know, the next gen firewalls have application aware protocols and they're looking at that, but not everybody even deploys those things. So a lot of times in these spaces, we have old, you know, stateful IP, you know, firewalls that are just, you

Aaron Crow (26:10.790): this port and protocol this destination and I can send a whatever packet down this as long as it's the right destination and it's using the right port number because it's not just getting that

Andrew Ginter (WF) (26:19.868): That's right. you again, classic software technology, you breach, you exploit either a zero day or even a known vulnerability that you just haven't had opportunity to patch yet. You breach that and you're in control of that CPU. can, principle, it's varying levels of difficulty, but in principle, send any message you want into the OT network. Here, if we breach the outbound CPU,

Andrew Ginter (WF) (26:49.094): does us very little good because nothing gets back in through the outbound hardware. If we breach the inbound CPU, the hardware is going to filter out any kind of message we try to send through that's not the encrypted keystroke and mouse information. And you might say, can't I forge the encrypted keystroke and mouse information? Well,

Andrew Ginter (WF) (27:12.762): No, it's not just encrypted, it's authenticated as well using the key information in the client. And so even if I breach that CPU, it doesn't have the key information either to decrypt or to encrypt and create forged inputs. we see sort of the, what's right, the opportunity.

Andrew Ginter (WF) (27:38.246): the sort of the low -hanging fruit in terms of who's going to use this first are organizations that have, that are higher on the consequence scale and who have a need for a lot of remote access. So I gave the example of manufacturing. Large manufacturing plants all have this problem. It doesn't matter what they're manufacturing. Another opportunity that I see is wind farms.

Andrew Ginter (WF) (28:5.963): Why wind farms? Well, think, you know, old school, a gas -fired power plant. What is it? Physically, it's a stationary jet engine that you feed natural gas into. The gas turns the engine. The engine's connected to a generator, turns the generator, produces power. All moving parts. Friction is the enemy of moving parts. You know, where...

Andrew Ginter (WF) (28:32.420): is what friction does to you, vibration is the symptom of wear in these rotating equipments. And so a couple of times, once or twice a quarter, every one of these massive turbines needs to be adjusted because they're connected. We recommend through a unidirectional gateway out to the internet, the vendor, the turbine vendor can see the vibration information in real time and can say, something's building up here.

Andrew Ginter (WF) (29:0.850): We want to maximize the life of the turbine between services. And so they call up the site and say, I need to service this. And they do it by remote screen view. And so once or twice a quarter. So you have once or twice a quarter, you've got a remote session with the vendor on each of your six gas turbines at the power plant. It's a session per week. In a wind farm, we've got 300 turbines.

Andrew Ginter (WF) (29:28.880): and they all wear out at roughly the same rate. They all need adjustment once or twice a quarter. Worse, the wind farm turbines, bluntly, no offense to the vendors, it's not as mature as the gas turbines. Gas turbines have been with us for a very long time. Wind turbines, less so. They're becoming more mature because they're being deployed so widely, but you might not have one vendor that needs to remote in once or twice a quarter for each turbine. You might have two or three vendors.

Andrew Ginter (WF) (29:59.058): because it's not quite as mature all in one. And now you've got, let's say, two vendors remoting, let's say, once, let's just say once a quarter, two vendors once a quarter into each of 300 turbines. That's 600 remote sessions in the quarter. There's only 90 days in the quarter. Someone has logged into these turbines all day long. Again, we've had wind farms come to us and say, Andrew, can you give us something here

Andrew Ginter (WF) (30:26.800): remote access is a nightmare for us. There's too many people in here. We're too exposed this way. And of course, I gave the example of mines, high on the consequence scale and intrinsically remote. And it's just expensive to pay people to sit there moving the mouse. So they're looking for something stronger.

Aaron Crow (30:55.142): Well, and this problem is it's all throughout OT, right? To your point, right? Those teams that are monitoring and they're managing these things, that team sits in Atlanta or they're sitting in Germany or they're sitting in Japan. You can't afford, none of these organizations can afford to have those people get on a plane and come here and do the required maintenance in person. Their staff doesn't have the ability or knowledge to do it either. So they're between a rock and a hard place. They have to have remote access.

Aaron Crow (31:24.444): They have to have this ability or they're going to cause damage to their equipment. They're not. They're going to have to spin it down so they can't run it as efficiently. They're losing money. All of these things are consequences. And up until now, they've had a they've just had to kind of close their eyes and hope for the best, right? Well, let's just get them directly into our environment and just hope nothing bad happens, right? And unfortunately, as we see with all of these different attacks, that's just not enough anymore, especially in critical infrastructure. Any.

Aaron Crow (31:53.020): But even if it's not critical, you talked about manufacturing, it's critical to them. Like I was at a plant yesterday, a manufacturing facility. And if that plant goes down for two days, they're probably out of business, right? Plain simple. they're a startup, they've been open for nine months, they've got four out of their eight lines running. But if they go down, they're probably out of business. I don't know that for sure. I'm just assuming, right?

Aaron Crow (32:16.740): is anytime you're in a startup, that amount of capital that you're no longer doing and you've got all these people staying around and all that kind of stuff, it's going to impact your business. And to your point, some of these things are not insurable because it's just not. understanding these and having a mitigation around them and one that is non -fungible, right? Not one that is, know good and well that you can't get in here unless I want you in

Andrew Ginter (WF) (32:37.106): That's right. it's a complicated world we live in. You gave the example, I think a telling example is central engineering. A lot of big businesses centralize their engineering teams. There's huge advantages to doing this.

Andrew Ginter (WF) (32:54.714): If you've got a big team of people who know similar stuff in one place, you have the opportunity for people to grow through their careers, to take on management roles, to become specialists, to exercise that specialty across a number of sites, to teach other people, bring them up the learning curve. You have all these synergies, all these benefits, but they kind of depend on

Andrew Ginter (WF) (33:24.082): the people being there together so that you can walk around and help each other out and whatnot. I talked about the most consequential sites. When you get higher on the consequence scale, nine times out of 10, it's also increasing in size. The facilities are just bigger. And part of the consequence is because of the amount of money tied up in these facilities.

Andrew Ginter (WF) (33:54.194): And think about it, if I have a billion dollar facility, nine times out of 10, maybe 99 times out of 100, it's producing a commodity. What is a commodity? It's producing gasoline. It's producing something that any other refinery in the world can produce to the same specification. And so how do I compete? Well, I can compete in terms of customer service. I can compete in terms of faster delivery time if I'm local. But most of the time,

Andrew Ginter (WF) (34:23.822): a big part of my competition is price. basic economics says that in an open market, in the international market where large amounts of commodities are traded, the international market is an open market. You might have communist countries saying, we control our market, but they can't control the international market. OPEC tries. But for most commodities, you're competing on price. And so we are constantly looking for opportunities to reduce cost.

Andrew Ginter (WF) (34:53.498): and centralized engineering has huge benefits and a lot of those benefits you can measure as reduced cost. And so if you want to remain competitive, you have to do this. If you want to avoid disaster, you have to deal with the security implications of doing what you need to do from a business perspective.

Andrew Ginter (WF) (35:18.054): When we looked at the solution space, we said, yeah, there's the high -end stuff, the inter -directional people are doing that. There's everyone else doing remote access software, and there's this gap in between. People are saying, you know, I'd like something in the middle, something that is truly interactive, so I don't need to be supervised, and that is stronger than what I've got. So there was this gap that we're stepping into. And I mentioned a moment ago,

Andrew Ginter (WF) (35:48.070): monitoring and helping out remotely. What I should say is, this is the first release of the product. And in the first release, we have a couple of key features. These are nothing new to the remote access software world. That's a very mature world, technology -wise. if we've got people, dozens of people every

Andrew Ginter (WF) (36:16.186): remoting in dozens of times to our plant. We'd kind of like to know what they're doing and so you can record those sessions. So there's a record. Some of these people we might not trust quite as far as do what you want, we'll record it. You might want to get a notice saying Fred wants to come in. Is he allowed? Yes, but let me watch what he's

Andrew Ginter (WF) (36:40.902): So, you I don't have to be moving the mouse, but I can be keeping an eye on what Fred's doing. Fred, Fred, you just logged into the wrong machine. Can you go over there? So, you know, extensive logging, session recording, session monitoring, these are, are, you know, built into the first version of the product. And of course, you know, there's going to be other, other features over

Aaron Crow (37:8.552): Yeah, that's great. mean, obviously understanding what's going on in the world and when I'm allowing somebody into my, it's no different than, I trust a plumber and I'm going to let him into my house. But depending on my relationship, it's just somebody I found on the internet. I'm probably not going to just give him the key to my house and say good luck and not have any oversight there. Right. But if it's a long time friend, family friend, I've known this guy for 40 years, maybe I do give him the key and I just let him in because I've built that rapport and that trust there. Right.

Aaron Crow (37:38.076): But that plumber not saying there's anything wrong with the plumber, not saying I shouldn't trust him, but it's trust but verify. I have no idea who this person is and I only want them to your point. That's the wrong bathroom, right? There's a reason why when you go into surgery, they mark on the, know, if you're doing knee surgery, they mark on your knee, which one? Are you sure that's the right knee? Yes, that's the right knee. So they don't do surgery on the wrong, because once they knock you out, you're not observing anymore. So that is a huge piece to this and understanding.

Aaron Crow (38:7.002): as well as documenting so I can go back and there's all sorts of benefits to that from a training perspective, understanding what changes were made. If I wanted to repeat this process again, I can look at the video, what happened, what, did they do? How did they go through this process? So I can more, I can train my people, et cetera. There's a lot of benefits to that.

Andrew Ginter (WF) (38:22.162): That's right. And it's not just, it's not just a question of trust. There's these other benefits you've mentioned. It's also a question of training. You know, Fred is, you know, with vendor X they're new has, you know, I'm looking at my, my list has Fred taken our safety training? No. Have they taken our safety awareness? No. Have they taken our security training? No. Have they tell you what you can come in, but I'm going to keep an eye on you. Okay. You need to do some training, but we need you today. And so.

Andrew Ginter (WF) (38:52.518): Yeah, it's not just trust. There's also capabilities that need to be considered.

Aaron Crow (39:1.702): Well, again, we do that. You know, I go to a, I went to a plant yesterday, like I said, and I was, I had escorted access. They let me in the door, but I couldn't be by myself. Like I had a badge on and it said escorted access. They trust me to be there as long as somebody else is watching what I'm doing. Right. eventually I could go through the training. I could get my own badge, all that kind of stuff. And then I would be, I would have unescorted access. but it's no different than what we're doing in these spaces. and unfortunately we're not always implementing those types of things in an OT

Andrew Ginter (WF) (39:19.730): That's it.

Aaron Crow (39:29.864): and there's potential impacts that, and again, I keep going back to CIE, but a lot of these implications and these risks, they go back to really understanding what you're doing before you roll something out, whether it's a process, a technology, et cetera, and really understanding the implications and what if, if I give you unfettered access to this environment, what is the worst case scenario? How could it impact me positively and negatively? And just really understanding those things so that I can make it an informed decision on how I need to

Andrew Ginter (WF) (39:56.976): Yeah, and risk is complicated. It's a subjective thing. I wrote a whole chapter on it in my latest book, and the feedback I've got is, Andrew, you've got some good ideas in there, but you didn't hit the nail on the head. I've got a formula in there. They said, look at your formula. What are the units on the formula? I do the units. The units are dollars. What number comes out the end? Billions and billions and billions of dollars. What does that number mean?

Andrew Ginter (WF) (40:26.124): sure what that number means. So I'm doing another version of chapter six. I've thrown out two versions already. But yeah, risk is complicated and it has to do with consequence. again, there's an unserved segment in the marketplace and this is the opportunity we hope to tap into. if I may,

Andrew Ginter (WF) (40:55.236): I've had some, in a sense, pushback when we did the announcement on social media. One of the bits of pushback was Andrew, if you've got one way out, sending countless messages out, and one way in, sending countless messages in, and you're taking my TCP and sending the in this way and the out that way, that's a wire, okay?

Andrew Ginter (WF) (41:18.682): Twisted pair wire has one way this way and one way that way. That's a wire. How much security is a wire got? That's not what we're doing. There are technologies out there that claim to be unidirectional that are doing that. They claim to be hardware enforced that they've got all the security of a wire and you're going, who are you trying to fool? We're not forwarding messages. All that goes out is the inside CPU scrapes a bunch of screen images and pushes it out. That's it. It's not a router.

Andrew Ginter (WF) (41:47.804): You know, people ask sometimes, what happens if I send Hera a message that Hera is not expecting? And they're expecting an answer like, well, there's rules in place for blah, blah, blah. There's no rules. know, Hera is expecting, you know, TLS connections. The hardware is expecting keystrokes. If you send anything else through, it says, I don't know what this is. You know, there was

Andrew Ginter (WF) (42:15.726): Even if the hardware let it through, the receiver would go, what is this? I don't know what to do with TCP requests. I'm not a router. And it throws it out. So buyer beware when you look at hardware enforced solutions. Ask the question, which attacks does this solution defeat that the competition does not? This is why it's essential.

Andrew Ginter (WF) (42:43.162): for all of us practicing the space to understand attack scenarios, understand how attacks happen, so that we have a laundry list of attacks and we say, okay, I've got two competing technologies. This one defeats these six attacks and these other ones are irrelevant. This one defeats only four of them. These two relevant attacks aren't defeated. This one is stronger than that one. So when you look at a technology like a wire or like Hera or like a firewall,

Andrew Ginter (WF) (43:12.454): look at attack scenarios and understand and to come back to Hera, any compromise of the internet exposed CPUs is unable to propagate into the OT network. That's the claim to fame. That's not true of the software -based competition. Now offense to the software -based competition, there's lots of industrial sites where it makes sense. There's lots of industrial sites where hardware enforcement makes sense.

Andrew Ginter (WF) (43:42.342): There hasn't been that option to date, and there is today.

Aaron Crow (43:49.328): Yeah. And using that analogy for folks that maybe are not following along, but you know, if you look at the difference between I'm going to restrict data flow via a firewall, like I'm going to connect these two devices and I'm going to send data through and a firewall is going to say yes or no, that that data can pass, but I can, I can misconfigure a firewall. can, I can obfuscate the data so that the firewall is not looking at it. Like we talked about a minute ago, which the difference of that

Aaron Crow (44:17.424): not even talking about a waterfall. If I just, let's say I'm gonna send data out and I have a fiber pair, one is transmit, one is receive. If I only connect the transmit side on my inside device and the receive side on the outside, I can't send data back. just like goes one way. I would have to roll both fibers. I'd have to do a whole bunch of physical things for that to work. Like there is absolutely no way that I could send anything back and it ever get back to the other machine. And that is the difference between

Aaron Crow (44:45.844): like both there's spaces for both types of technologies. But like you said, the higher up the risk that you are in the on one end of that spectrum, you can't take a you're less likely to be willing to accept the risks of a software only solution. Again, I'm not I love firewalls firewalls. There's a lot of use cases for them in a lot of different spaces. But to your point, in a nuclear environment, I'm never going to have a external facing device ever

Andrew Ginter (WF) (44:54.266): I came to a few years ago after working in the space for a long time.

Aaron Crow (45:12.102): waterfalls are used at almost every nuclear facility I've ever been at. And that's because I cannot risk anything ever getting back in data out. And that is all I want to do. I'm sending PI data. I'm sending monitoring and logs and alerts, but nothing ever gets back in. If I'm going to control, I'm walking in the

Aaron Crow (45:45.896): It just says.

Aaron Crow (45:52.048): I've never seen that before.

Aaron Crow (46:7.954): Well, why don't you close the tab and we'll go right back into it and we'll just start again. I'll hit stop on this.

Transcript lightly edited for readability.