Own Your Industrial Airspace: Wireless, Vendors, and the Parking Lot Crane Hack with Scott McNeil

Intro and Guest Introduction

Aaron Crow (0:00): Hey, thank you for joining me on another episode of Protect It All Podcast. Man, I get to have a lot of fun having conversations and geeking out over nerdy shit. So it's really amazing the amount of folks that I get introduced to, that I come across, people that are doing awesome, fun things.

Aaron Crow (0:22): Scott, I'm super excited. We've already talked. In fact, I had to stop us from talking before we started recording because we were going down too many rabbit holes that I wanted to make sure we were capturing. But I'm excited about this conversation. So with that, Scott, why don't you just introduce yourself, tell the audience who you are, what you do, your background, all that kind of good, fun stuff.

Scott McNeil (0:40): Sure. My name is Scott McNeil. I work for a company called GPA, Global Process Automation. We are industrial integrators. We have an automation team, we have a machine intelligence team, and I am one of the senior engineers on the infrastructure team.

Scott McNeil (1:02): I've been a network guy for twenty plus years. I've been strictly OT for the past ten years. I got recruited by one of my old mentors from back in the day and never turned back once I went into the world of OT. I love it so much compared to IT. But I've also been a wireless engineer from the get-go, so I'm a twenty plus year wireless engineer.

Scott McNeil (1:29): After the transition into OT, I've become one of the few people who specialize in wireless applications and deployments in industrial and manufacturing settings that doesn't work directly for a vendor. With GPA being that we're integrators, we're whores. We'll work with anything. It doesn't matter, right? So we have to kind of maintain that vendor agnostic status, which is fine by me. It gives me a lot of opportunity to work with a lot of different gear, and then bring it all around to the fundamentals on what you should and should not do and all that fun stuff.

Scott McNeil (2:18): But on top of all that, I'm also the host of the Industrial Wi-Fi Shop podcast, where we talk about all things wireless in the industrial universe. Shout out to my co-host and buddy Jeremy Baker. Peace out, Jeremy. We've been doing that for two and a half years. We don't have a ton of episodes because we do one about once a month since we work full-time, but it's a lot of fun and we do deep dives and go down rabbit holes, everything from industrial clients to security to what's on your workbench. It's fun, man. I have a good time with it.

Commercial Off-the-Shelf Gear and Over-Segmentation

Aaron Crow (2:44): That's awesome. And again, we were talking before we started recording. You don't think about a lot of the time, or many people don't think about a lot of the time, networking and how important it is in all the things that we do, but in OT specifically. We turned that bridge a long time ago when the vendors started bringing commercially off the shelf gear into these spaces. When I'm doing assessments or I'm looking at these networks, nine times out of 10, the gear is Cisco, right? We're dealing with stuff that's commercially off the shelf. Sometimes there's Netgear, sometimes there's D-Link, and obviously there's wireless all over the place. It's insane what you see. But at the end of the day, they just need to make it work. So a lot of times they're trusting their vendors, the vendors are bringing in things, and it depends on the maturity level when they bought it.

Aaron Crow (3:14): You go to Emerson today, Emerson's stack is really sophisticated and capable. But if you bought that stack when they first implemented networking in 2015, it's probably not as mature as it is today. At one time they did like /30 micro-segmentation. Everything was routed and it was just insane. Yes, segmentation is great. That was over-segmentation. It was overly complex to be able to make anything work whatsoever.

Aaron Crow (4:08): And nobody could make it work, so they just undid it. They did segmentation, but then there were no firewall rules. Everything was allowed to route to everything else. So then what's the point of segmentation? It's not really segmented. You're just making it complex.

Scott McNeil (4:19): Well, then you look at DCS vendors, right? You look at your big DCS vendors like ABB, Honeywell, Emerson DeltaV. They've gotten rid of all of that segmentation now, and they've done this actually for years now. They essentially run on whatever their own flavor of PRP is for redundancy. So you have two physically separate networks and they duplicate everything. It's incredibly expensive to put in. Now granted, your uptime and your failover time is flawless typically if it's set up and deployed right.

Scott McNeil (5:03): But who can afford that? Even a small change is a capital project. When you start looking at your smaller facilities, they don't have that kind of budget, right? It's insane. I come from the IT universe. I came from higher ed, and so we were so used to VLANing everything out. Then when I came into the OT universe ten years ago, I was floored when I would go to a site and there would be thirty-two subnets on VLAN one, and each subnet is a 192.168.1.x, 2.x. And I'm like, this is IT 101.

Scott McNeil (5:53): But that's when you also realize, especially with the smaller sites, more independent sites, not so much with your big conglomerates, that they just don't have the resources to get a true architect in there and architect their network from the ground up. They just ad hoc as they grow homogeneously over time, and then they end up with this huge thing we refer to as an ugly baby.

The Ugly Baby: Spanning Tree Nightmares and Skids on Corporate

Scott McNeil (6:22): It's just a big mass of things. Hey, we need connectivity over here. Let's get a switch, plug it in, and we'll leave it alone. They have no idea about spanning tree. They have no idea about these other protocols. Ten years later, you have this massive spider web of a network that's all on one VLAN with no spanning tree configured. Every time they connect a new switch, the whole network shuts down for 30 minutes while it recalculates, and then allows them to continue. So there are a lot of these nightmare scenarios that you tend to run across out there.

Aaron Crow (7:02): It's so huge. I literally just saw this in an implementation at a site, and it had one of the fairly new vendors. I'm not going to call out the vendor specifically, but most of the control vendors are that way where they have these very rigid physical segmentation, which I love. Don't get me wrong, in OT I love it. They have a primary and a secondary network, and it's completely isolated from anything else. All their devices are dual homed to both systems. Then they have a tertiary system for backup, so they technically have three networks. But because they're so rigid and because that vendor is so locked in, only their things go on there. This customer had a third party skid that they added after the fact. Because they didn't have any place to plug it in and the only quote unquote OT network was this control vendor's network, and obviously you can't plug it into that network because that's not vendor X, they would not allow that.

Aaron Crow (7:57): So then they just said, well, we don't have any place to plug it in, so we'll just plug it into corporate. So it's plugged into the business network that has internet access. And also because the vendor needed remote access to it from Germany, these devices are directly plugged in on the business network so that their vendor has remote access, because it was the only place they could plug it in. And it's because these sites are small, they have small budgets, they don't have technical architects, they don't understand the actual reasons they're doing the things that they're doing. It doesn't have to be overly complex or expensive to do that differently, where you're bringing that in and you're putting it in an OT space.

Aaron Crow (8:55): Or I'm adding another switch, doing something and bringing it down to a protected area, not putting it basically for all intents and purposes on the internet as an OT device, which is a bad idea.

Scott McNeil (9:06): Totally. But man, you go to these sites and it's just a hot mess. Then you can sit there and explain to them there's so many things they can do that are going to cost them either nothing or very little. There's a ton of low-hanging fruit out there that can really up your protection game without a huge monetary investment. But if there's not a monetary investment, then it's going to cost you in a different way. It's going to cost you in time and effort. So you've got to start dedicating the time and effort to take care of those things.

Low-Hanging Fruit and Defense in Depth

Scott McNeil (9:48): From simple things like password management to something a little more complex like understanding your network, getting a network inventory, taking a look at your actual switch network, understanding spanning tree, and even just setting some baseline priorities. Going through and doing a lot of these little basic steps will first stabilize your network, but then on top of that, they'll start helping you move further along the road to those security goals.

Scott McNeil (10:19): Because it's all about defense in depth, right? Nothing's going to be perfect. Anyone who's dedicated all their time to try to get into your system more than likely eventually will. But the whole goal is to make it as difficult as humanly possible. Like I said, there's a lot of low-hanging fruit that especially the smaller shops can, if they just dedicate the time and effort to, they can save a lot of money at the same time but still lower their risk. Because that's what it's all about, especially when they go to get some sort of cyber insurance. It's all about that risk. So whatever you can do to mitigate as much of that risk as you can, you really need to.

Scott McNeil (11:00): And then start stashing cash for bigger forklift upgrades and the changes that you need to do during outages and whatnot. Because god forbid, you can't do it on the run.

Aaron Crow (11:13): It's funny, like you, I came from a networking background. I came from IT. I've spent more than half my career in OT, but I've also seen the other side. Coming from networking, networking always really fascinated me. I had the Cisco switches sitting in my rack and I was doing all of the CCNA and CCIE tests and playing around with all that type of stuff.

Aaron Crow (11:42): And I still use those tools today. A prime example: when I do a discovery, when I'm doing an assessment, I want the switch configurations, I want the firewall configuration, but I don't want just the switch configuration. I want them to do show CDP neighbors, talking Cisco IOS, right? Show MAC address table. You can map your network with your switches because it knows the MAC addresses that are plugged into it and the ones that are adjacent to it. I can do an 80 to 90% map of your network running a few commands on your switches, and then doing a MAC address search on Wireshark's OUI database. I basically have a really good list of here are the MACs, the types of devices they are, and the ports they're plugged into in your network.

Scott McNeil (12:31): Absolutely. When I can't go inside and do it myself and the site's going to send me configs, I have a whole block of commands where all they've got to do is copy and paste this in and hit enter, then copy and paste the result into a notepad and email it to me. That's all they've got to do. It gives me all the things that I normally look for. A lot of people have CDP neighbors, but a lot of people don't even think about LLDP. LLDP is the link layer discovery protocol that the entire rest of the known universe uses.

Scott McNeil (13:12): When running LLDP, if you have that active on your switching, you can start getting information for non-Cisco devices. You're not going to get everything, but by God, you'd be surprised at the information that you can get off of that. So you take that network mapping one step further, and you're starting to get kind of an asset placement map at the same time. Again, these are things that don't cost anything as far as money. It's time and effort. However, it all goes back to you don't know what you don't know. If you don't even know to look for these things, it's not going to be on your agenda. So it does take some initiative to educate yourself on these things, or spend the money and get someone in to do it.

Scott McNeil (14:02): I always go back to this. There's a hundred different ways to skin a cat. Not that I advocate skinning cats, but the point is there's multiple ways to do things. So you've just got to pick and choose what's going to work for you.

Foundational Blocking and Tackling: Stabilize Before You Secure

Aaron Crow (14:19): And to your point, you just don't make it, hey, I don't have any money. Awesome. There are other ways that you can skin that cat that are not super expensive. I love cats. We have a cat. It's usually bothering me about right now. To that point, I talk about this all the time. I've talked about it on stage, I've talked about it on my podcast. Most of the foundational things that most OT places need to do are not sexy. They're not buying a new technology. It's basic blocking and tackling. It's segment my environment, know what's on my network, do basic monitoring, have some kind of disaster recovery plan that I can actually test. Do you have configurations and understand how your environment works, and could bring it back up if you needed to? Just basic things that we should be doing anyway.

Scott McNeil (14:55): It's interesting because everything you just mentioned there, those are all checkboxes for the foundation of the latest buzzwords of the zero trust model, right? You've got to have your segmentation in place. You've got to have all these different things that you were just talking about in order to get to the point where you can then control everything through a firewall and get that zero trust model working. But it takes time. Everyone has to realize they're not going to be able to do this overnight. This is not a forklift operation. One of the sites that I have been working with, it's been four and a half years and we're finally getting to that point. But we had to get in there and stabilize the network first, because if you don't have a stable network, what's the point?

Scott McNeil (16:08): You have bigger problems if your network's completely unstable and things are dropping and going down. You need to take care of that before you even jump on the security bandwagon. A stable network, that's the foundation of everything. If you have a terrible foundation, everything you pile on top of that is just going to be horrid. Once you get that stability going, then you just start building on all your fundamentals like we were just listing off: network segmentation, device inventory and maintaining that inventory, basic monitoring. Start working on some firewall rules, get that firewall in there to segment you off of IT so that your two sections are not cross-pollinating, because that's always bad news.

Scott McNeil (17:07): It all goes back to something that I touched on a minute ago that they've been teaching for 30, 40 years, and that's defense in depth. I have yet to find anything that doesn't come back to defense in depth. Understanding that concept alone can take you pretty far and start to stabilize and develop your own networks. And then there's wireless on top of all that.

Into the Wireless Universe: War Driving and Open Networks

Aaron Crow (17:40): Well, let's walk that path. I've done these asset discoveries and assessments and walked into these places and helped them with things. The worst I've ever seen, and most of them are bad, I showed you before and you laughed at me because your tools are better than mine. I have the little Pwnagotchi Pro, or I've got the Marauder back there. It can plug the Rat into different things and run. Obviously I'm not trying to attack networks. I'm just mapping, doing basic war driving, documenting. Because most environments don't know what assets they have. They also don't necessarily understand that every one of those skids they brought in probably is broadcasting wireless and they don't even know it. If you don't know that it's there, does it need to be on? Are you utilizing it? I'm not saying wireless is bad. You can absolutely do wireless, but you need to be intentional about wireless. Is it turned on intentionally? Does it have credentials? I can't tell you how many times I'll walk around and it's in an open network that's directly plugged into their OT network and gives me an IP address when I connect to it.

Scott McNeil (18:48): Man, so real quick, we were talking about tools. I'm a NetAlly junkie and these guys have been game changers for me. I've got a LinkRunner and a CyberScope Air. As far as my toolkit, these are relatively new. Only the past couple years have I had these. Before, everything was a much more manual process using smaller tools. I had even gotten a copy of GrassMarlin back in the day for passive network mapping and whatnot. There's a ton of smaller free tools out there that you can use. But again it goes back to our conversation of it may not cost money, it just costs time and effort. You've just got to go find them and then use them. But they've been game changers for me because it streamlines the process and takes a bunch of the steps out.

Scott McNeil (19:49): When you start getting into that wireless universe, this is one of the biggest pet peeves I have with most skid vendors. A lot of them, when they drop off their chemskid for whatever process it may be, they have their box mounted to that skid that's got all their stuff in it, and they say, here, just plug this into your network. And then half the time they don't even tell the site that there's a cellular modem on that for them to remote in and take a look at the system, to see if it's time to send the refill, or to get monitoring information about skid levels so they know when to send another tote. And then on top of that, like you were saying, they'll be broadcasting Wi-Fi so if one of their technicians goes on site, they can connect up and take a look. The problem with that is that's a direct line into your OT network.

Scott McNeil (20:45): Half the time that wireless is an older version, so it's like Wi-Fi 4. If you're lucky, they're running WPA2, if they even put any encryption on it at all. A lot of these vendors just say hide the SSID and roll with it. Well, the whole security through obscurity is no security at all. All it takes is a quick scan and you see, hey, I wonder what this MAC address actually is. Let's connect to it. And it's just a quick packet capture and you see all the information and you're good to go. You can connect right up and then you're in Happy Land because you have access to everything.

Policy First and the Generational Shift Toward Wireless

Scott McNeil (21:15): But there are so many things. Where do I even begin with security as far as wireless is concerned on OT? Honestly, it should start with policy. It should start with site policy, and have a list of thou shalts. So vendors, thou shalt not implement wireless without approval. Or employees, you're only allowed to use your mobile hotspots in certain areas, and different things like that, to cut down on all the extra stuff. But the big thing is controlling what the vendors are putting in. If nothing else, by policy, making sure they know they have to notify you. And if you are running encryption on that, they need to give you the credentials in order to access that.

Scott McNeil (22:09): There's so much else that goes into it, man. And then you start thinking about all the different wireless systems that have started migrating their way into the OT universe. Because 20 years ago, it was all your old crusty engineers and they didn't trust wireless at all. It was a fight to get any sort of wireless device in the OT universe. But then over time, as wireless has advanced, it's changed some hearts and minds. More so, engineers are retiring and you're getting new engineers in, and these newer, younger engineers have grown up with Wi-Fi. It's been part of their lives from day one.

Scott McNeil (23:05): So it's not even a conversation with them. They expect it. Well, why don't we have wireless systems in here to take care of this? Why are we going to spend a hundred thousand dollars on infrastructure for this one section out there in wastewater that's three quarters of a mile away, when we can spend eight thousand dollars and put up a reliable wireless link to pull all our data? It just doesn't make any sense to them, because they've come up with it. They have that inherent trust of the technology. But there are a lot of things too that even those who are familiar with the technology don't necessarily look at from a secure standpoint. There are a lot of things that you need to really pay attention to.

Beyond 802.11: Industrial Wireless Protocols and the Wild West

Scott McNeil (23:58): Especially with all the latest, greatest advancements when you're looking at Wi-Fi 6, Wi-Fi 6E, and now Wi-Fi 7, and then Wi-Fi 8 is right around the corner. But the scary thing, well, I say scary, but it's not really scary. The thing is that it's not just Wi-Fi. There is a ton of other wireless protocols out there as well that's not 802.11. Most of which are 802.15.4 based, but they were actually intentionally designed industrial wireless protocols. You look at WirelessHART, which is the wireless extension of the HART protocol, and you look at ISA100. Those protocols were designed by conglomerates of DCS and controls vendors. They were designed specifically for these environments.

Scott McNeil (24:54): Because they're 802.15.4 based, they already have encryption built in as part of the standard that they're built on. So while their application layers may be different on how they operate and how they handle data on the back end, at least from a physical standpoint, you already have some security built in, which was cool that they actually thought about that some twenty years ago when they were designing it, before everyone really started running around with their hair on fire realizing how vulnerable all these OT systems are.

Scott McNeil (25:23): And then on top of that, you have a ton of manufacturers out there who have their own proprietary stuff for wireless I/O and sensor suites, like Phoenix Contact. For their wireless I/O they have their own proprietary protocols, down in the 900 megahertz range and then 2.4. Banner Engineering, all of their sensor suites and everything run on their own proprietary protocols. Now it's starting to become the Wild West out there, because you have your use cases for 802.11, which you have standards with. You can build to specific ideological platforms as far as security and design, and you have standards to follow. But then when you start looking at all these other things that are very useful in the OT universe, they don't have those standards because they're all proprietary. So you've got to kind of build your own mental checklist of, does this meet XYZ requirements in order for us to maintain our secure environment.

Power Plants, Abandoned Wireless, and a Fiber Nightmare

Aaron Crow (26:39): Right. Going back to that, I've been doing this a long time. My dad was in power utility for 40 something years, so I grew up around it. He was a controls guy, an I&C guy. He built control systems for these power plants. Many times, especially early on, in the OT space around 2014, 2015, I would ask him these questions. Hey, he built this system, and I would go to him, why did you do this, or why didn't you do that? There are so many systems that I saw, not just from that time but even beyond that, where there were wireless systems that these sites spent, let's say a million dollars to implement, industrial wireless from some of these large vendors. And then they just abandoned it in place.

Aaron Crow (27:32): It was because they thought it was going to work at the time, but then it wasn't working reliably enough. Because in a power plant, there's all this metal structure. They couldn't get the signal. Sometimes the sensor would work and sometimes it wouldn't. In these really critical environments, you can't have a hiccup. At our home, if I drop my packet for a few milliseconds or even five or ten seconds, it's fine. It doesn't impact anything. But in a very critical system where sub-milliseconds matter, those things are unacceptable. So sometimes it's like, I'd rather run a fiber out to that wastewater site. Yeah, it's going to cost. I literally ran infrastructure in a power plant. It was about a two to three hundred thousand dollar project. At least half of that was running fiber. It was 900 meters of fiber. It came on a big spool, and we had this whole problem with it because they put the ends on the fiber while it was on the spool. It was Kevlar jacketed. So when we unspooled it, the jacket was longer and it basically broke the fiber.

Aaron Crow (29:00): So we installed it and didn't know that we should have tested it, or the installer should have tested it on the ground. It had MTP connectors on it. It's 48 strands of fiber in this one wire. Once we installed it, we're like, man, this isn't working. Some of them are working, some are not. About three foot in from the end of the cable, the wire was broken. So we had to fusion splice it in a power plant. They brought in this whole gear and plugged it in and the unit was offline. When they plugged in this fusion splicer, there was so much RF noise that it almost tripped the turbine control in the other unit, not in the unit we were running. It was going across the bus, and there was so much RF that it almost tripped the turbine. The sensors just went nuts in the control room.

Aaron Crow (29:55): It just goes to show, we spent that much money because we knew we couldn't reliably get wireless to that place without a lot of complexity. I think you could do it today, but the technology that was there back then, we were not confident that we could do it in a reliable way, and in a way that we wouldn't be troubleshooting it constantly. Because we'd already experienced where a control vendor that starts with an E had deployed wireless stuff and it just didn't work. It worked great where it worked, and then other places it didn't work, and no amount of their troubleshooting worked. So we basically just left it abandoned. Those folks like me and my dad and others like them, we are not in that decision place. To your point, wireless should be able to work. It should be a lot easier to do those things. But sometimes in OT, you don't do it because it's easy. You do it because you have to make sure that it's going to work. In a nuclear power plant, you don't care how much it costs. You want to make sure safety, it all counts.

Where Wireless Shines: Mobility, Fixed Assets, and NERC CIP

Scott McNeil (30:59): Absolutely. When you're looking at power generation, that's a whole different universe than even your standard OT, especially when you start looking at NERC CIP and all of those rules that you've got to follow and maintain. While wireless has come an amazingly long way in the past twenty years, I'd be skeptical with certain things on wireless in a power generation environment too, or anything that's mission critical, a chemical process or anything like that. Because while your reliability in wireless has come leaps and bounds and is great for mobile devices, bots that are moving around, trucks running around mining sites and things like that, that's where wireless really shines, because it can move and adapt with the mobility aspect and roaming.

Scott McNeil (31:57): Where fixed assets, if a fixed asset can be hardwired and you can reasonably afford to hardwire, then hardwire. At one site where we do a bunch of wireless for wastewater, the reason they don't maintain hardwire is because all the heavy vehicle traffic keeps crushing all of the buried lines that the fiber was in. So it was getting too expensive every couple years having to redo that entire infrastructure. At that point they made the call. In this instance they're pulling environmental data, and being able to do controls remotely is a bonus, but they know if they need to, they can just drive out to the site out in wastewater and do manual controls on it. They always have that as a backup. But wireless definitely has its place, especially when you look at different innovations. With just general 802.11 Wi-Fi, there are paper mills now that when you're around a paper machine, all their HMIs are on mobile pedestals.

Scott McNeil (33:15): Their HMIs are wireless. They park that HMI right in a certain spot, and they've designed and developed solid wireless coverage around the paper machine. So they can move that HMI to wherever they need it, whatever they're monitoring at that point, and it gives them a whole lot more flexibility. And then, because most HMIs are Windows-based workstations, they can add extra VLAN connectivity to documentation repositories. They can downsize the HMI and pull up blueprints of something they're working on. Or if they don't want to do that on the HMI, because they have wireless working around, they have heavy duty tablets that they can carry around that have access to all that. So they're not having to deal with large printed blueprints or running back and forth to the office to look something up. There's a ton of other uses that are non-critical to the process where wireless really shines. It's getting people to understand that and invest in it.

Aaron Crow (34:20): In those larger spaces, power plants are not a great example where I'd want to use wireless, it's not the use case. But at the Tesla factory where it's 10 million square feet and it's wide open, it's super easy. In every other manufacturing facility I've ever been into, it makes perfect sense. Why would I want to hardwire everything? They all have tablets. Those robots are moving all over the place. Wireless is embedded into the OT process, and it should be. Even in power utility, we would have microwave, which is wireless. We'd have microwave on the power lines. We would have a remote location that was 20 miles away. Are we going to run fiber out to that place to get some tank sensor level or something that we need once a week or some shit like that? No. We put in RF and wireless microwave because we didn't need it. It was just like a homing beacon that says, yep, I'm up. It'd give me a basic level, a number, 10, 15, whatever those things were. I'm not going to run fiber out for that thing. That's insane. And I don't need cellular either. I just need a signal back. That's it.

Private Cellular vs Wi-Fi: The Handoff Difference

Scott McNeil (35:43): Yep. Well now, private cellular, that has its place too. There are actually a lot of really solid industrial or manufacturing use cases for private cellular. Because there's a big difference in how cellular handles clients versus how Wi-Fi handles clients. And the way cellular does it is a lot more stable in the grand scheme of things. Because in the cellular universe, when you're running around with your cellular device, I'm a bit of a Star Wars junkie in case you haven't been able to tell, no, don't tell anybody. In the cellular universe, when you are roaming from tower to tower, it is not your device that's making that decision. It's the network that's making that decision. The network says, okay, this device has reduced signal strength here, but it's coming into this other cell and we know they're traveling in this direction. We're going to go ahead and hand off.

Scott McNeil (36:53): That's what gives you your seamless voice and video as you move, because it's able to prep your device and say, hey, we're getting ready to switch. Three, two, one, we switch. In the Wi-Fi universe, it's the client device that makes that decision. And every manufacturer's client devices, be it Apple, be it Samsung, be it Ericsson, whoever, they all do it differently. So there's no consistency. When you look at laptops and other computers and your whole wide variety of different client device chipsets, everybody does their roaming algorithms differently. So it's very difficult to standardize on that. All you can do is make the environment as controllable as possible to where you think you would cause roaming points.

Scott McNeil (37:49): That one significant difference between the two is what puts private cellular really in there for more deterministic use cases, because you have much better control over that. But both have their places in the universe. One's not going to outshine the other across the board. They have their particular use cases where they both make sense. So you really have to pay attention to your use cases and what you want to do before you just go willy-nilly start integrating something.

Don't Trust the Vendor: Field Firewalls and Isolation

Scott McNeil (38:16): I'm a big advocate of don't just take what the vendors say at face value. You need to know everything that that vendor's trying to put in. You can't trust them explicitly, because they're not the ones who are going to be supporting it 24/7. You are. They're not the ones having to defend your environment. You are. It's in your best interest to understand everything that they're putting in and control it as best you can. Even if you have to put out a little field firewall to segment them off and only allow the data you need. I actually see that's very common. You can get a little Phoenix MGuard firewall, or if you really want to go overkill, you can get one of Fortinet's DIN rail firewalls that actually do everything all their regular firewalls do.

Scott McNeil (39:18): You can put those out there and start isolating those things to only allow the traffic that you actually need to get off of those skids. And then, screw the vendor. They can do whatever they want because they're isolated off. You don't care anymore. They're really no longer an attack vector when it comes down to it, because they're their own little isolated island. The only traffic you're letting through is very, very specific. It's just a much more secure way to do things.

Aaron Crow (39:47): And then the only thing they can impact is that one process. That's where, as an asset owner, as the OT guy, the operations guys, the engineers, the plant manager, whomever you are in that role, that's what you need to be thinking about. Not only what is this process, what am I bringing in, but what can it impact? How can that impact other things? Because that's the thing that most don't necessarily understand. They see, well, this is just a chemical, it's a non-important process to my critical process. It's just this thing that's out there for wastewater. So I just brought in this vendor and they did that. But they don't necessarily understand, yeah, but then you plug that into your network, and now you've brought that risk, and now you've put the other systems in your process at risk because this system is weak.

The Weakest Link and False Confidence

Aaron Crow (40:44): It's that old adage. A chain is only as strong as its weakest link. You've now added a back door, you've added a side door, whatever analogy you want to use. And then add on top of that, you already don't understand what's on your network. You already are not monitoring. You probably have a very porous firewall if you have any rules at all, if there's not an any-any, which I unfortunately see way too frequently. So all of these things combined make a really bad recipe for something bad to happen. So many of these sites are overly confident that their system has worked, and it has worked. I can't tell you how many times, well, we've had this for 10 years and never had a problem. My response to that is two things. It's usually, well, how do you know you haven't had a problem? What is a problem for you? Does that just mean that nobody has crashed your system and you haven't gotten a ransomware attack yet, or do you know that there's not issues?

Aaron Crow (41:42): I literally had a person on my podcast the other day, it hasn't been released yet, but one of the big things that are resurging in OT is all of these old viruses. We've got Windows XP and things like that running in OT, and they're seeing Conficker viruses, those old worms, actively in these OT spaces. The thought is that they've probably always been there, but for the longest time we weren't monitoring. So now that we actually have visibility in these spaces, we're starting to see all these weird things happen. When you actually do a deep dive, it's got Conficker running in these spaces, which is really hard to clean out. Now granted, it's not actually sometimes doing any harm.

Aaron Crow (42:33): And you led to this in the beginning. Sometimes these networks are so overbuilt that a lot of the noise, there may be traffic storms that you would notice on IT and fix. These OT things, because OT traffic is usually not that robust as far as how much communication it's doing, sometimes these networks are overly over-designed. So they brunt a lot of that misconfiguration or issues before you notice anything. It just takes one bad thing to happen, and then the whole dominoes thing falls and you're having a really bad day. But you don't even know that it was that that caused it. You just reboot the system, it comes back up, and then you don't realize that your network configuration caused a problem, or you had Conficker, or any of these other things, because you're not looking for those things.

Device Lifecycle: XP, Windows 7, and Virtualization

Scott McNeil (43:27): The existence of XP and Windows 7 machines still on these networks really shows the difference in device lifecycle in OT versus IT. When you look at the fact that we still have to deal with XP machines out there and there's nothing you can do because they're at a specific patch level, that's it. They're not being developed anymore. Windows 7 is in the same boat. But because the system that they work with just works, they're like, we're not going to replace it, it's too expensive, and there's no guarantees it's going to work any better. So now you have to defend these ancient OSs. Really the best thing you can do is then get rid of the physical machines, virtualize those so you can secure them in a VM environment and roll with it in that direction. Because then, at least, since you can no longer secure the device, you have to secure the perimeter around it.

The Parking Lot Crane Story: An Open SSID

Scott McNeil (44:32): Going back to talking about vendors, I was at a site and I was sitting in the parking lot, and because I'm a nerd junkie, I want to see what I can see before I even go in. I am seeing this SSID being broadcast out that is obviously their crane anti-collision system. So they had a radial crane and a big gantry crane, and these, when they get to a certain proximity, it will stop one or both or whatever. That SSID was open.

Aaron Crow (45:18): Of course it was.

Scott McNeil (45:18): So I was like, well, that's an open SSID. Let's try to connect to it. Well, I couldn't connect to it. Again, I'm just sitting in my car out in the parking lot. I'm not, I haven't gone through security, I'm not on site, I haven't talked to anybody. Granted, this is a site I've been to before, so it's not like I'm just randomly going up to strange sites and doing this. I was like, this is interesting. So I do a quick PCAP with my handy dandy CyberScope on that channel and see that they just don't have DHCP running, that's why I'm not connecting up. But now that I have that PCAP, I see what their IP range is. So I just set myself to that IP range, connected right up, and did a quick passive scan. Within the span of 20 minutes, I had all the information I needed to know on that anti-collision system.

Scott McNeil (46:17): There were four Rockwell controls devices on there. There were several other HMIs and workstations, and I had all that information right there. Conceivably, because they're PLCs, and these were Rockwell PLCs, they had ENT2 cards that are notorious for not being very strong. I could easily have taken them offline with just a concentrated ping sweep on them, a high volume ping. I guess you could call it a ping attack, but I don't know if there's a specific word for that.

Aaron Crow (46:59): Denial of service, basically, right? It's just an nmap.

Scott McNeil (47:10): Essentially just hit them and could have shut down their cranes right there from the parking lot, and they never would have known any different. Now granted, as soon as I went in there, I immediately was talking with them and said, hey look, guys, this isn't good. You need to secure that. And they're like, what do you mean secure? We need to put a password on. Okay, we can do that. And ten minutes later we had it fixed and all is good in the universe. But if you don't know to look for it, how are you supposed to fix it? I guess that's why too, guys like you and I have a job. They get us to come in and do these assessments so they can find these holes and get a path to remediation, so they know what they need to fix and work on. Because they don't have the in-house expertise to figure that out for themselves.

Advice for OT Buyers: The Questions to Ask Vendors

Aaron Crow (47:51): That's the thing. I want to ask you to tell the audience, obviously I have everybody from new folks that are wanting to get into this space, cyber, IT, OT, and everything in between. On the IT side this is an easy process. We would never have this issue on the IT side. But let's talk to those OT folks, or people that are wanting to get into that OT space, or maybe they're going to be there in the future. What are some things, thinking about most of the people that many times are making the decision on what product they're buying, the skid? They're not buying the skid because of the wireless stuff. They're buying the skid because of the prime function that it does. It just happens to have wireless. So what are some of the things that they can and should be looking for and demanding of their vendors to hold up to? Asking the questions of things like, do you have wireless? Do you have Bluetooth? How do you configure those things? Do you have remote access? Do you have a cellular card? And if the answers are yes, what is that next question of, okay, how do you secure it? How do I know what's in there that nobody else has access? What are some of those things that you advise folks so that they can take this back home and start asking their vendors these questions?

Scott McNeil (49:21): Well, and we touched on this at the beginning. It all comes back to policy. Does your site have a policy for these things? Because if you don't, then you have nothing to really stand on, and there's no framework there for people to back you up on these things. One of the things I would suggest is start developing both OT wireless and network standards, and develop that into a policy. Because once you have a policy, then you have company enforcement to back you up on that. And there are tons of templates out there. There are lots of different consultants like us out there that can help them figure out, hey, what's going to be the best policy to suit our needs? Because while to a degree you can use a framework, it's not going to be 100% cookie cutter, because every site is different. Even if you have two paper mills side by side making the exact same thing, on the back end they have lots of differences. So you have to take that into consideration.

My House, My Rules: Owning the Vendor Relationship

Scott McNeil (50:30): So I always advise policy is where you want to start, so that you have the company buy-in and the backup for when you start telling these vendors these different things. Say, hey look, we have a policy that no vendor skids can be broadcasting wireless of any sort because of our wireless systems. And if they kick and scream about that, we'll say, well, okay, then we're going to go with vendor B. Because this is one thing that the sites have to learn. You are paying for this. This is a service they are providing to you. You need to dictate how they communicate to you, not them dictating to you how everything works. Why am I going to let somebody else come into my house and tell me how I have to arrange my furniture? No, it doesn't work like that. This is my house. I live here. I decide where everything goes.

Scott McNeil (51:20): And then when it comes to remote access, this is a huge contentious point. Vendors always want to take, well, we put this in to remote support that. No. We have or are developing a secure environment. We have remote access systems in place. This is how you will access our systems on our terms. Because that is a huge, huge deal. So many people don't even realize their systems are out there that have all these little cellular devices on them, and you have no idea.

Remote Access Control and the Target Breach

Scott McNeil (52:01): I know this is a bit on the paranoid side, but you have no idea if this guy from Sweden who is remoting in to work on your specialty systems, you don't know what kind of day he's had. You don't know what credentials they have. You don't know if they are even maintaining a secure process on their side. And because they're halfway around the world trying to support your gear, you don't even know what times he's logging in and working on things. You could be in the middle of a process and he's in there changing what he needs to change because this is just what they do. Remote access is one of the most important things that a site really needs to get a handle on, and the site needs to dictate how that's done.

Scott McNeil (52:50): There's remote access systems from Claroty, there's remote access systems that Honeywell uses, there's BeyondTrust, there are all these great things you can put in place that not only give you time and place control, but they record the sessions. So there's vendor accountability on top of all that. And there's access control. Not only am I restricting you time-wise, but when they go to log in, it kicks off an email to you with an accept or deny button in the email to even allow them to start work. And these are critical systems that you're allowing remote access into. But then, to take a step back, it goes back to policy. If I'm going to allow you remote access to my systems, do you meet these security requirements? So if we look back, I don't know, six, seven years ago, sorry, I got teenagers, so.

Aaron Crow (53:49): You obviously have kids like I do. I get it. I've got a 12 year old, a 14 year old, and a 17 year old. So I can't say the numbers six and seven in the same sentence ever, because I want to strangle my especially 12 year old. But yeah, go ahead. If you have kids, you know what we're talking about.

Scott McNeil (54:08): So stupid. So, look at the Target breach, right? Do you remember the Target breach? Do you remember how the Target breach happened?

Aaron Crow (54:14): Yeah. A hundred percent. The supplier, the back door.

Scott McNeil (54:20): It was their HVAC vendor. Their HVAC vendor, who controlled their own remote access into a store. And it was the HVAC vendor who originally got compromised. So while Target's network was absolutely horrible because HVAC was on the same network as their card readers for their payment systems, yeah, that's some serious PCI compliance violation, but don't get me started there. After working at a college for a long time, you have to understand PCI compliance, and that's no joke. It was the HVAC vendor systems that were compromised, and they got in through the HVAC remote access.

Scott McNeil (55:01): If that doesn't tell you how important that is. And that's a Target store. Who cares if you're not going to be able to get your item on sale that particular day? That's not going to make something go boom. When you look at a complex chemical process and you get compromised via remote access, and they get in there and start shutting things down in an improper fashion, if you're not closing a valve or opening a valve in time or maintaining a proper temperature on a chemical exchange, it's explosions, it's damage to property, it's potential loss of life, it's loss of reputation. It just goes on and on. So that's just another huge thing that the site needs to take ownership of and start dictating to their vendors. That was a really long explanation for that one sentence there at the end.

Owning Your Standards and the Reliability Side

Aaron Crow (55:55): But it's so true. As an asset owner, a previous asset owner, I was that same way. I pushed my vendors, and this was back in the 2015 time frame. No, you're going to follow my standard. I don't need your domain controller. I have a domain. You can add your group policies to the domain that I will build. I don't want your firewall. I have a firewall. I will put in what policies that I approve for your segment of the network, but I control the firewall. I don't want yours. Well, we'll just put it in there anyway. No, you're not, because I'm not paying for it. I don't want it. It's not going to be there.

Aaron Crow (56:23): The other piece that goes beyond. There's the cyber side of things, and we've spent a lot of time today on the cyber side. I want to be conscious of the time because we're nearing the end. But the other piece is reliability. The other side of this conversation is every one of those, every time I have a media converter, every time I have fiber running out there, but my old OT networks only support copper. So I go copper into a media converter to go to fiber, then fiber long haul to another media converter that goes back to copper, because I don't want to put in a switch, because a switch is literally a media converter. But no, I'm going to get one that I buy on Amazon for fifty bucks because it's cheaper.

Scott McNeil (57:25): Or they put in an unmanaged switch, which is just as bad, because you get no visibility and you're forwarding everything.

Aaron Crow (57:29): Correct. Just as bad. But the point I'm getting to, and that I want folks to understand, is yes, cyber, everybody's worried about China and Iran and all these different nation state attacks, and all those are valid. We're also worried about, yes, you were nice and this was your client, so you weren't trying to break anything. But some kid that's got his laptop and just happens to see those things and is watching a YouTube video on how to do this stuff, or he just bought a Pwnagotchi or a Marauder and he's just playing around with it and happens to be close enough to your network that he sees it. He or she may not know what he's doing, or maybe they do, and they do bring down that network. So some of this is reliability. Some of this is, hey, we need to be sure that we can function, that this thing works.

The Zip Tie in the Fan: Why Monitoring Matters

Aaron Crow (58:24): One of the first things I did, way back in the day before there were OT products in the space, and I've told this story a hundred times, so if you've heard this, guys, I'm sorry, but you're going to hear it again. We started monitoring these OT networks that were built by the vendors. They were super redundant, dual stack switches everywhere. Everything went to both switches, a true redundant environment. It was gorgeous. I loved it, except that they had no monitoring. So we turned on monitoring. We configured spans on all the different switches. We configured syslog on all the switches. We turned it on, and within five minutes, there was this one switch where 90% of the logs were coming from. So we started digging in. What is this one switch screaming about? And it was power supply failure, power supply failure, overheat, overheat, overheat.

Aaron Crow (59:22): So we were like, where is this switch? We figure out the name of the switch, look at the drawing. Where is this? We go find it and walk up to the cabinet. We open the cabinet, and of course it's just blinking error code, error code. Nobody ever saw it, because it's in a rack that's black and nobody can see it. We open it up. We're like, man, what is going on with this thing? Why is it overheating? It's not hot in this room, it's air conditioned, there's no reason. So we walk around to the back by the power supply, and there was a zip tie in the fan by the power supply. So we grab the zip tie and pull it out, because it was stopping the fan from spinning. We pull the zip tie out, the fan spins up, the heat goes away.

Aaron Crow (60:05): It'd been screaming for who knows how long. It didn't cause a problem because they had redundancy. There were two switches in a stack. The backup was not working, the primary was. So it never caused an issue, and it wouldn't have caused an issue until, if that primary switch failed. They thought they had redundancy and they really didn't, because that other switch had basically shut down everything. If the primary switch had failed, all this to say, the reason I bring it up is all the things we're talking about. Obviously I focus a lot on cybersecurity, but also the why. Nobody wants to do cyber because of cyber. You and I do, maybe. I'm a nerd with cybersecurity. I enjoy it. But most people don't want cybersecurity, just like you don't want insurance for your car, but you get it because you don't want the impact of, if I get in an accident, I don't want the liability of the damage to the other vehicles, the physical property, the health and all that kind of stuff. It's the same thing. Cyber is like that insurance policy for you. But on the flip side, a well-architected network, whether it's wireless, wired, whatever, it helps your reliability. And any of these things can impact, from a misconfiguration to a bad actor. The impact is my system is down, and that's all that should matter.

The Wireless Footprint That Leaves Your Site

Scott McNeil (61:31): Absolutely. It's nuts, man. It's awareness of what you've got. That's what makes inventory and that knowledge of everything that you have and having that recorded so important. There's so much awareness you need to have when deploying wireless systems in the industrial universe, because there are so many different interferer types, from machinery to drives, so on and so forth. But not only that, when you start deploying wide swaths of wireless across your site, you also need to take into consideration your footprint. Because most sites don't realize how far that wireless is going off their site.

Scott McNeil (62:30): When you have a woodyard that's depending upon anti-collision crane systems and other wireless aspects for the woodyard, but you're right next to a neighborhood, all of your wireless is on top of cranes. How far is that going? You can go to the neighborhood and sit next door and nobody is the wiser. You're just a car sitting on the street. But you're seeing all that traffic. You may not be able to connect to it, but you can sit there and capture it, from a security aspect. So it becomes that much more important to make sure you're securing everything, but then also understanding what that wireless footprint looks like and doing what you can to mitigate that. There are so many different things. This is why I have my own podcast, to talk about all that stuff.

Call to Action: The Industrial Wi-Fi Shop

Aaron Crow (63:19): So let's do the call to action then. We could talk for freaking weeks about this topic, all the things that are networking and wireless and infrastructure and OT and all these different spaces. Obviously we scratched the surface today. Man, I've enjoyed this for sure. How do people find out more? If they're interested in wireless and OT, if they're interested in figuring out what questions to ask, tell them about your podcast again. How do they learn more information? All the calls to action for folks to check out the more details.

Scott McNeil (63:57): Sure. Well, you can find me on LinkedIn, Scott McNeil on LinkedIn. And the mothership, industrialwifishop.com. That's where we host everything. Jeremy and I have a downloads page on there for some best practices documents that we are writing. I'm always working on some kind of white paper to add to it. Some handy dandy little charts that are always nice to have, milliwatt to dBm conversion tables and stuff. And then the show can also be found on Apple Podcasts, Spotify, and here recently I've started working on the YouTube channel. So the past couple episodes are full video. All the episodes are on YouTube, but only the past few are actual full video. I'm trying to start leveraging that. My massive thirty views for a show, but hey, I'll take what I can get. The audio on Spotify and Apple does really well. I have listeners all over the world. But the YouTube thing is a newer thing that we're working on. I've got a face for radio, so I can't say that I really blame anybody.

Scott McNeil (65:02): But yeah, we dive into all different aspects of wireless in industry and manufacturing, from security to applications to actually talking to different vendors and understanding what their devices are. We don't get salesy on there at all. We talk about all the different technical aspects of things. We had a really good episode with the ProSoft guys, and I've got a lot of good friends who work for Phoenix Contact. Several of those guys have been on multiple times because they have a lot of really great industrial wireless innovations, like wireless power transfer, which is amazing. And then just being able to pull in other people in the wireless community who have done work in manufacturing. And then talking to guys who are developing things like Wi-Fi HaLow. We had the guys on from Australia and it was all about Wi-Fi HaLow. It was amazing. We cover such a wide swath, but it's all, well, to me anyway, that's why I do it, it's super interesting. Because you have this gigantic wireless community, but we really are like a niche subset.

Scott McNeil (66:45): Anybody in OT gets that, because while we use a lot of the same equipment, we have a completely different operational philosophy. So we're kind of our own little tight group there.

The OT Community and Closing

Aaron Crow (66:57): A hundred percent. Well, that's awesome, man. This has been a great conversation. I think we could probably do another one, and maybe we should dive further into a specific topic at some point. I appreciate your time. We'll put all those details in the show notes, everybody. So definitely check out Scott and their wireless podcast, deep dive into all things wireless and OT specific type stuff. Reach out to him on LinkedIn, all the things. That's the one thing I love about this community, everybody that I get to meet and run across. Almost everyone I've ever met in this space is open and willing to share. It's not a competitive, well, I'm not going to tell you because then you know how to do it. No, no, here, I'll give you the blueprint. You can go do it. It's fine. If you don't call me, you don't call Scott, do it yourself. Please call someone and do this, because it's too important not to.

Scott McNeil (67:43): That's right, absolutely. Because sharing the information makes us all better. That's a huge difference between IT and OT. And that's one of the reasons I'll never go back to IT, because we're not kingdom builders. We want everybody to make sure their stuff works. So good stuff, my man.

Aaron Crow (68:06): A hundred percent. Awesome, man. I appreciate it, brother. Have a good one, and definitely reach out to Scott and check out his podcast, y'all, and the new video on YouTube. Awesome, man. Thanks a lot. I appreciate it.

Scott McNeil (68:18): On YouTube. Yay. Yes, sir.

Transcript lightly edited for readability.