Million-Dollar Buttons: How Offshore OT Is Embracing Containers and Modern Cybersecurity

Intros

Aaron Crow (00:06): Hey everybody, thank you for taking time joining me on another episode of the Protect It All podcast. I'm always excited to talk to new folks I've never met before. I told the guys today that I use the product in the background. You can't see it because it's virtual, but it's running on systems in the background. So we'll dive into the thing. Matt and Josh, why don't you guys introduce yourselves, tell the audience who you are, a little about your background and what you're doing nowadays.

Joshua Herr (00:43): Sure, I can start it off. I'm Joshua Herr. I've been working in tech for about fifteen years, kind of all over the place. I started in structural engineering software development, moved from there into the cloud engineering space, and from there it's kind of a natural progression into security. It's been a wide breadth of different technologies and environments over the years, and that's helped a lot with security, because you see some very obscure environments and use cases you don't often come across in the wild. I've really enjoyed it. I love where I'm at today, and the security space is actively exciting all the time.

Matt McLendon (01:28): Mine's way less succinct. I'll try not to date myself, but mid-to-late nineties IT, core discipline at an MSP and ISP locally in the Houston area. Linux systems administration, Windows systems administration, really gross references like IPswitch products. I've physically played on Sun SPARC and SunOS, old IBMs running Solaris, things like that. Then military component electronics at the intermediate level, FLIR and radar, for about six years. Then sixteen-ish years in offshore oil and gas, ranging from a major OEM to consulting to a deepwater drilling contractor: primarily drill ships, but also semi-subs and jack-ups at different stages of my career. And now I'm doing what they never say oil field hands can do: I'm a product manager for a software startup called Portainer.

Aaron Crow (02:39): Which is awesome. I talk about this a lot. It's always amazing to see the trajectory of people's careers and where you started from, especially for those of us with some gray in our beards. Very similar to you, Matt: I've been in this for 30-plus years. Mid-nineties is when I started, with token ring and coaxial network design, Exchange, Windows NT 3.51, Novell NetWare, all the way up to AI and everything I'm doing today. Mainframe to AI and everything in between.

But to your point, Josh, there's value in that. There's value in how we've done things in the past, and the gotchas. It's not like a new product comes out and we haven't solved this problem before. I gave an example earlier: when we first started using virtualization in OT, that was like 2015. Virtualization wasn't new in 2015. It had been around a long time. OT was just still not an adopter. Today we're talking containers and Docker. Again, not new things, but OT is sometimes scared of them because they don't necessarily understand them. So how are you guys approaching the OT space with these capabilities? Not even new, but new for what people are actually doing in the OT world.

Finding Portainer as a customer

Matt McLendon (04:49): I met the Portainer guys because I approached them as a customer. That's how I ended up here. I'd used it from a home-lab perspective since probably the 2019 timeframe. My prior IT experience was virtualization focused, VMware certified and that whole thing. I had a very large cluster in my house just to play with and learn that ecosystem at my own pace, and what I was used to was abstracting everything with VMs.

When I started tinkering with some of the early LLM and AI and ML subjects, they all tend to be Python projects. At the time, things like uv didn't exist, and you end up in this nightmare where the Linux kernel uses Python too. If you goof up one command and overwrite your OS's environment... people were writing tooling not assuming it's in user space. An ecosystem rife with pitfalls, we'll say. One of the mitigating ways to deal with that, at least to get some abstraction, is the container route, without having to spin up VMs every time.

I was doing containers from the CLI, then started looking at multi-host. I tried one tool, it's deprecated now, and it was absolutely terrible. I'm used to loading up people's little pet projects and having them fall on their face and require weeks of tinkering. I found Portainer shortly after that, installed it, and it worked. It just did what it said it was going to do, and I could get on with what I wanted to achieve. That was not something I was used to from an open source product at the time.

The sensor quote that never delivered

Matt McLendon (06:44): Then, working for my prior company, Seadrill, I was fed up with some vendor quotes we'd received on a pretty simplistic scope: add sensors to better track our fuel consumption offshore and tie them back into an existing control system that had spare IO. Sell us six sensors, we'll handle class and the pipe work and field cable, all the labor-intensive parts. Tie them back to your control system, display them in the UI, and transmit it through the same medium we get the rest of our data from. I think the quote was in the hundreds of thousands of dollars and twelve to eighteen months of delivery time. When I left, that scope had been pushed forward and I think it still hadn't been delivered to that particular vessel.

I'd seen it throughout my oil field career: people getting taken advantage of. The people that own these vessels are the maintainers of them. They lean on the OEMs, they lean on system integrators, and they get stuck. Add a single tag to a physical chain that already exists, but the tag's just not there: 15K for the software change privilege. Things like that.

So I said, okay, the usual conversation is the security implications. Let's just de-risk. How do we de-risk it? Let's not touch the control system. Do we need a sensor? Cool, let's get the IO too. Does it need a PLC? Can I tie it back to conventional computing in some way? What's the most elegant way to do that? Through sheer happenstance I learned about Portainer's free node tier. I used my work email so it would look more legitimate, and they reached out and started telling me about the edge and industrial features and roadmap. I never even knew about it. Walk me through it.

Deploying to a rig without setting foot on it

Matt McLendon (09:35): It turned into such an easy, repeatable process. The usual thing would be: I build a widget, it solves a problem offshore, I standardize it, spec out all the parts, get the quotes and vendors, and either I get drug out there or we subcontract someone to do the install. That's fine for one. What about the fleet?

For the proof-of-concept rig, deploying data flow from some switchboards for the system I designed, I didn't have to set foot on the rig. For the initial setup I had a hardcore, knuckle-dragging offshore electrician doing the cable runs for me. He did the initial boot-up, read screens back to me, got connectivity set up, and even troubleshot some firewall issues outside of what he normally deals with. Both containerization and the distributed capabilities of systems like Portainer: the repeatability is massive when you can't walk to the other side of your office to troubleshoot something, or even remote into a terminal.

From our perspective, a rig should be able to have the world ending outside of it, and as long as they have food and fuel, they're achieving technical uptime and can be autonomous. I've absolutely been on a rig where we were without VSAT for eight days because of storms. You get into precarious scenarios: do we need to start paper-logging stuff? What does it look like when we reconcile all the maintenance data? People make assumptions that everything's a building they can easily go put hands on when something goes wrong. Front-loading the effort to make things repeatable and reliable is a huge value proposition.

You can kind of do it with VMs, but there are a lot of moving parts where you can trip up, and the time constraints: you're dealing with what level of abstraction the hardware is, you have faux boot time. Why is that a thing? The hardest concept, and this was a struggle for me coming from virtualization and bare-metal sysadmin too: unless it absolutely needs otherwise, treating every stage as ephemeral or programmatic makes for much better system design. The drum I always beat is de-risking and parallel interaction.

Don't touch the control system

Matt McLendon (13:22): It's a legacy control system. It's unpatched. It's not going to get patched, because it's either cost-prohibitive, or the patch doesn't exist, or that company isn't around anymore. But you're not going to make the data request go away. They want the telemetry, they want to monitor it, you want the maintenance data. So what creative ways can we do that?

The amount of conversations that start with "we're gonna put a tap in here." No, you're not. You were not a class-assessed part of this control system. You're not going to change its design and introduce failure modes. You're just not. We had people giving us assurances about taps, and I've absolutely seen a failure mode on a network tap where, no, it didn't lose connectivity when the tap lost power, but it reset connectivity when it came back on. That's a non-starter. "It's a bug, we fixed it." Why would the sample unit you gave us... you're telling me you didn't test it? People just don't think about that stuff the same way.

Aaron Crow (14:33): The implication is so much larger in an OT space. As an asset owner, I was at one of the large power utility generators in Texas, and I had plants from Odessa to Paris, Texas, almost to Louisiana, as far south as San Antonio. My office was in Dallas. It could be eight hours to get from plant to plant. If something breaks, and I've got a team of six and 40-something power plants, how do I support that? I don't have a corporate jet. I drive pretty damn fast, but not that fast.

And the other piece, Josh, you hit on it earlier: as cyber practitioners, you see the fear selling. China's gonna attack my thing. Okay, that probability is not zero. But what's more likely to happen? Something's gonna impact my operations.

The million-dollar button

Matt McLendon (15:51): That's what I try to tell people. There's a button on offshore oil rigs that someone can just walk up to, and no one's manning the room, that at minimum is costing that operation a million dollars if you push it. So what is a USB blocker doing for anyone? But these are the approaches people default to, because the solutions were coming from conventional IT, and either the contextual conversation isn't happening or it's just visible compliance. It differs from enterprise to enterprise, but the OT guys aren't going to be super supportive of offering ideas when that kind of thing's being mandated. They're the ones that have to walk around and stick them in. And then: "I have 300 of these removal things. Where do these go?" Every blocker kit is four blockers and a removal tool. Not that security theater doesn't exist in conventional IT spaces too, but it's particularly frustrating.

Aaron Crow (16:55): But the impact is different in OT. And that's what I'm excited about with conversations like this: the art of the possible. Yes, there are cyber capabilities, analytics out of the environment, vulnerability and threat detection, all that. But that's not how you sell the rig manager and the operations guy on this tech. Nobody gives a shit about your firewall. They care: can I be more effective? Can I troubleshoot? Can I get my rig back up and going faster? Can I make more money? Can I get better maintenance? Can I find failures before they happen, so I stay up instead of having a trip or an outage? Those are the conversations we need to be having. Technologies like this get pushed off because they're presented as "hey, I'll get you better dashboards for cybersecurity," and the operations guy is like, I don't care. I'm in the middle of the damn ocean. Half the time I don't have internet. What are you talking about?

Death by alarm

Matt McLendon (18:03): And in industrial spaces, human-factors design for HMIs and control systems is much more mature than conventional IT. Death by alarm is quite common. If you can't do meta-alarms automatically, you're already thirty years behind. You can't give someone the fire hose and expect them to be enthusiastic about it regardless of what visibility you're showing. It needs to be quick, actionable, and not a nuisance, or it will get ignored. It's human nature. It gets normalized and they move on. If the noise, no matter how annoying, doesn't tell them what needs to happen right then to address it, people go, "What would I do with that information?" And they end up left vulnerable because of that.

Aaron Crow (19:02): You walk into a control room, and on these custom keyboards that are like ten thousand dollars for the older control systems, there's a mute button. Nine times out of ten that mute button is the very first thing that breaks on every single one, because they just smash it constantly.

Matt McLendon (19:27): It's just mashed. Alarm acknowledged, alarm acknowledged. It'll have the text ripped off it, the little waterproof barrier blown out. We've seen it from a systems perspective. I've been part of major contract preparation where we had discussions about visual clutter for the person operating the equipment, because every vendor comes in with their little operationally critical system, and the operator needs to see it contextually while he's running the tool. They were just filling up his visual space. Guys, he's got to be able to see the tool.

Industries like this, because we're designing for cohabitated spaces of heavy equipment and people, you always have these weird tight areas where you have to consider the full scope of what it looks like for somebody day to day. Are they cussing your company's name? Are they shaking their fist at you? Do they not know who you are because you fall into the background? Or are they thinking "his company spent the money on your product" every time they use it?

Rolling iron vs spinning containers

Aaron Crow (21:14): Diving further into Docker and virtualization: the space problem. When I'm rolling out hardware across, I did a large program for a power utility and we rolled out across three thousand sites. That was a lot of iron. Imagine if I had to roll iron every time I wanted to do an update. When you start talking containers, I can have one box, it does X, Y, and Z today, and when new capabilities come out tomorrow you spin up another container. I don't have to go back to site, because I've already got something there that can be my host.

Matt McLendon (21:55): Specific to oil and gas, there were a couple of vendors trying to do this, but I always felt they were trying to black-box a little too much, keep a tight grip on it. They wanted licensing fees for the vendors involved, when in reality the end customer could dictate this stuff. If you have a shared services platform, let your IT do the base level, bare metal up to VM, set security policies for the OS to namespace to user space, and containerize to your heart's content. And steer your vendors: why does your software need that permission? It's going to get taken away. Does it actually need it? Show us why. Make them adapt. In a container world, sometimes the answer is just them changing their deployment template. Then the speed to new features, concurrency of deployment: you can horizontally scale at eye-watering density and speed without adding a bunch of physical hardware.

It alleviates the usual thing, which is either everything's a SaaS, which some industries just won't work for, or everybody's got their little edge PC they want to upcharge you for as part of their deliverable. And in eras like COVID, those were unobtainium. IPCs were just gone. We had projects pushing dates out 18 months for small component electronics. The closer the component got to consumer grade, the worse it was.

DC fault propagation: detail engineering matters

Matt McLendon (24:16): Every industry has its characters, and dynamic positioning has a lot of them. There's this great guy, in retirement, who fixated on one failure mode and built a little simulator for it: DC fault propagation. Common approach was, on a DC rail you could have a bunch of redundant power supplies, because you're not phase-locking them to each other. You provide for redundancy, and what feeds them can be redundant also. He developed a rig to show DC faults could propagate across redundancy groups via a common DC rail or shared voltage areas. That eventually got pulled into class standards, which makes it to insurance companies, which makes it to class renewals, which means it's mandated. Now everywhere this happens, they have to segment them all.

When you're adding a widget, it uses power, AC or DC. Is there available capacity? Sometimes the answer is no. It's not just physical space; it might be load space. Is this causing risk to other equipment just because we plugged it in right here? Service ports and cabinets aren't meant for just anything. There's a lot of detail engineering in control systems that tries to account for every failure mode of every component in that cabinet. When you add some random thing they never knew existed, are you ready to be the one they come to with your box smoking, saying your system did this to my facility and caused this much revenue loss? I don't think people consider a lot of that.

Joshua Herr (26:08): That's an interesting point, because some of the stuff we're moving to in the broader technology space is running into these same problems. We're working on some projects with battleships, and DC power supply there is very particular. You have to spec out all the DC power constraints, because the last thing you want is for the battleship to lose power because you pulled too much, or you blow out all the breakers. Those considerations at these low levels are becoming more prevalent, especially as we start talking about data centers in space, satellite concepts, things you can't reach to any capacity.

Matt McLendon (26:56): Highly integrated systems like that, a vessel, a spacecraft, an airplane, go through full integration testing because they have follow-on effects to other systems, especially at power distribution. They bake in safety factor with all the detail engineering before anybody buys a piece of kit. Coming in after the fact and adding an edge AI compute node that's pulling ten times the wattage of a normal little IPC maybe wasn't accounted for. And the default right now, from a security perspective too, is that the person responsible for upkeep and maintenance and logging of all that equipment is the one everybody expects to push back. I've personally told people: no, you're putting in a whole rack, you're not co-locating in our control systems racks. We'll pull you a breaker if you tell me your loading, and a network drop, but you're not going in our rack. Floor space is at a premium, and cooling for that matter. There are a lot of dependencies beyond somebody going "hey, let me plug this thing in."

Why you can't just install Windows 11

Aaron Crow (28:24): That's a big piece many people miss. I travel around, DEF CON Singapore, DEF CON Vegas, RSA, Black Hat, and I have these conversations with people, especially from the IT side, who don't understand all the reasons. Why don't we just rip everything out and put in Windows 11? Beyond the obvious: imagine how much power and space, all these factors you're not thinking about. And the risks. I'm dating myself, but Novell NetWare, that stuff would run. I remember systems that had been running for ten years and never rebooted. There's probably not a system alive in an IT world like that; we're patching and rebooting them all the time. Many of these OT systems you literally don't touch. It has just been running. Leave it alone and it'll run forever, as long as it doesn't fry out electronically.

We don't have that level of rigor in traditional IT systems. They're designed to be replaced every two to three years. Software end-of-life, no parts anymore. OT is a vastly different environment from a space and power perspective. The hardest thing to find at a power plant is clean power. As ironic as that sounds, you can't find any power at a power plant.

Joshua Herr (30:05): The way I always think about it is breadth versus depth. In OT the breadth is very complicated: more dependencies to consider, power constraints, physical capacity and space. You don't typically deal with those in IT. But then there's the depth too. We come across things all the time: this company's running Windows Server 2012, which is end-of-life and not supported, and the kernel modules are all really old, so installing new software is almost impossible. But the manufacturing vendor only built for 2012 and not for newer versions. So you're in these OS-locked areas of OT, and stack-locked areas. Someone on the physical plant has specific security concerns around physical proximity, but they're not thinking about the networking layers on top. Someone else is thinking about networking. You end up building all of these gaps into your security stack, because there's a different person at every layer thinking about them, and no one thinking from that 360-degree view. We see it all the time.

Matt McLendon (31:26): The minute you let the floodgates of vendors through your gates, the expectation is that you're the filter, and that person may not be there or have the expertise to ask the right questions. The vendor may not know either. I've seen quotes in the eight and nine figures where a Windows Server version that was already end-of-life, or about to be, was included in the scope by name. I had to explain to the vendor that LTSC existed. It was just their default, what they were comfortable with. Microsoft won the general familiarity wars, so you end up with systems designed on something people think is supportable because they know how it works, but it's too general-purpose.

The real legacy control systems were very narrow-scope operating systems. When you don't have a bunch of interaction that isn't required, that's complexity reduction. It adds stability and reduces security exposure to an extent. They can go a really long time with appropriate documentation. I love those, it's like reminiscing: you go into the recovery document and see old DOS commands, and you're like, ooh, this is going to be an interesting one to tinker with. I had a colleague consulting on a platform whose primary drilling control system was running on a precursor to NT. Windows 3-based, I think NT 3.5.

"If it ain't broke, don't fix it" and the airplane test

Aaron Crow (33:33): I've literally walked into plants of all kinds and had Sun Microsystems or an NT system still running. An IT person walks in and says, how could you possibly allow that? And the answer on the OT and operations side is: if it ain't broke, don't fix it. Why do I need to upgrade it to something I have to patch, that's unreliable? This thing is so old it's actually probably more secure than newer things, because it has like two vulnerabilities, and it's not on the internet. It doesn't have the capacity to do anything but what it does, and that's all I need it to do. I can put another device, like what you guys are talking about, that gets me the data out of it without replacing this thing. Let's keep this thing really reliable.

The analogy I always give: you're in an airplane, and the airplane control system has a vulnerability. Do you want them to patch it while you're in the air? Or do you want them to land the plane? Let's land the plane. But take that a step further: do you want them to replace the control system with a Windows 11 device? I don't. I'd rather keep it on the device that's been running in a Boeing for 30 years. Is it the most secure? No. But is it reliable? It flies planes really, really well, and they've worked out lots and lots of bugs.

Matt McLendon (35:06): That reminds me of my favorite example. I thought I was escaping IT when I joined the military. Then I got to my first duty station, and there's this particular workbench that plagued my entire military career repairing electronics and FLIR. I remember reading one of the bench's boot and troubleshooting steps to recover from an image, with the big cartridge CDs. Someone had just made a Word doc with the commands. I'm looking at it going: this thing's running Linux. It was the most basic switching-to-the-kernel-on-the-disk to boot and copy over. No one there had any context for what they were typing. They were just repeating what was on the page. That was probably my first time at that CapEx level realizing you really can just build anything you want with what's out there, and nothing should be a black box in that regard.

Aaron Crow (36:20): Security by obscurity.

Joshua Herr (36:20): I think all three of us can lament that Microsoft kind of won the war on the operating system.

Matt McLendon (36:29): Grep is going to be part of PowerShell now. We've come full circle. They're taking open source CLI tooling and pulling it back into PowerShell. And no, I don't mean WSL. In PowerShell you'll be able to grep stuff. Don't tell them about ripgrep. They're already thirty years behind, I guess. Gotta start somewhere.

Joshua Herr (37:10): But that's what we see all the time. You get pin-locked into these Windows operating systems, and Windows is notorious for poor forward compatibility. They break their kernel drivers all the time, so Windows is very finicky if you want to upgrade the OS. Linux in particular is a lot easier. I think parts of the OT space are starting to shift that direction because they're realizing it: you get more stability and the ability to upgrade if you move to Linux. We've been encouraging that for some of our OT customers.

The missing generation of sysadmins

Matt McLendon (37:48): But we did kill an entire generation of that detail curiosity. In the nineties we didn't have the bandwidth to download a Linux image, but we still managed. I remember compiling the kernel from the mailing list. That wouldn't work today, but it was such an eye-opener, and as soon as someone heard about it, you had the mega-thick books you were digging through. It's a mindset I feel like we lost. Microsoft pushed the gas pedal, the whole world got familiar enough, and they got locked in. It's having a weird resurgence because of gaming and Windows being particularly aggressive now, but it's going to be a ten to fifteen year delay before we see those people in the workforce.

What we've seen in the industrial space is they lack that middle generation of sysadmins. People with enough shell scripting, because your best sysadmin is your laziest, right? The one with his scripts and cron jobs done so he didn't have to do anything. That's who would be designing these base OS and firmware images and keeping them up to date. What's happening instead is the embedded OS route, extrapolations of DD-WRT, then more polished: PTXdist, BusyBox, and Yocto as a much more structured approach. I am seeing that, even at that layer.

But then they go a little too deep in my opinion: as soon as they make the transition, they want rootless everything. You can bifurcate your implementation and cause yourself fewer development headaches. Since being at Portainer, going to conferences from this side and talking to hardware partners: the number of people that either have options in that realm or are moving in that direction, then quickly progressing to containerizing, is significant. They're still very much on legacy ARM architectures, stuff predating ARMv7, which makes it rough sometimes. You might have an ARM-compatible container, but it won't work unless it was specifically compiled for it. From a containerization perspective, that's something you can feed back to a vendor you have an impactful enough relationship with: I have these ARM devices, can you compile an image and tag it? Some will pursue it, depending on how much of their customer base you represent. And Yocto seems to be the flavor of the last few years, as a way to handle security better, with whitelisted repos and package installers, giving extensibility to the customer without letting them tinker with everything.

37 devices phoning home

Aaron Crow (42:03): The irony: I literally just did an assessment on a site. I had multiple sensors out there grabbing PCAPs off network switches to get network flows, multiple data points, a terabyte of data I'm going through, just on the OT side of the network. And as I'm parsing through flows and PCAPs, there are like 37 devices on the OT network that's supposed to be isolated, and they're all phoning home to Microsoft. We've got to cut these things off.

But we're scared on the Linux side because, like you said, we don't have these admins. Those sensors in the backpack there: to your point, I'm the lazy guy. I've created scripts. I literally pull up a script, it gives me a menu: are you starting a new one? It wipes it, builds it from scratch, wipes all the data off, puts it at the latest version, and it starts, and I put it back in the case. It uploads the data for me. I've got parsers built in. I did it once and didn't want to do it a second time, so I spent a little longer scripting it, and the third time it's just pushing a button.

Matt McLendon (43:32): You can say Ansible, you're not gonna hurt my feelings. We solve different problems.

Aaron Crow (43:39): It's not even that. We've got Portainer running on it, local scripts, Raspberry Pi boxes, OnLogic and other hardware vendors, DIN-mounted DC-powered Raspberry Pis with inputs and outputs like a PLC.

Matt McLendon (44:05): One of our partners, and it's a work in progress, has one of their fancier power supplies with an optional clip-on thing that snaps in and gives you an MQTT or OPC UA client endpoint. Little web UI, dead simple, probably running on a microcontroller. From their perspective that's maintainable code, very simplistic, right-sizing the device. Not super secure, but if they're at that network layer, you're already in the envelope at layer one. It's when you egress that you need to tamp down things phoning home, or control the endpoint you put into that space for whatever value-add you're getting: extensibility of the existing system, modernization, security logging, data analytics.

You can't afford the downtime

Matt McLendon (45:29): Oil and gas is one of the industries where people say, you guys could afford it. The reality is, even during the times when we could, you can't afford the out-of-service time to do the upgrades. So you prioritize some and start triaging. Even the control system manufacturers have these staggered hardware lifecycle extensions, just like the software vendors: pay us this much more and we'll literally hold these controllers in a factory in Norway for you, based on their failure rate, so you're covered for this amount of time. And at the extremity: we had a rig getting cold-stacked, possibly turned into razor blades, and we sent people out to gut physical hardware. We were pulling VFDs out of a rig to keep another whole rig running for a contract period.

If that's the kind of stuff we're doing, no, I don't care about your patches on your IPC. Is it working? I'm not gonna touch it. So what's the actual solution? Because to say that software development, IT, distributed systems are inflexible in finding solutions that satisfy dependencies and security is silly. That just means you're not trying. There is always a creative way. Is it an arms race? Absolutely. But that's the most fun part: being clever enough to be a couple steps ahead, or able to tell when you fell behind, in a safe manner.

Horizon question: one exciting, one concerning

Aaron Crow (47:20): This is usually my wrap-up. I used to ask about the next five to ten years, but things are changing so fast that's a dumb perspective. Who the hell knows? What's one thing coming over the horizon you're excited about, and one that's maybe concerning? Both of you.

Joshua Herr (47:47): The thing I'm probably most excited for, candidly, and I teed this off: I'm seeing a big switch from the Windows ecosystem into the Linux ecosystem by and large. There are a number of factors. I thank Valve for a decent chunk of helping with Linux. That one's the most optimistic to me. Where I'm more concerned is definitely the AI front. I feel like we're creating new exposures and attack surfaces everywhere, and we're not closing a lot of them. We're just building up more exposure over time. AI is another vector of attack exposure that we have to deal with, and we're not currently.

Matt McLendon (48:38): I agree with the Linux transition. I hope the people with the purse strings behind that realize that even though it's been the historic open source project of our lives, thankfully with a benevolent dictator, go look at the maintainer roster and look at their email domains. Every blue-chip company you've ever heard of that leverages Linux contributes their employees' time to maintaining open source projects. If you keep wrapping them up into your industrial products, some of that has to go back, or you're putting yourself at risk down the line if you don't in-house the expertise. It's not a free MIT-license lunch. My first Linux distro was Slackware 2.3. My entire professional life has had some touch on it, and every time I think I escape it, there it is again. It really does run the world.

On AI, I'm interested not because of what people are doing with it, I have an intimate understanding of how it works, but in how it changes the way people think. Especially in open source, it has the potential to hurt us. I feel like we're moving into what's going to be a post-trust ecosystem. Without vetting, or some set of automated tools for ensuring secure deployments, or at the extremity catching malicious code, I think it's going to hurt us overall. Let's be honest, npm was going to happen. It's the most hilariously insecure-by-design thing, its creator regrets the way he structured it, and everybody keeps band-aiding it. And did you see this morning? Now the AUR too, and it's still propagating via npm. More of that's going to happen. We're at the whims of front-end developers and JavaScript right now, but as time goes on some of that falls back to the more core, machine-layer languages, and that could be rough. I don't want industrial people to get freaked out and revert to something they think is more supported or secure when it's actually probably worse; we just don't hear about it because they're allowed to keep it secret.

I'm interested to see what systems and assessments people develop to navigate the new landscape. And I have some personal ones, from stuff we're working on that I'm actually surprised and excited we're moving forward with, touching all these subjects. Josh knows what I'm talking about. We're working with one of our partners on integration work and scoping for customers to make that transition smoothly: from bare metal or VM, regardless of the deployed solution, into VM first, then containerized, repeatable, and programmatic, with security as a native part of it. Whatever you're pulling from can be secure at the moment of deployment, and you don't care where it came from. Kind of like how I throw a random AI tool of the week on my home server to play with, because I've appropriately isolated it, but at an individual container level. Use LLMs for what they're good at, which is pattern matching. Even hackers have primitives. At a tooling layer that doesn't care whether it's a container or a VM. Not giant SaaS running at ring zero, because we've seen how that doesn't work well. I think we're rife for solutions of that nature that aren't a cash grab and are actually trying to be at least somewhat altruistic, given the landscape we're moving into.

Call to action and close

Aaron Crow (53:36): A hundred percent. What's the call to action? How do people find out more, use more Linux, more containers, more Portainer in the OT space?

Matt McLendon (53:52): We're portainer.io. I'm our edge product manager. If you engage with us in a professional capacity, I will end up talking to you in some form or fashion. I talk like this in our initial customer meetings. I love everything about my career history and the tooling and skills I've learned over time, and I love brainstorming architectures to solve problems. I think one customer onboarding call went three hours one Friday, just talking about different approaches he could take with his architecture. I thoroughly enjoy it, and I feel like all of our FAEs are in the same boat. We're about solving problems and making it so people can spend their brainpower where it adds value, not maintaining their infrastructure. At the end of the day, infrastructure's fun because I'm weird, but if your job or passion is delivering a software product or service, the infrastructure should be what you're relying on, not what you're managing.

Joshua Herr (55:07): And I'm over at Xiid, that's xiid.com. We're doing a lot of work right now with Portainer around connectivity and post-quantum resistance, which is actually less important for OT spaces mostly, but still very interesting work. If you reach out to Xiid, you'll definitely end up talking to me, because I do all of our customer integration work, and that's my favorite thing to do: dive into those environments.

Aaron Crow (55:33): A hundred percent. We'll put all the details in the show notes, guys, and definitely reach out. It's exciting to see all the different things we can do. I absolutely think quantum and all those other pieces are coming to this space as well. We just happen to be behind the eight ball, and some of these things we can't be behind too far or it's going to impact us negatively. We need to be thinking about things left of bang instead of right of bang.

Awesome. Well, thanks again for your time today, guys. Nice to meet you both. I'm sure we'll cross paths again in the future.

Joshua Herr (56:08): Absolutely. Thanks for having us on, Aaron.

Matt McLendon (56:08): Appreciate it.

Transcript lightly edited for readability.