Building Resilient Tech Environments: Lessons from Dennis Maldonado

Aaron Crow (0:1.514): Awesome. I'm excited for this conversation. Dennis, why don't you introduce yourself? I met Dennis last week in person. I love meeting people in person and at conferences. He was introduced to me by another friend and we did some chatting and he actually was speaking on stage about some really cool stuff that we're going to dive in today. So Dennis, why don't you introduce yourself to the audience, who you are, kind of what your background is.

Dennis Maldonado (0:23.712): Hey everyone, yeah, I'm Dennis. A lot of people know me in the Houston area, but I'm kind of a person here that wears many hats. I serve mainly as Director of Technology for a local government entity, Harris-Forp and ESD100.

Dennis Maldonado (0:38.172): They're an ESD here and I can kind of explain what that means in a little bit, mainly I operate, built and operated a 911 dispatch center. So we answer 911 calls for fire and EMS and we dispatch those to their respective fire and EMS departments. And so just managing all that technology, make sure it stays reliable and resilient and stuff like that. I also do technology for the local fire department as well as kind of consult for some other

Dennis Maldonado (1:7.938): fire departments and dispatch centers in the area. I have about 15 years of information technology experience, 12 years in cybersecurity, another 10 years in public safety. mean, they all kind of overlap, so I kind of did them all at the same time. And I've spoken at quite a few conferences. I've spoken three times at DEFCON, DEFCON 23, 24, and 25, if I remember correctly.

Dennis Maldonado (1:34.640): Some other local conferences, InfoSec Southwest was a conference that used to happen in Austin. That was one of my favorites. Houston Security Conference, where I met you. I spoke at this last one last week, as well as some previous Houston Security Conferences. It's a great conference, by the way, if you're ever in the Houston area or you want to come stop by, it's growing. And then just other small little local meetups in the Houston and also Austin.

Dennis Maldonado (2:0.622): Texas area. And lastly, I founded a few local meetups. I love to do a lot in cybersecurity. maybe about 10 years ago, I founded Houston Locksport. It is exactly what it sounds like. We just hang out at a restaurant, pub, or bar and pick locks. We've been doing that for over 10 years. And then my favorite is Houston area hackers anonymous. So I founded that almost nine years ago. It's still extremely popular. We had our meetup, actually a monthly meetup yesterday.

Dennis Maldonado (2:30.468): where we had about almost 50 people show up. So it's back to pre-COVID numbers. But yeah, it's just a meetup where a bunch of hackers, geeks, nerds, cybersecurity people, and everyone else show up, hang out. We share knowledge, network, talk, and eat food and drink. So yeah, that's me.

Aaron Crow (2:44.846): That's awesome. You know, before we dive in too much of the details, it's really important for that networking side of things, right? So I had a mentor and I say this frequently, but I had a mentor a long time ago that told me all businesses are people business. So it doesn't matter if you're the CEO, the janitor, the cyber guy, the engineer, whatever, right? We all have to interact with people. That people element is what makes us work faster. And another thing I say all the time is we do business at the speed of trust.

Aaron Crow (3:14.316): And how you build trust is by building connections with the people that you work with or that you may interact with. So when you're coming in from outside, people don't know you, the faster you can build that trust between me and you, the more likely we're going to be able to talk about whatever it is when you talk about, right? So if I need to deliver bad news or I'm asking you to change the way you do things, or I need to, you know, impact something that you're doing, the more likely, the more that you trust me, the more likely that you're willing to listen to what I have to say and take it.

Aaron Crow (3:42.604): That doesn't mean you just trust me blindly, but you're more likely to at least listen to what I have to say if we have that trust built. So that networking aspect technology people, us nerds, us engineers, those type of you, we don't always focus on that networking side because we don't necessarily, I can just sit in my basement and, and, you know, hack and compute and all that kind of stuff. And what's the benefit of going to a networking event? I'm not a sales guy. Why do I need to do that? But I think the fact that you see that and, and, and so many,

Aaron Crow (4:11.586): I see a lot of folks that get it and a lot that struggle with it. Even if you're an introvert, even if you don't like those types of things, a lot of us are. Like I'm very outgoing, but also I'm perfectly also fine just being locked in a cave and doing my thing, right? But I have to pull myself out of that and put myself into these social situations. And that has grown my career exponentially because I've made those networking connections before I needed them. So when I lose a job,

Aaron Crow (4:41.260): or I'm looking to change it, Aub, or any of that type of stuff. I have those connections and that helps me faster, go faster, right?

Dennis Maldonado (4:42.000): I

Dennis Maldonado (4:50.308): Right, yeah, no, I completely agree with you, Aaron. I feel like your story you just told about yourself, your personal growth, is exactly the same motto and the same story that I had. I, many people, in fact, I don't think anyone will.

Dennis Maldonado (5:4.568): would agree with me on this, I do consider myself an introvert. But when it comes to those social situations, I feel like at least when I was starting in the career and growing my career and growing my personal growth and connections, I forced myself to go out there. That's the whole reason why I started Houston Air Hackers and Arnavists or HAHA. I'm going to call it HAHA from now on. I love that name. And the way I started it was I wasn't even sure if people were interested or were going to come to it, but I just put it out there like, hey, every month on the first Thursday of the month, I'm going to be here.

Dennis Maldonado (5:34.654): regardless if anyone shows up, I'm going to be here doing stuff and talking about stuff. Show up if you'd like. And over the past nine years with COVID kind of putting a pause on that, it has grown exponentially and it has become a great place to facilitate people talking and sharing information, sharing knowledge. And I just love it every time I have that meetup and there's always a record number of people and there's always people after the fact saying, thank you for hosting it. I love it.

Dennis Maldonado (6:4.310): And then seeing relationships get built from that meetup, like personal relationships, but also professional relationships, people getting jobs for my meetups, just makes me feel warm and fuzzy. And then of course, when I talk more about Westcom and cybersecurity and public safety, I took that same mindset is I wanted to facilitate.

Dennis Maldonado (6:27.478): relationships, inner communication, information sharing. And so that's kind of what I what I try to foster in the Houston area of public safety community. I started this mailing list with other, you know, emergency communication, emergency services agencies where we can share and disseminate intelligence and cybersecurity information and stuff like that. So if any one of us have an incident, we already have that relationship together so we can share.

Dennis Maldonado (6:55.822): that threat intelligence or that indicators of compromise to the other agencies and we can all kind of just grow from it or get stronger from it. So yeah, I didn't used to like talking to people and going out there and being extroverted, but I think I've grown into it and I'm having fun with it now.

Aaron Crow (7:12.686): think that's the key is, is people think, I'm not extroverted. I'm not outgoing. It's just like anything. It's a muscle. You can, you can get better at it, right? That doesn't mean you'll ever be the most extroverted person. But if you actually look at, a list of podcasts the other day, I can't remember which one it was, but they were talking about most of the large cyber leaders or not even just cyber leaders. most like Elon Musk and, and, and Steve jobs and some of those, many of them are introverted.

Aaron Crow (7:42.414): but they forced themselves to be more outgoing because they know they need that to be successful. Right? So it's just like anything. Nobody wants to go to the gym. Not nobody. Many, most people don't want to go to the gym, but you know, when you go, you get a benefit. Like you don't, you don't want to do your taxes, but you know, if you don't, then you're going to get thrown in jail for not doing your taxes. there, there are certain things that, know, you've got to do. And this is one of those, I think.

Aaron Crow (8:6.954): most people don't necessarily put the right amount of focus on and how beneficial it will be to you and to others, not just to you individually and what am I going to get from this, but also when you tell your story, you're building someone else up. Like if he can do it, I can do it, right? If he's out there and he's introverted and he can come out and start these organizations and have all this stuff, why can't I just show up? Like he's done all the hard work. I can at least show up and get some benefits from that, right? It helps other people see

Dennis Maldonado (8:25.231): Mm-hmm.

Aaron Crow (8:35.918): put themselves into your shoes and see themselves through your eyes and empower themselves to say, hey, I can do it too, right? It's so empowering to just be there and just show up and do those types of things even when you didn't think that you could, right? And failure is okay, but it's fun to get out of your comfort zone sometimes, especially in things like this.

Dennis Maldonado (9:0.206): Yeah, and the benefit may not always be immediately realized, but the benefit is absolutely there. I'm sure you get a lot of questions of how people asking you, how can you get started in cybersecurity? How can you get a job in cybersecurity? And I have a lot of different opinions and advice, but one of the biggest ones, of course, is go out there and network.

Dennis Maldonado (9:18.798): Get known. You don't have to, I guess, do what I do, right? Start all these meetups and then start a blog or start doing talks and everything. But if you just get out there and get known, even if you may not be the best, most technical person, you have a persona, you're well-liked by the people that you've met.

Dennis Maldonado (9:39.668): And whenever they know of an opportunity or they have an opportunity, they're going to remember you and you're going to get the, you're going to probably be one of the first people to get those opportunities. And that's kind of what happened to me. I've been fortunate where most of the jobs that I can remember, if not all of them have been a result of having a connection or building a network and someone recommending me or advising that I apply for this job. And, and I've been able to use kind of the, the, reputation that I've built to, to,

Dennis Maldonado (10:9.562): get those jobs and progress over the years that I've been working.

Aaron Crow (10:12.398): Absolutely. mean, I've doing technology and all this for a long time, more than 30 years. it's, I can honestly say none of the jobs I've ever gotten in my life have I gotten blindly. Meaning I didn't just blindly apply, submit my resume and get the job. Every single one of them were because I got referred. I knew somebody at the company.

Dennis Maldonado (10:32.474): Mm-hmm.

Aaron Crow (10:39.296): I knew the position, that kind of thing. And I used my network to get my sister works there or my best friend works there. And, you know, I still had to do the work and I still had to pass the interview, but it's amazing when you go in and you like, if you knew me and I'm, and I have a, open job. And since I know you, you say, Aaron, I think I'd like to apply for that. Awesome. me your resume. I'm going to put you on top of the stack. So you're not going through the normal channels. You're already at the top. Now you still have to deliver.

Aaron Crow (11:8.418): You still have to pass the interview and be have the capability and all that kind of stuff, but you're getting considered because of your networking and who you know, right? Use that to your advantage. It's a benefit to you and it's a benefit to the business, right? So it's cheating. It's the cheat code, right? It's back in the day, Super Mario Brothers, you know, or Street Fighter. It's knowing the secret codes to get the special thing. That's the cheat code.

Aaron Crow (11:35.074): how many people are applying with their AI resume and all that kind of stuff to all these jobs on LinkedIn. talk to folks all the time and they're like, I've applied for 500 jobs and I haven't even gotten a recall response. like, so obviously you're doing something wrong because I transitioned in January, was downsized at the company I was at and I was an executive in the company, right? And within a week I had 10 offers, a week.

Dennis Maldonado (12:5.166): I'm

Aaron Crow (12:5.342): And it's not because Aaron is the smart, and I say this all the time, it's not because I'm the smartest guy in the room. It's not because I'm the best person in any of those types of things. It's because I built that network. And once I told that network, Hey guys, by the way, I'm on the market. had multiple people reaching out. I don't know. I don't have an open position, but I want to make something for you. want to do something like, I want to work with you. Right. And it's because of that networking that I'd done way before I ever needed it and continue to do.

Aaron Crow (12:32.256): and building relationships with people like you. And you just never know how this connection can impact you or I or others in the future. And it's so valuable to have this. So I know we've dove very deep into this thing, but I think it's so powerful and so important for people to understand. This is like a deposit and investment into your future. You put your money in now, you put your time and your efforts into this networking today.

Aaron Crow (13:0.182): and you never know when you're going to need to pull out, make a withdrawal from that environment or from this account in the future. And maybe it's for you, maybe it's for you, you know, the opposite side. Maybe it's the other person that needs to make a withdrawal from you, but that account can continue to be built upon and you're going to need it in the future at some point for some reason that you can't even imagine today that is going to be helpful in the future. So I dig it, man.

Dennis Maldonado (13:29.390): Yeah, and I actually have a great story of that. Like you said, you never know when those opportunities are going to come up from that network that you built. know, unrelated, I've been a volunteer firefighter at the community fire department here in the Houston suburban area for over maybe

Dennis Maldonado (13:46.608): 12 years now and on top of the typical, you know, firefighting, doing fire truck, responding fires and stuff. also offered and volunteered to help a lot with the technology, you know, growing the technology, managing all the technology for the fire department, radios, IT infrastructure, all that stuff. And, and because of that, I've grown, I've obviously grown, be known to, I was, I became known as the IT nerd, the geek for the fire department.

Dennis Maldonado (14:15.672): And so, you know, whether it's for the fire department or personal, the people would come to me for help with IT and stuff. so, you know, many years down the line, there was without getting too much into the story, but there was a need for the fire department to get a new dispatch center.

Dennis Maldonado (14:37.544): because the existing dispatch center was, there's some politics and stuff, they were going to cease to exist in a few months. And so if nothing was taken care of, then my fire department, as well as 12 other fire departments would not have anyone to take and dispatch the 9-1 calls. And so because I had this reputation of just being able to handle IT and build and fix and manage things, some of the leaders came to me and asked, hey,

Dennis Maldonado (15:6.296): We need a dispatch center. We need it in three months. Can you build one? And I thought a lot about it. ended up, I wasn't sure if I could do it, right? Because although I had a lot of IT and infrastructure and cybersecurity experience, I hadn't really built like a full enterprise grade, resilient, reliable network from scratch ever.

Aaron Crow (15:24.511): Sure.

Dennis Maldonado (15:27.754): The most I've done was manage something at the University of Houston when I used to work over there or build something on my home lab. But I decided, I think it's a good challenge to do and so I decided to take that challenge on. much to my surprise, I was able to do it. I was able to build like a full 911 dispatch center.

Dennis Maldonado (15:49.934): that was reliable, resilient and worked in three months. And they hired me to do that. And so that's just, it came out of nowhere, but it came from networking and having that reputation and knowing people. Yeah.

Aaron Crow (16:3.084): And you built that trust, right? So you built that trust. They knew you as a person and they said, hey, instead of grabbing somebody, some consultancy or some contractor company that I don't know, you know, this system, I know you, I know if you tell me you're going to do something, I believe that you're going to do it or you're going to figure it out or you're going to ask the right question again, because you had that, that trust, you were able to move really quickly. So that, that is an amazing example of it's not accidental. It's not.

Dennis Maldonado (16:12.826): Mm-hmm.

Dennis Maldonado (16:23.640): Yeah.

Aaron Crow (16:31.200): It's not coincidence. You're not lucky. You're in the position you're in because you made it happen and you built that relationship with those people and that organization that they trusted you as a person and your skill set because you'd shown them what you can do over time. It didn't come out of the blue. It wasn't just like, hey, could you do this? It was like, hey, you've shown that you can do this. Let's up it. Like if you can do this, let's see if you can do this next level. And of course you knocked it out of the park, which is awesome, which is where we're to get into next when in that.

Aaron Crow (17:0.852): story. talk about what this what this 911 center is and and and do that's that's super exciting to build from the ground up. Very few people get the opportunity to build a greenfield from scratch brand new environment or know redundancy and and especially something critical infrastructure like that is doing 911 dispatching and the the importance of that to the community that it serves.

Aaron Crow (17:27.042): So obviously it's got to be resilient because when I call 911, it's got to go through. Like I want my, I want the firetruck to show up at my house. I want the police to show up. I want the ambulance to be here and all of that keys on going through this dispatch center.

Dennis Maldonado (17:43.460): Yeah, absolutely. So.

Dennis Maldonado (17:44.696): Yeah, so we're a 911 dispatch and we're officially called Westcom. That's the name that we decided. And so on the radio, they will refer us to as Westcom or whatever. But we dispatch right now for 11 fire departments and that's currently growing. We're additional fire departments are working to figure out how to come to us or we're onboarding others. But we specifically focus on fire and EMS. So we don't do law enforcement, which I kind of wish we did because I think it would be a little bit more fun. But with law enforcement,

Dennis Maldonado (18:14.730): there's a lot of additional things we have to worry about like, know, CGS and TLED, cybersecurity related things. Not that cybersecurity isn't important, but with the state of Texas, when you'd handle law enforcement information, there's a lot of specific...

Dennis Maldonado (18:30.768): procedures and policies that we need to follow. So we don't do any of that at this time. It's just fire and EMS. But yeah, so we dispatch for 11 different fire and EMS agencies here in the Houston suburban area and we're what's known as a secondary answering point. And so what that means is when anyone calls 911 in our areas of coverage, they first go to a primary answering point, which would either be

Dennis Maldonado (18:58.864): Harris County Sheriff's Office or Fort Bend County Sheriff's Office wherever they are physically at when they call 911 and when they say they need fire or EMS

Dennis Maldonado (19:8.644): that call is immediately transferred to our center and then we take care of it from there. And so at that point, we ask for specific information from the calling party, the person that's calling 911, that is their phone number, their address, their name and stuff like that. And then we finally ask, you know, what's going on. And as soon as the person tells us enough information, even when they're not complete with their sentence, when we get enough information to determine, okay, this is going to be a medical call or a fire call or something like that, we are already dispatching it. The system already

Dennis Maldonado (19:38.578): took it, the fire departments have already been alerted and we already have an ambulance or a fire truck or whatever we need to send en route while the call taker is getting more information and adjusting that. And as we get more information, as the call taker tells us more about what's going on, we can adjust that dispatch. So if we dispatch an ambulance and all of a sudden it becomes more critical.

Dennis Maldonado (20:1.880): and we need a second ambulance or a fire truck or a supervisor unit, then the computer-aided dispatch, the CAD, will accommodate for that and...

Dennis Maldonado (20:10.796): send an additional recommendation to where the fire departments will get additional dispatches to add to those incidents. So yeah, so there's a lot of technology that goes into that to make it work. Of course, we have the 911 call taking equipment that handles the 911 calls. We got the CAD, the computer aided dispatch that is pretty much the brains of any dispatch center is the dispatch. It takes in the 911 information and it

Aaron Crow (20:33.812): Mm-hmm.

Dennis Maldonado (20:39.574): outputs the dispatch recommendation all the while, you know, the notes and stuff get taken in there and it's all stored for historical reasons and record-keeping reasons. And then you've got the radio infrastructure. We actually need to talk to the agencies in the field, the fire departments and the EMS services. So that's the radios that we communicate with the fire departments. And then everything else that supplements that, right? Back in the day, that's all you really needed, a 9-1 phone and a radio. But nowadays we've got additional technology. We've got

Aaron Crow (21:4.270): Bye.

Dennis Maldonado (21:8.988): Every fire truck and ambulance has a computer in it that tells them the details for the calls that they're going to, routes them with their mapping and stuff like that. And so we support that as well. We have automatic station alerting. So, you know, at nighttime...

Dennis Maldonado (21:23.824): that the firefighters don't have to keep the radios on and stay awake and listening the whole time. They've got automated alerting system that will turn on the lights, put alert tones on the speaker, open up the bay doors, and even turn off the natural gas in case they left it on so that you get alerted, wake up, go to the call, and the fire station stays safe. So all of that supplemental additional technology that we put in place and support as well.

Aaron Crow (21:48.002): That's awesome. And all of that goes to the faster that you can get to and respond to an incident, the better it's going to be. Time is of the essence and the more information that the responders have of the medical condition that they're walking into the state of the building. If it's on fire, if it's a wreck, if whatever those things are, the more information they have, the more prepared they are to go on scene. What do I need to bring with me? Like, is it, is it a chemical fire? Is it a,

Aaron Crow (22:18.094): house fire is it a car fire is an accident like a do we need the jaws of life like do we need one truck or two or any of this stuff it's it's that triage so they're starting that triage process before they ever get to scene in theory they'll be able to respond quicker and you'll be able to save more lives and save more property damage and all that type of stuff because they're more prepared which is amazing that's the whole benefit and and and value add with technology is

Aaron Crow (22:45.876): even one second, even one bit of information can save a life. Right. And when we're dealing with critical infrastructure like this, lives matter. Right. Everything you're talking about is specifically around human safety. Yes, property. Yes. All that kind of stuff. All those things are important. But the the priority statement there is human life and human safety. Right. And saving people's lives, getting them to the hospital, get them the care that they need as quickly as possible. And if technology can help them do that faster. Wow. That's amazing. Right. So it's it's awesome.

Dennis Maldonado (22:51.888): Yeah, absolutely.

Aaron Crow (23:15.832): For the firefighters, it's awesome. For the paramedics, it's awesome. For the dispatchers, like everybody in that can be just a little bit more efficient and how much the domino effect of the impact of that is huge.

Dennis Maldonado (23:28.814): Yeah, and then responder safety is also extremely important too. And I like to say that, you know, when an incident just starts, no one knows more about the incident. Of course, not talking about the person who's calling, but you know, the person who knows the most about that incident is the dispatcher who answered that call. And so it's important that that dispatcher is able to convey that information to the units in the field.

Dennis Maldonado (23:53.432): so that they can be the most situational aware of what's going on, what they're going to, and then they can make their informed decision on how to handle that call, whether there's a potential for danger, so we need to stage and call law enforcement, or it's just a run-of-the-mill medical call or fire call and just respond as per usual. Yeah.

Aaron Crow (24:9.859): Yeah, so so dive into what does the tech stack look like as much as you can go into it? Like how much is there like what obviously you've got phone systems. You've got radios. You've got this CAD system you talked about. There's got to be some redundancy and power and and networking and alternate threads of redundancy and network connections and all that kind of stuff. But just kind of dive into what is what does that architecture look like to provide something as robust in a critical system like this?

Dennis Maldonado (24:32.752): Mm-hmm.

Dennis Maldonado (24:39.182): Yeah, yeah. So I feel like I like to fool people because, I talk about 9-1-1 and critical infrastructure and, you know, people want to know about what like what's so special about technology, but really it isn't any special. It's like any other conventional corporate IT technology. just, you know, when it comes to cybersecurity, cybersecurity is what protecting confidentiality, integrity and availability. I like to say that, you know, when it comes to 9-1-1 dispatch, availability is the most important thing by far. Sure, confidentiality and integrity are still important.

Dennis Maldonado (25:8.900): but also we're a government entity, everything is available to the public anyway, but availability is the number one thing, we need to stay available. all the technology that we built here at Westcom.

Dennis Maldonado (25:20.236): was designed with that resiliency strategy in mind, to be as available as possible. And if an outage or an issue does happen, how can we recover as much as possible? So, and that's the benefit I had going into this building this when I started a few years ago was since I have the cybersecurity experience and background, I was able to build this with cybersecurity in mind, with cyber resiliency and availability in mind. And thankfully I had a board, an ESD

Dennis Maldonado (25:50.130): Governmental Board that approves all the budget and everything they supported that and so when I pitched them here's we want to build here's I want to build it here's what it's gonna cost They completely agreed with me thinking that's what you talked about earlier is that trust they trusted me but the trust goes both ways I trusted them to give me the support the tools and the funding we need

Dennis Maldonado (26:11.022): to do this right, to build it. And so before we took on this project, we talked with a lot of other agencies in the area. We toured a lot of other dispatch centers and fire departments and server farms and stuff, and just learned, took some lessons learned from them, learned about the incidents that they've experienced, the mistakes or the...

Dennis Maldonado (26:35.428): the improvements that they wish they could have made, and we are able to implement that here. So when it comes to resiliency, we try to be, I try to threat model every possible threat, manmade, natural, or otherwise, that I consider, and then kind of balance how much money can we spend on that and how can we remediate that threat. So I feel like we've done a pretty good job. I invite you to...

Dennis Maldonado (27:1.124): come up with any random threats and we can talk about whether I've protected against them. But for example, like natural threats, lightning strikes. We had a previous dispatch center that did dispatch for my fire department before Westcom. And there was several incidents over the past few years where anytime they got struck by lightning or lightning struck nearby, their services were degraded, whether some radios went down or their cabin went down or whatever. So we took a lot of steps into ensuring that

Dennis Maldonado (27:28.404): lightning strikes or similar weather-related incidents don't happen. So we're very heavily grounded. We have lot of grounding all over the place. But not only that, we have a lot of redundancy in case certain systems like the radio tower does get struck by lightning and does go down. We've got redundancies in place so we can still maintain radio communications. Multiple internet service providers come into the facility. So if one goes down, we're still good. And then

Dennis Maldonado (27:57.164): most importantly electrical. So for those who are familiar with the Houston area, we don't have the most robust or even the whole state of Texas. We don't have that most robust electrical grid. So Hurricane Barrel came to the Houston area a few weeks or a few months ago. And like most of the region, Westcom had no utility power for about four days. But because of our resiliency strategy, we have not one but two generators

Dennis Maldonado (28:26.640): on the property. And even if both of those generators were to fail, we have a large UPS system that could hold us for several hours until we can either call an external generator, drive them up and hook up to that, or fix whatever problem our two generators may be having. So luckily none of that was needed, but our generators did protect us for the entire four days that we had no power.

Dennis Maldonado (28:50.916): And so, and the great thing is no one here at Westcom really noticed until I kind of gave the report that, we were, you know, our generators are active for four, for four days straight. And that's, that's what I tried to do. I tried to build this resiliency strategy that would be, protect us from.

Dennis Maldonado (29:10.380): availability issues, but even when we have to recover from something, it is almost transparent to our dispatchers. So they can just completely focus on taking those 911 calls and our constituents will get the 911 services they need. yeah, as far as the other technology, it's the conventional stuff, right? We have a network stack where we use...

Dennis Maldonado (29:35.534): various different vendors of switches, but everything is redundant. We have two of everything. From our firewalls, we have two of those that are hot, warm spares, ready to go. We got switches that are either virtual, attached to each other, or stacked in one way or another. And everything that's critical, like the servers, the CAD workstations, they are connected to multiple different lines, multiple stacks of switches. So even if we have a full switch failure or

Dennis Maldonado (30:3.740): cables fail or a network interface card fail, we're still good, we're still protected. A lot of things have to fail at the same time in order for it to actually impact our services. And then same thing with power. We have not one but two large 100 kVA UPSs on the facility. So again, everything critical is plugged into each UPS one.

Dennis Maldonado (30:29.424): power supply on one UPS, one on the other. So if one were to fail, which we actually did have happen, all the equipment is still powered and we'll have some sort of resiliency even after that. So yeah, I'll pause.

Aaron Crow (30:39.340): Yeah, it's amazing. So I've grown up in critical infrastructure. A lot of my time was in critical manufacturing, critical power generation, transmission distribution, that kind of thing. So for me, it's very easy to see because every system I've ever been in and designed and been part of has been that critical, right? It's redundant power, it's redundant UPS, it's redundant backups, it's redundant switches.

Dennis Maldonado (30:54.489): Okay.

Aaron Crow (31:4.962): like absolutely everything. And this redundant switch has two power supplies. One's going to two different power feeds. And I've got two network connections to everything. And even my Windows machines or my HMIs or the engineering workstations, they have dual power supplies and they've got dual network feeds and like all that kind of stuff. But that gets expensive, right? So you can't have every system in every critical or every non-critical system do that, right? You only focus that, you're not trying to platinum code everything.

Aaron Crow (31:32.770): You're trying to put it so that the critical systems can maintain an outage, right? And the second piece to this goes back to, you've designed this robust system, this network, all this kind of stuff. The other part that I see, and we haven't gone there, so that'll be my next direction for you is...

Aaron Crow (31:53.198): in these critical environments and architecting these highly robust capable systems, monitoring is important because if I've got two switches and everything's redundant and I don't know that one of them went down, I think I'm redundant. But I'm really not because one of them's already failed and I don't know it's failed. Right. My UPS is down. Like you knew that it was down. So obviously you have this implemented, but I've seen super critical capable systems where they don't realize that a switch is failed.

Aaron Crow (32:21.226): It's in series, they've got two of them and it's done its job. So the system, they never noticed anything went down, but they also didn't know that it was down. So they didn't know to get it back up and going. So it didn't actually cause a problem, but it could have if the secondary system also failed, you you've got primary and backup. If the primary fails and it goes to the backup, they never noticed anything goes down. Now they're only running with one system.

Aaron Crow (32:47.274): If that backup goes down, then we have an issue. that monitoring is also important. So talk a little bit about how you guys monitor all that stuff and how you notice that your systems are good and healthy.

Dennis Maldonado (33:0.686): Yeah, yeah, so we do a lot of monitoring here, but of course, there's always room for improvement. for the viewers who can actually see my background, I have a monitoring dashboard.

Dennis Maldonado (33:12.090): where I can just see the current health status of all of Westcom systems. Green is of course good, red or orange, there's an issue that needs to be fixed. But we do get alerts for almost anything that happens. And yeah, there is such a thing as alert fatigue, but I think we've done a good job in triaging and prioritizing those alerts. So whenever we get an alert, whether something is down or maybe a power supply fails, or there's a cybersecurity incident or an intrusion, something like that, the IT team here gets...

Dennis Maldonado (33:39.572): an automatic email as well as the support system that then automatically prioritizes it and we can respond to that. So I'll give you a good example. And I know you've already talked about CrowdStrike like six different episodes. I've counted them all here. It is a shame because I love talking about how CrowdStrike impacted us because it did because I feel like we learned a lot of lessons from it. But

Dennis Maldonado (34:2.384): when CrowdStrike happened, so July 19th, right at 0000, so right at midnight, I just randomly started getting alerts on my phone while I'm getting ready for bed at home that some servers are going down. And at first, you know, when one or two servers went down, like...

Dennis Maldonado (34:17.040): That's okay. That sometimes happens. The server reboots, comes back up. But they weren't coming back up. And then when I started getting additional alerts for additional servers were going offline, I noticed that there wasn't a specific pattern. They were random servers. And usually, if a hypervisor goes down, then all the servers on that hypervisor go down. But there wasn't that pattern. So I'm starting to get concerned. So I VPN. I'm at home, right? So I remote in, I VPN. I start remote desktopping into servers that are still online.

Dennis Maldonado (34:45.750): And as I'm investigating, looking at things, I lose connections. So now I start freaking out. Not really freaking out, but like I'm starting thinking, okay, what's going on? So I do have Dell iDRAC, which is pretty much like local access to one of my hypervisor servers so I can connect to it and an interface just like if I had a monitor and keyboard. And so I remote into that and I'm looking at the hypervisor, just checking the health. And then I see right in front of my eyes, it blue screen.

Dennis Maldonado (35:14.606): And that's where I internally panic. I'm like, dang. I'm a cybersecurity guy. I always talk about cybersecurity guys. I'm thinking to myself, I finally got ransomware. I'm finally the victim. Let's start working on this. But thankfully, while I'm investigating that, someone sends me a news article that...

Dennis Maldonado (35:35.744): some airlines are having IT issues and are grounding all their flights. And I almost immediately correlate, okay, there's some global IT outages. This has probably been not me. So, but the alerting system, going back to that, clued me into that and I was able to address that and start investigating relatively quickly. I used to brag that in three years of Westcom operating, we've had zero outages until CrowdStrike happened.

Aaron Crow (35:59.982): Mm-hmm.

Dennis Maldonado (36:3.824): So that CrowdStrike actually did take us out, but because I was able to be alerted so quickly and respond so quickly, we were able to find the root cause analysis attributed to the CrowdStrike Falcon Center and come up with our own temporary fix to get servers back online, even before CrowdStrike acknowledged an incident and published their own fix. So even with the outage, we recovered from that within 62 minutes.

Dennis Maldonado (36:33.232): 62 minutes and 19 seconds. And I'm particularly proud of that recovery period because talking to other peers and agencies, they weren't as quick. But there was a few things that contributed to that. So our alerting contributed to that. Our process of quickly activating our contingency plans worked really well. even though we were down, our CAD was down for 62 minutes, we still had a contingency plan. We were still able to take 911 calls.

Dennis Maldonado (37:1.858): Our radio still works and we can still dispatch to the fire departments. And so our contingency plan was pen and paper. Our dispatchers took the calls that were already on the CAD at that time on pen and paper. And any additional calls that came in, we had about five 911 calls during our outage period. They were able to just do that in pen and paper and dispatch that to the fire department. So the great thing is the fire departments had no idea. It was still business as usual. And we never missed a 911 call. All of our...

Aaron Crow (37:5.697): Sure.

Dennis Maldonado (37:30.929): all the people that we serve still got as good of a service as we possibly can. And we are able to recover that CAD and go back to normal operation within 62 minutes. So the one lesson learned though, going back to the alerting is my alerting server is on the same hypervisor that CrowdStrike killed. So after that was down, the alerting was also down, but...

Dennis Maldonado (37:58.456): Of the many lessons learned, I did an actor action report and I shared that publicly. That's one of the lessons learned is, you know, have different alerting platforms that are completely separate offsite. So they're resilient for if we have a major system wide outage, those alerting systems probably will not be affected as well.

Aaron Crow (38:13.654): Yeah, that is huge. So the fact that you guys recovered that quickly shows that a it was resilient in the design, but also your recovery plan, right? So your recovery. Obviously you had backups, you tested those backups. Everybody knew what to do. And the fact that the systems went down, but you had the worst case scenario, everything's down. What do we do now? We can still take 911 calls and we can still dispatch. So the fact that that happened.

Aaron Crow (38:42.860): And again, the fire departments didn't know that that's the best scenario. Like you hope it never happens, but you have to plan for those things because you just don't know. You can't assume that that's never going to happen. No matter how resilient you make your network and how many switches and how many UPS is you put in and how many generators you put in, there's going to be something that happens that you can't see. So you have to plan for, okay, I designed the best network in the world. It's still going to go down. What do I do then? Right. So

Dennis Maldonado (39:6.480): Right.

Aaron Crow (39:12.332): The fact that you guys thought through that process, at least at some level, shows that everyone needs to do that. Now, again, if it's a not critical system, you're managing a self-storage place, you don't need to have that level of capability, but the same thing. Okay, my gates are automatically controlled in that scenario, somebody needs to go and lock the gate, or I need to have somebody standing there and opening the gate for customers or whatever. That's the kind of process that you need to go through

Aaron Crow (39:42.422): And I think that's missed a lot of the time, whether, you know, most organizations are doing a tabletop exercise and they're probably doing that once a year. Unfortunately, in my experience, they're doing that at super high levels and they're not necessarily including all the people that are in the know. So they don't get the real true risks like what you just experienced. Like that should be turned into like you did a lesson learned. What can we learn from this?

Aaron Crow (40:9.782): I should learn from my mistakes. I should learn from my wins. We can always be better. Like it's not a negative on you or anybody in the organization that you had an impact. And the fact that if you don't learn anything, it's that whether it's in personal development or anything, and there's that line, you either win or you learn, right? There is no lose. You only lose when you quit. You only lose when you give up. If you have something that you can't, you fell short on, what did you learn?

Aaron Crow (40:38.958): so that next time you don't have that same thing happen again, right? Now, if you have that same incident and you didn't learn anything, you didn't change anything, and you just continue to repeat that process over and over again, then yeah, that's a problem. But the fact that you guys learned something and you made it a little bit better for next time, that's an awesome win. So the next time this happens, you'll have your alerting done faster and all that kind of stuff like you guys, we've done this before, we know what to do.

Dennis Maldonado (41:4.216): Yeah, yeah. And then going back to our previous discussion on like, you know, communicating with people and networking and then sharing information is, yeah, we did learn a lot. There was things that we can improve. So I wrote a full after action report over here. Well, here's what happened. Here's the timeline. Here's the conclusion. Here's the lessons learned, you know, like building an incident, solidifying an incident response plan or fixing some policies or a backup strategy.

Dennis Maldonado (41:31.714): Not only did I share that within Westcom, but I also shared that to the fire departments that we serve, but I took that further. I shared that to all the other public safety agencies here in the Greater Houston area and in the state of Texas, just so in case they want to read the report and see how it impacted a similar organization and the lessons learned. So hopefully maybe some of the lessons learned that they can apply to their agencies. And yeah, and that's what I love to do. Even if I may...

Aaron Crow (41:54.104): Exactly.

Dennis Maldonado (42:1.328): not look in the greatest light because one of the lessons learned is probably something I should have done anyway. It's still something I'm willing to share so hopefully other agencies can learn from my mistakes. And then, you know, one thing that I kind of thought about the other day was when I built Westcom, I did a lot of threat modeling and that is, you know, considering what cybersecurity threats do I want to consider and how do I want to build for it? And

Dennis Maldonado (42:29.813): When I threat modeled whether I should bit locker the CAD workstations that are here inside the secure facility, the big workstations, I decided, no, I don't need that. I probably don't need to spend the time and effort to physically encrypt the disks for the hard disks that are...

Dennis Maldonado (42:45.892): for the CAD computers sitting in the locked desks in the locked facilities, because that'll save me some time and effort and maybe, you know, something like that. But the unattended benefit I got is because I did that threat model, which I feel is an accurate threat model, not needing to encrypt the desk computers, that greatly improves the amount of time I needed to recover from CrowdStrike, because now I no longer had to...

Dennis Maldonado (43:9.700): consider dealing with BitLocker. I just had a straightforward recovery plan to fix those computers. And that's how we're able to cover so quickly. A lot of other agencies, they BitLocker everything and that's great. Better security is often better, but at the cost of recovery time or at the cost of having to get more people to help, more resources, more tooling and stuff like that. that's a of a threat model benefit that I think I should talk about more.

Aaron Crow (43:35.054): No, I agree. So I've seen time and time again that, examples like that. So again, my career mostly has been in critical infrastructure and places like that. And, and when I'm having conversations with my IT, you know, counterparts coming from corporate that can't understand why. So a lot of the environments I'm in, like power plants, they're, they're, they're antiquated, they're older technology. So we may have a windows XP machine serving as a critical function.

Aaron Crow (44:4.330): or they're not, you know, they don't lock their workstations. There's no password. They don't have individual logins. Like things like that in a power utility are a little different. but to your point, you know, the CIA triad, and I say this all the time in power utility and critical infrastructure, the CIA triad is, is kind of flipped upside down and availability is at the top. Not that I don't care about confidentiality, not that I don't care about integrity, right? It's that availability is way more important than anything else. And, and especially in this space, I say availability slash safety.

Dennis Maldonado (44:5.070): Mm-hmm. Mm-hmm.

Aaron Crow (44:33.164): Right? So it's more important that I can control the process than it is that I know who's logged into that machine. So I have other mitigating factors in a control room. There's physical security. They had to get into the room. They had to get onto the facility. It's a small group of people. So if some random person walked into the control room and somehow got that far and they sat down at the desk, that's not going to happen because they're going to know who the heck are you? What are you doing in my chair and get away from that keyboard or I'm going to hurt you.

Aaron Crow (45:3.864): So I don't need to lock the workstation. I don't need to have the screens, you know, somebody individually log in. I've got cameras. I know who's on shift during that time. And it's a small group of people. It's five people, it's 10 people, it's 20 people. I know who has access. So I can narrow that down very quickly. There's never a situation where a control room is unattended ever in these environments, right? It's just not. So I don't need to, to your point.

Dennis Maldonado (45:4.501): Thank you.

Dennis Maldonado (45:28.335): Yeah.

Aaron Crow (45:31.624): It would would BitLocker and encrypting all this stuff and having secure logins. Hypothetically, would that improve security? Sure. But does it reduce my availability? Absolutely. And so what's what's the risk reward? And this is where you have to look at this. More cyber security is not always better. Right? Sometimes less security and more availability is the right answer to the question.

Dennis Maldonado (45:46.093): Mm-hmm. Yep.

Aaron Crow (46:0.394): And a cyber person, a strictly cyber person, just blows their mind. Like, why would you not want to be more secure when it, when it hinders my availability? If, if I have to choose between availability and cybersecurity without making it, you know, I'm not going to put on the internet. Like, I don't mean, obviously there is a line where I have to say, no, you're not doing that. I'm not going to allow you to secure mode or remote access in from China without any authentication. Like, no, of course not. Right. There is a delineation.

Aaron Crow (46:30.082): but that delineation is not the same on OT, and especially in these critical infrastructure environments, as it is in IT. In IT, they would never allow Windows XP machine on your network, ever, under no circumstance, right? But in IT, or OT, it's there all the time. I see it at every power plant I walk into, and sometimes older stuff than XP, believe it or not, but we put other mitigating factors around it. So it's off the network, or it's off the internet, I've got firewalls, I'm monitoring it, like there's a lot of other things that I can do.

Dennis Maldonado (46:43.941): Right.

Aaron Crow (46:59.404): because I can't patch it. can't do a lot of the things that I would do in an IT world. And that's okay. Right. Understanding that difference is the key and being able to communicate that in a way that says, yes, I'm not doing this. I looked at that. I did the threat model and this is why I think it's okay. Here's how I'm going to accept that risk and why I'm going to accept that risk. And that's the most important piece.

Dennis Maldonado (47:24.720): Right, and that threat model is just for that specific threat of someone physically coming in and stealing a workstation. That's not to say that just because you're inside the facilities of Westcom or inside the Westcom network that it's free reign. I'm a huge proponent and believer in the zero trust model, the continuous verification, all that stuff, least privilege. So even if you're on the network or you get access to something, there still is just...

Dennis Maldonado (47:54.636): every step of the way, every chain, every rung and ladder, you're continuously verified to make sure you have the access you need to or you're granted when only when you need it. So we've actually had a few several several minor cybersecurity incidents here at Westcom that luckily ended up not being

Dennis Maldonado (48:15.586): a big deal because we had that zero trust model. So one of them being like everyone falls victim to business email compromise, email phishing. We recently had a successful email phishing attack where three of our relatively new employees were successfully convinced to click on a link and type in their Microsoft 365 credentials. because of our alerting, we were able to see recognize that real quick and stop it. But

Dennis Maldonado (48:43.158): even with, even had they maintained access for a little longer than they had, they would have hit a brick wall. Like all they would have had was the emails for the relatively new employees, which isn't much emails, and those employees don't have access to anything that they don't need to have. And for the things that they do need access to, like the...

Dennis Maldonado (49:5.614): the computer to dispatch software, the recorder, the station alerting stuff. All those have additional steps of verification. So someone who just has control of their password or a session token may not be able to access those resources. And then another thing, another incident we had, one of our, this is fun, our Palo Alto firewalls a few months ago, there was a pretty widely known,

Dennis Maldonado (49:31.864): public, I don't know the word, but there was a zero-day vulnerability for parallel firewalls. One Friday, parallel firewalls disclosed that, there's a 10 out of 10 critical remote code vulnerability for the Global Protect VPN portal. There is no patch at the time, but here is a setting change you can do to temporarily mitigate this vulnerability until there's a patch. So it's Friday afternoon. I implemented that temporary mitigation.

Dennis Maldonado (50:1.890): And then on Monday, I came back and the Monday afternoon, I reread that advisory, I see they've updated it. And I forgot the wording, but the advisory effectively said, just kidding, that mitigation didn't work. the whole weekend, we were seemingly vulnerable to this 10 out of 10 critical remote code execution exploit. And I never got an alerts or anything that we were compromised, but just because I knew we were vulnerable, started.

Dennis Maldonado (50:28.194): investigating the logs and I actually did find indicators of compromise. I was able to find that someone did exploit these chains of vulnerabilities and were able to pull our entire firewall configuration. So whoever the threat actor was had access to our private certificates, VPN keys and credentials and all that stuff. But

Dennis Maldonado (50:49.846): I wasn't too concerned because again, we have this zero trust model continuous verification. Yes, this is something that I need to address right now and remediate, but even had they tried to take advantage of the information that they got, they wouldn't have gotten far. And from my forensics after the fact, I was right. There was nothing done from that. And we were able to completely blast those firewalls and resolve the issue.

Aaron Crow (50:56.568): Mm-hmm.

Dennis Maldonado (51:15.840): And just to brag again, because of our resiliency strategy, our redundancy we had, we were able to completely reset and rebuild those firewalls without taking Westcom or the network or the facility down at all. Again, one of those things where no one noticed until I published the after action report, but yeah.

Aaron Crow (51:32.142): That's awesome. Again, it's the culture of be okay to make a mistake or be okay. Things are going to happen. It's not a lack of, it's not like your environment wasn't built correctly. It's going to happen. Threat actors are really capable and smart people. And there's going to be zero days that there's no way you can know about why they're called zero days. Like you can't patch, you can't update. There's gonna be those vulnerabilities. The fact that it's redundant, the fact that you've got the resiliency, the fact that you're able to recover,

Dennis Maldonado (51:58.285): Mm-hmm.

Aaron Crow (52:0.800): and respond to those things is vastly more important and valuable than it is to have a perfect environment. Nobody's going to have a perfect environment in any situation. There has never been a perfectly secure and there, you know, I hate it when vendors and or people say I've got a secure environment, my environment secure. No, it's not. You have no, there is, it's impossible to have a secure environment.

Dennis Maldonado (52:26.576): Yeah, I always tell them, I set those expectations. Like my board member, I report to them every month. I give them a report of, you know, what's going on with technology, any incidents that we had. And I always tell them, even when we were building Westcom, this will not be unhackable. Just because I'm a cyber suit guy doesn't mean I'm going to make this, you know, perfectly secure. It won't be. Incidents will happen, just like the crowd strike. And a lot of us get that question. How could you prevent, how could you have prevented crowd strike? There really isn't a good answer when you...

Dennis Maldonado (52:54.448): When you depend on one vendor to provide cybersecurity endpoint detection and response to all of your servers and workstations, you can't really protect against an outage like that. what you can do is improve your resiliency. again, resiliency, I like to define reliability and resiliency, right? Reliable is protecting against failure, against downtime, preventing that entirely. Resiliency is...

Dennis Maldonado (53:21.078): your ability to recover from failure and get back into operation. And that's what I focused mostly on is there will be additional outages. There will be incidents. People will get into our network that shouldn't be, but our ability to respond and recover from that quickly, efficiently, effectively is what's super important. And that's what I like to set the expectation to our board and our fire departments and other agencies, you know, work on your resiliency plan, build it.

Aaron Crow (53:23.181): Exactly.

Dennis Maldonado (53:50.480): prioritize what's most important so you can recover those first, but then not just document a plan, but practice it, test it, exercise it. Because much like backups, if you're doing backups but you're not testing the backups, you have no backups. Same thing with an IRP, an incident response plan or a resiliency plan. If you don't test and exercise your plan and then continuously improve it, you effectively have no plan. So that's what I like to push for my agency and other agencies.

Aaron Crow (54:17.314): Yeah. Yeah. And that goes to, you know, not everybody, as we kind of kicked this off, not everybody gets the opportunity to build a ground up, you know, environment that has fully redundant and that, you know, does all of those things and is looking at it that way. That doesn't mean that you can't be resilient, right? So if I can't have redundant switches everywhere, what can I do? And I dealt with this a lot in power plants and wastewater and critical manufacturing and things like that. Okay. So I can't have redundancy everywhere. I don't have the budget, let's say.

Aaron Crow (54:46.594): Well, what can I do? Well, I know I'm gonna eventually lose a switch. So if I can't have redundant switches, what's the next best thing as far as resiliency to your point? Well, I can have a spare switch. I can't have 10 spares, but if I have one, that'll help me get back up. So if I lose a switch, I'm gonna have downtime. But if I have an onsite switch and I have the configurations of all the switches in a certain spot, then I can drop the config on that switch, rip the old switch out, put the new switch in, boom, I'm back up and running. Like that's...

Aaron Crow (55:14.840): hours of time sucked out. And not to mention that if I don't have that capable, then I'm going to have to order one that could be days of outage versus a few maybe minutes, hours of time for me to just throw a switch in, drop the config on boom, I'm back up and running. And now maybe I've got two of them. Maybe I've got three of them, or I just got one. And then I order a new switch to replace that one that's broken or RMA, the one that's broken and I get a new one in. There's lots of ways to be resilient. And it's not all

Aaron Crow (55:42.190): complete ground up design of my environment and an environment like you've created, which is the goal of every engineer and networking guy and nerd like you and I, right? We all want to build that environment. Not everybody has the ability to do that. That doesn't mean that you just give up and throw your hands and say, well, they'll never let me do that. Okay. What can you do with the resources you have? That's the question you listener should be having is this is the resources I have. How can I, with knowing that, how can I be more resilient?

Aaron Crow (56:11.766): Right? Not redundant, resilient. How can I do that? And sometimes that's not a technology problem. Sometimes it's you talked about the planning. Sometimes it's going to be making sure I have a plan. I'm testing my backups. I've I've making sure that all the parties know what they're supposed to do in an outage. Like, how do I work around these things? If the switch goes down, what do I do then? Like those conversations cost no money. That's just time and experience that is going to help you be more resilient. The faster you can get back up and running.

Aaron Crow (56:40.928): And that's not just going out and buying technology. Everybody focuses on the technology and the switches and the crowd strike and all the different capabilities, because they're fun and they're sexy. That's not the only way to be resilient in all environments. that's kind of the lesson learned that I say from my experience and what I hear and what you've done, you're not always going to get to that great place of perfect resilience or redundancy, but there's still ways to be resilient.

Dennis Maldonado (57:9.604): Yep, and it's all just the contingency plan. Like I said, with the CrowdStrike outage, all of our technological contingencies failed because of CrowdStrike, so we went right back to pen and paper. And every industry is going to have their own way of doing things or what's important to them, but in public safety, it's still just getting to the now one calls and dealing with their own call. So if we have to do that with just pen and paper and archaic,

Aaron Crow (57:19.726): All right. All right.

Dennis Maldonado (57:37.636): rocks and whatever, we can. And that's part of our contingency plan is how to operate completely with no technology at all. All while we're doing that, we're still working on recovering the technology. So in this day and age, people, you know, a lot of people older than me love to say that, you know, the kids are way too dependent on technology. And it's true in one shape or form. Even now when the computers and the fire trucks are down,

Dennis Maldonado (58:6.512): The firefighters can still do their job, but they love to complain about that. So we still fix it with urgency, ignoring the complaints, they can still do their job. They can still get to the call. We still have key maps and they can still put out the fire without a computer or without a cell phone or whatever. So just practice that. Practice your working without the technology.

Aaron Crow (58:20.834): Yeah, yeah, that's funny.

Aaron Crow (58:25.474): Yeah, like, like I showed you before we started recording, I've got a fire hat up there. That was my grandfather's, he passed away long time ago, but he was a, he was a volunteer fireman for 40 something years. He, his casket was delivered by the fire department. you know, on the, on the whole parade thing, but you know, they didn't have technology. They had radios. That's it. They had these little pager things that would go off at all times of the night and you'd hear all the broadcasts of all the things and he.

Dennis Maldonado (58:45.520): Thanks

Aaron Crow (58:50.518): I remember spending the night at his house and you know, those things would be squawking all night long. And when he was on call, he had to take those calls and show up and, all that type of stuff. you know, he had nothing right. Except a radio and, and his PPE, right. You know, his, his fire suit. and he'd come in from work and throw that stuff on. And sometimes I'd go with him and I'd sit in the truck and watch him put out a fire or watch him show up to an accident or all that kind of stuff. but you know, we get relying on technology because it makes us more efficient.

Dennis Maldonado (58:56.602): Mm-hmm.

Dennis Maldonado (59:7.536): Okay.

Aaron Crow (59:18.562): but we have to be able to plan for the worst case scenario. What if I have none of that stuff? And we see that right now in North Carolina with all that's going on in that environment. 911's down, phone lines are down. they can't, know, the local forces are not able to get to those places. So it's taken people reaching out and helping and helicopters coming in and bringing in supplies because roads are shut down. You can't get in or out. So we have to...

Aaron Crow (59:44.770): do that contingency planning for worst case scenario, right? And there's gonna be a whole bunch of lessons learned coming out of that incident. You guys in Houston deal with hurricanes all the time. They're not used to hurricane, not that level of hurricane. Like there's people that have lived in their houses for 50 plus years and have never flooded, ever, right? And now their house got six foot of water in it, right? And they had to rescue themselves by getting on the roof and wait until the water subsided, right? It's insane the amount of damage.

Aaron Crow (60:13.666): But that's the type of incident that allows us to say, we've got to learn from this and do different next time, right? It's not just a failure. It's not just point fingers at people and say, you messed up. Instead, let's focus on what we can do to get better and more resilient next time.

Dennis Maldonado (60:30.508): Absolutely, I completely agree with that man. And yeah, we do have hurricanes all the time and that's why we not just us, but you know the the whole region we're constantly discussing and practicing how can we operate? How can we respond to major disasters without our usual comforts of you know the radio system working, the internet working, the electricity and Hurricane Harvey was great

Dennis Maldonado (60:57.046): Example of that is is Westcom didn't exist when Hurricane Harvey existed, but a lot of the fire departments

Dennis Maldonado (61:3.310): did have to deal with just what they had, boats and maybe some point-to-point radios, but certainly no wide radio network because a lot of things were down. And then Hurricane Barrel was a recent one where the power was down for lot of facilities. Some fire stations didn't have power for a while and some dispatch centers had issues, but we were able to work through that, but also we were able to work together. And that's one thing I love to do is foster that communication and that working together because if one dispatch center has issues,

Dennis Maldonado (61:33.894): maybe Westcom can assist that dispatch center, whether it's technology or operational. Likewise, maybe they can help us. And we do have that contingency. If we were to fail, we'd disappear off the face of the earth. Our 911 calls automatically get routed to another dispatch center. Likewise, we are backups for others. So help each other.

Aaron Crow (61:49.760): Absolutely. Well, so we, we, we went on a roller coaster ride today of all the different topics from, from a career to, you know, resiliency, which I love. I love having these conversations because we're more than just the things that the technology stuff, it's more than just the, the, the, the, ones and zeros and, the, the, the redundancy and all that kind of stuff. It's, it's a lot of like, like we said, right, it's the connections, it's the networking, it's all those types of things. So with that in mind, and I did prep you for this, that it's coming.

Aaron Crow (62:19.200): What is one in the next five to 10 years, what is one thing that you're excited about coming up over the horizon and maybe one thing that's a little concerning around the cybersecurity footprint in your space and that you've seen or thought about.

Dennis Maldonado (62:33.390): Yeah, so...

Dennis Maldonado (62:34.414): I guess one thing that I'm excited for, it's happening now. In the past five years, think cybersecurity has now become a household name. I know when I started in this industry, it was brand new. No one really knew, understood what I did when I worked in cybersecurity. No one knew what a red team engagement is or what ransomware is. But recently with all the latest things in the news and even CrowdStrike becoming a household name, cybersecurity has become a household name. So everyone is talking about it from CEOs

Dennis Maldonado (63:4.388): CIOs down to, you know, my mom in the kitchen, like asking about, what's lock bit up to these days? but that, I think that's great. And that's beneficial because it gets more attention to the issues that cybersecurity brings it. It, it gets more endorsement from the decision makers to invest into cybersecurity and to train their employees for cybersecurity and, just generally more people.

Dennis Maldonado (63:32.386): more students are interested in cybersecurity want to get into the realm. So I think in the next five or 10 years, there will be more organizations who have budget and consideration for cybersecurity, but then there'll also be more really good people in the industry working in cybersecurity because more institutions are pushing for cybersecurity related curriculums. More people are interested in it. More people are just learning and getting into it. And so we're going to have a good

Dennis Maldonado (64:1.912): I think we're going have a good cybersecurity workforce that would help kind of defend all the entities and organizations we have and just defend everything, defend our country, defend the world. that's what I really look forward to is how the industry continues to rapidly grow and evolve.

Dennis Maldonado (64:21.114): in the next five, 10 years. As far as what I'm worried about, I don't know. I didn't have a lot of time to think about what I'm worried about. I like to stay optimist positive. So.

Dennis Maldonado (64:32.344): Really, I'm just excited to see where all of this goes. Now, I do follow a lot of threat intelligence, so I do follow a lot of what's going on with ransomware groups and nation state attacks, stuff like that. So, you know, that stuff's only getting worse, right? That does worry me a little bit because as these threat actors continue to target additional things like critical infrastructure, know, the recent being...

Dennis Maldonado (64:58.606): The water facilities, a lot of PLCs and water facilities getting hacked, it's not going to stop there. Additional critical infrastructure, my critical infrastructure will get targeted. Who knows what else can happen? And so that does concern me, but my previous, what I'm excited for will hopefully be a good defense against that as more people get into cybersecurity, become aware and get interested in being involved. We will be able to protect from those threat actors, those bad guys. Yeah, I think that's it.

Aaron Crow (65:23.180): Yeah, absolutely. dig it, man. All right. So how can people, you, you talked about a couple of organizations that you have, how can people find out if they're in the Houston area, they want to reach out on your, your ha ha and, the lock pickers as well as maybe see you speak at a conference or what do have coming up that you want people to know about?

Dennis Maldonado (65:43.684): Yeah, so I have LinkedIn and Twitter. I'm not as active in Twitter as I'd like to be, but I'll try to be more. But mainly on LinkedIn, I do post a lot of what's going on in LinkedIn. I answer a lot of messages or questions. So you can find me. I'm Dennis Maldonado on LinkedIn. I think my Twitter handle is Dennis Linus. But as far as I don't have a lot of

Dennis Maldonado (66:4.714): Super public things that I'm talking about, but I know I actually got asked to be on the hue sec cast podcast That's a hue sec con podcast. I'll be on that Sometime next week. I'm not sure when that'll be actually released but it is October is cybersecurity awareness month So I have had a lot of engagements people are companies asking me to give talks either for their organizations or panels everywhere so I think the next public one will be a

Dennis Maldonado (66:31.370): Sam Houston State University have asked me to be on a cybersecurity careers panel. So no idea if that's public, but if anyone wants some advice on cybersecurity careers, you can go to that. I'm speaking at a local ISACA conference here in Houston. That's certainly public. I'll be doing that October 25th and then various other small meetups and universities and stuff like that. if you need anything, if you want to reach out, ask more about Westcom or hell, you want to tour, hit me up on LinkedIn and I'll

Aaron Crow (66:50.926): Very cool.

Dennis Maldonado (67:1.208): I love talking about this, so I'd love to do it.

Aaron Crow (67:2.752): Awesome, man. Hey, I really appreciate your time today. Hopefully it wasn't too painful. I know you were, you were a little anxious around it, but dude, you did great. I really appreciate your time and it's great knowledge for folks. This is what people enjoy. Like they want to hear the real stories behind the scenes of what's going on, how, what was your thought process and how you got there? Cause I think we all know that there is no right answer to this, right? Is it's an ever changing environment.

Aaron Crow (67:25.848): where we have to constantly improve and learn and grow and all that kind of stuff. And that's no different if you're in critical infrastructure, you're a mom and pop shop, right? It's the same principles. It's just a matter of where that risk tolerance is and adjusting accordingly, right? So thanks again for your time, man. And it was a great conversation. I look forward to maybe coming down and doing a tour of your system and seeing that in person.

Dennis Maldonado (67:49.592): Yeah, anytime. Come on down. I love showing it. So my tours range from 15 minutes of showing around to like three hours if we talk and have, you know, ask questions, whatever you, I love showing off the place. Anytime. And thanks for having me. It was fun being, I was a little nervous, but when you get me talking about the topics I'm very passionate about, the time flew by, but thank you for having me.

Aaron Crow (67:59.565): Awesome,

Aaron Crow (68:9.024): Absolutely. Absolutely. Well, thanks man. Appreciate it. again, thanks for your time and, definitely reach out to Dennis, for more information.

Dennis Maldonado (68:19.088): All right, bye.

Transcript lightly edited for readability.