Three Days Later, Minnesota

Three days.

That is how long it took between me writing about advisory AA26-097A and more than 30 Minnesota community water systems finding out what that advisory looks like when it stops being a PDF and starts being a Monday.

I am not going to spend a word on being right about it. Nobody in Braham cares. And honestly, the advisory was not a prediction. It was a description of things already happening in places that had not gotten around to telling anyone yet.

Here is what we know. On July 26 and 27, coordinated malicious cyber activity hit more than 30 community water systems across Minnesota. The state and MNIT disclosed it on July 28. In Braham, a town of about 1,700, attackers disabled the computerized operating controls. The well and treatment plant went offline for roughly two hours on July 27, announced at 9:34 in the morning and restored around 11:30. The town ran on tower reserves and asked people to go easy on the water. No boil order. The city called it what it was, a malicious cyberattack on computerized operating systems by unknown actors. The public works crew figured out the problem and dealt with it.

In Plymouth, about 80,000 people, they disconnected cellular-connected equipment at two water towers and several wastewater lift stations and kept running on manual procedures. Their words: water levels and water quality are unaffected, the water is safe. South St. Paul had some automated controls affected and kept services running. Maple Plain, about 1,800 people, had automated control functions impacted, activated contingency procedures, and the mayor declared a local state of emergency to pull in help faster.

Read that list again. In every disclosed case, the water stayed safe.

That is not luck. That is people.

The part that is not new

I said this last week and I will keep saying it. Fundamentals win every time.

Nobody picked Braham. Nobody sat in a room and said, let us take down a town of 1,700 in east central Minnesota. This looks like the first genuinely distributed attack of its kind, dozens of small utilities hit at once, almost certainly linked by a shared weakness class rather than 30 separate targeted operations. A shared integrator. A common communications architecture. A device class that answers when you knock. A scan found something that answered, and the scan did not know it had found a water tower.

The scale is the story. Minnesota recorded nine attacks on critical infrastructure in all of last year. This was thirty-plus in forty-eight hours. No ransom demand, no data theft, no obvious financial motive. The objective was disruption. And think about why water keeps ending up in the crosshairs. Minnesota has fewer than 100 electric utilities and more than 1,000 water systems serving roughly five million people, most of them run by small teams on tight budgets. Fragmentation is the attack surface.

From the book: you do not get hit because you are special. You get hit because you are reachable, and because something you are doing makes you easy.

There is a thread worth watching here, and I am going to be careful with it. CSO Online reported on July 30 that investigators are looking at whether exposed Rockwell Automation MicroLogix 1400 controllers were a common enabling factor, and that a researcher found two of them sitting on Plymouth’s public IP space through Shodan, inside cellular carrier IP space. Rockwell issued an advisory covering that family the same day. Everything about this pattern says broad scanning for widely used internet-exposed devices that happened to find a concentration of Rockwell PLCs in Minnesota, and if that holds, the urgency goes well beyond one state, because those same controllers are sitting at lift stations and well houses everywhere. Still, that is an open question, not a conclusion. Officials have not publicly named the vector, the product, or the vulnerability. Exposure is not proof of exploitation, and I am not going to hang a verdict on a Shodan hit.

What I will say is the durable part. A controller can be actively supported and still carry a design-era security model, which in plain language means essentially no meaningful authentication on the control protocol. If you can reach the port, you are effectively the engineer. That is not a defect anyone hid. It is what industrial control looked like when it was built for a wire between two cabinets. The 1400 family has an advisory trail going back more than a decade, including 2017 findings rated CVSS 10.0 where an unauthenticated attacker could read the master password and change ladder logic. And here is the budget trap: “Active” status never triggers a replacement line item the way an end-of-life notice does. The most complex things we run are controlled by the simplest.

While we are being honest about causes: cellular is not segmentation. Small utilities put PLCs at lift stations and towers on cellular modems with public static IPs because it is cheap and it works. That connection bypasses your own perimeter entirely. It gets indexed by Shodan and Censys permanently. And it hides the owner from researchers trying to warn you while hiding nothing at all from an attacker. Ask your carrier about a private APN. Put a VPN-capable industrial gateway in front. Or make the telemetry outbound-only.

On attribution: there is none, formally. Officials say preliminarily that Iran is the likely culprit based on tradecraft and the absence of any ransom demand, and they are clear it could change. No group has claimed it. An Iranian state-linked outlet pointed at Handala, but Handala has said nothing about Minnesota. The tradecraft looks like the CyberAv3ngers ecosystem we have been watching for three years. Fine. Here is the operator version: your exposure is opportunistic and your impact is not. Adversaries benefit from ambiguity. You do not. Do not wait for a name before you act. It is a swarm, not an army.

Left of bang gets all the money

Bang is the moment it goes wrong. Left of bang is everything you do to prepare. Right of bang is everything after.

Almost every dollar in this industry goes left. Firewalls, assessments, exposure reduction, awareness training. Do all of it. I am not knocking any of it. Get your PLCs off the public internet today, terminate remote access at a VPN or controlled gateway and never at the controller itself. CISA is saying the same thing publicly, and on July 30 issued fresh guidance for water and wastewater utilities saying exactly that. They are seeing a significant increase in actors going after PLCs at water systems. That is left of bang, and it is the highest-value work you can do this month.

But Minnesota is the argument that left of bang is only half a program. Because on July 27, both halves got tested on the same day.

The reason those endings were good is right-of-bang muscle that nobody in a budget meeting calls a cyber program. Operators who could run the plant by hand. Tower reserves. Written contingency procedures somebody actually practiced. A public works crew in Braham that diagnosed and mediated an attack in about two hours without a SOC, a threat feed, or a vendor on retainer.

Water also has a physical grace period, and it is worth naming. Water systems are mostly gravity-fed, with real elasticity built into them. Shut down a pump or a well and the towers and the physics keep things moving for hours, sometimes days, before anyone’s health or safety is at risk. That buffer is exactly the window manual operations needs. But a buffer only saves you if somebody knows what to do with the time. Gravity bought Minnesota the hours. People used them.

Celebrate those people. Then notice the uncomfortable part: that muscle is use-it-or-lose-it, and automation quietly erodes it. Every upgrade that makes the plant easier to run from a screen makes it a little harder to run without one. If the last person who knows the manual valve sequence retires in March, you did not lose an employee, you lost a control.

The half of right of bang almost nobody has

Manual operations is one half. Here is the other half, and it is the one I want you thinking about.

After you stabilize, somebody has to answer two questions, fast. What did they touch? And is the logic running in my controller the logic I approved?

For most small utilities the honest answer is, I do not know. The only copy of the ladder logic lives on the integrator’s laptop. And per AA26-097A, the adversary may have a copy of your project files too, pulled with your own vendor engineering software, with changes tucked inside Add-On Instructions where a human eye skips right over them.

Remember how that advisory described the trick. Shutdown and alarm logic disabled underneath, while the screens looked completely normal. The screen is not evidence. You have to read the controller.

This is where a capability class belongs in your program, and it barely exists in most of them. Automated, scheduled backup of PLC programs. Version control, so you have a known-good baseline with real history behind it. Continuous change detection that alerts you when the logic running in the field differs from the logic you approved. Visual diff, so an engineer can see exactly what changed, including inside the reusable modules where things hide. And fast, verified restore, so recovery is measured in minutes and hours instead of days spent reconstructing from somebody’s archive folder.

And here is the good news. This is a mature category with real options in it. Copia brings a modern, Git-style approach to version control and change detection across multi-vendor OT fleets. AMDT’s octoplant, the platform that grew out of versiondog and AutoSave, has been doing automated backup and version management across environments with over 160 vendors’ equipment for years. If you are an all-Rockwell shop, FactoryTalk AssetCentre does scheduled device backups, compares, and audit trails inside the ecosystem you already run. Your OT network monitoring platform may also flag configuration changes on the wire, and that is useful, but detection without a versioned baseline you can compare against and restore from is half an answer. Call the whole category OT DevOps if you like the phrase, or call it backups for the stuff that actually moves water if you do not.

I do not care which one you pick. I care that somebody in your program owns the question, is the logic running in the field the logic we approved, and can prove the answer on demand. If those 30-plus Minnesota utilities had that in place, “did they modify our logic?” would be a report you pull before lunch instead of a forensic project you fund next quarter. Braham’s two-hour recovery was operators being good at their jobs. A verified baseline is what makes that repeatable by design instead of dependent on heroes being on shift that morning.

Now the caveat, because you have heard me say it before. There is no silver bullet. None of these tools will save you if you buy one and walk away. Who reviews the diffs? Who owns the baseline? How does a legitimate change get approved so the alert means something? When did you last actually restore from it, on purpose, on a slow Tuesday, like a fire drill? Buying the platform without the process is buying a smoke detector and never putting batteries in it. People over tools, every time. But this category belongs in OT and cyber programs the same way backups belong in IT, and almost nobody budgets for it.

And let us be clear about the field we are playing on. There are roughly 50,000 community water systems in this country. About 81 percent serve fewer than 3,300 people. More than half serve 500 or fewer. AWIA risk assessment requirements only reach systems above 3,300, which means roughly four out of five systems sit below the line where anything is required at all. EPA has no authority to mandate controls. Water has nothing like the electric sector’s mandatory NERC CIP program.

So let us say the quiet part out loud. We are asking small local governments to defend essential systems against foreign adversaries and sophisticated criminals, with limited staff, aging technology, and budgets that were never built for this fight. That is unfair. It is also Tuesday. You still have to do it anyway. Do what you can with what you have.

What to do this week

1. Find your own exposure before someone else does. Search Shodan and Censys for your public IPs. CISA publishes a how-to, and free vulnerability scanning is available at [email protected]. Left of bang.
2. Walk your remote sites and inventory every cellular modem, including the ones nobody remembers installing. Pull the carrier bill and account for every SIM. Unknown SIM, unknown door.
3. Get PLCs off direct internet exposure. Remote access terminates at a VPN or controlled gateway. Never at the controller.
4. Call your carrier about a private APN, or put a VPN-capable industrial gateway in front of that modem. Cellular is not segmentation.
5. Change default and shared passwords on everything that has them. Long, random, unique.
6. After validating the project file, put the physical mode switch back in RUN. Free, and it raises the bar.
7. Get your own project files from your integrator. All of them. You cannot verify what you do not have. Right of bang starts here.
8. Stand up automated backup and change detection for control logic. Copia, octoplant, FactoryTalk AssetCentre, or at minimum a disciplined manual baseline-and-compare process you actually follow. Treat an unauthorized logic change alert like any other alarm.
9. Drill manual operations like a fire drill, before you need it. Then validate a restore end to end. An untested backup is a rumor.
10. Know who you call at 2 a.m. CISA offers free incident response at 888-282-0870. FBI. WaterISAC. Your state. Put the numbers on the wall, not in someone’s phone.

Minnesota was the fire drill you did not schedule. The water stayed safe because people had practiced being human when the computers lied. Build the other half now, so next time you can also answer, with a report and not a shrug, exactly what they touched.

Pick one thing off that list this week. Just one. If you only have room for a single item, make it number seven, get your project files in your own hands, because everything on the right side of bang depends on knowing what good looks like.

Hope is not a strategy. Not left of bang, and not right of it either.

Stay safe and secure out there. Let’s get to work, and let’s protect it all.