If we have a cyber incident, my IT team will get us back online asap.
I'm sure you saw on the news recently that the city of Saint Paul Minnesota experienced a major sophisticated cyber attack. (link) The first detected suspicious activity was on July 25 which rapidly escalated. On July 28 officials made the difficult and necessary decision to shut down all municipal information systems, including public Wi-Fi in libraries, internal applications, and even payment systems to contain the threat and prevent further compromise. While 911 operations remained functional most other city systems were down and as of August 4 remained off-line. At the time of this writing there is still not an estimated time of these systems being restored to operation.
This process is not unusual when there is a cyber event.
Incident Response Process
When any cyber incident takes place an organization must follow its incident response plan. The incident response plan will outline the incident response process that the organization needs to go through and reduces stress by providing a playbook to team members and executives on what to expect and how to make decisions in a time of crisis. The incident response plan and process includes a number of components, but at a high level includes the following major steps:
- Detect and identify
- Contain the threat
- Eradicate the cause
- Recover operations
- Document and learn
Without going into any deeper detail, you can see from this list of steps of recovering operations is toward the end of the process! Most often to properly contain the threat it is required that systems be taken off-line. Having an established incident response process and doing a tabletop exercise of that process on at least an annual basis helps everyone in the organization to understand the outcome of such a cyber event.
Plan to have systems offline... possibly for a while
Your incident response plan will likely be unique from that of other organizations. However, in the smaller organizations we work with typically would follow a similar process. Most likely their IT services are provided by a managed service provider or other third-party, or perhaps they have a very small internal IT team. Those teams and providers are likely not positioned to be skilled incident responders. The actual incident response would fall on a third-party incident response specialist. That incident response team, most likely will be dictated by your cyber insurance carrier and their preferred vendors.
But before you even get to the point of engaging in incident response, your first step most likely would be to contact your cyber security insurance carrier to open a claim and begin action. While that carrier will act very swiftly, there could be a delay of a day or even more before teams are on site and mobilized. So before you even know much about your incident and the scope of which you're dealing with, you're already into potentially a day or more of downtime. As the incident response team lands on site follows the process does their evaluation, depending on the type of incident you've encountered easily this can stretch into days or even weeks of total or partial systems downtime.
How to prepare
Step 0: Leadership Buy-In
While there are lots of guidelines out there on how to create an incident response plan and process I believe that the first step is actually organizational awareness. The owners of the company, the leadership team and other stakeholders need to clearly understand the importance of building this plan. While I love most of my customers I will say that it is very difficult to motivate the average business owner to understand the importance of this plan. Despite what I say, there is a belief that the IT provider will make everything right and will take care of it all. Which, unfortunately in this day and age is just not true. It takes a village to support any organization that becomes the victim of a cyber incident, especially when the incident is perpetrated by unknown third parties that could even be large scale, sponsored actors.
Regardless of the size of your organization, we recommend the following high-level steps:
(Yes, even small 10 employee companies should follow this process!)
1. Create an Incident Response Plan and Test Annually
Even a short document outlining:
- Who has ownership of the document? Where are copies located? (especially if systems are offline - need hard copies!)
- Who’s on the Incident Response Team? Does the team have the authority to act and purchase needed items/services?
- The high level steps to be taken
- Who to call (IT consultant, legal, insurance carrier, etc)?
- Who is empowered to open a claim?
- A public relations communication plan. Who is authorized to speak on company's behalf, and potentially deal with media, customers, and external stakeholders?
- Contact information for each identified team member
- Verified backup and restoration procedures
- Downtime plans
This plan doesn’t need to be complex, but it must be written down and known by everyone in the organization. Once authored, an annual 'tabletop exercise' is recommended where you work through an incident like it is real, to identify deficiencies in your plan in order to keep it updated an appropriate.
2. Test / Update Your Downtime Processes
Assume your systems—email, CRMs, cloud docs, payment portals—could be offline for days or even weeks.
- Could your team function using just phones?
- Do you have offline access to critical customer contacts?
- Are your backups tested and recent?
- Rank your data needs from most important to least important. Be sure to clearly talk this through... while Payroll is everyone's first thought, it usually is not that important. (Because you could likely have your payroll team just re-run the previous pay week to get by.. whereas other items might be much more critical to the business!)
3. Build External Relationships Now
If you’re breached, you’ll need help... likely very quickly!
- Your IT provider or IT department
- Additional emergency IT staff to augment existing teams (very important during the recovery phase!)
- Your attorney or legal/compliance expert
- Purchase Cybersecurity coverage, and understand how to access your cyber liability insurance carrier urgently to open a claim. Also understand what their required Incident Response process is, such that you aren't doing double work, or accidentally invalidating your coverage by moving too fast or with the wrong people/process in place!
Trying to find any of this information - resources, procedures, plans and even phone numbers - after an incident starts is far too late. Remember - you may be in a place to need all that info and have no functional computing devices or systems!
The City of St. Paul Had A Plan
When the mayor declared a local state of emergency in response to the security incident, it empowered the city's Emergency Management Department and the Office of Technology and Communications to mobilize resources and trigger response. This included processes such as bypassing procurement for rapid action. It also activated its emergency operation center, which coordinates incident response across multiple agencies including state, federal and other cyber security partners.
This is clear evidence that the city had formal response procedures, and infrastructure in place well in advance of this incident.
Despite this their systems continue to be offline.
This underscores the importance of understanding that prolonged system downtime during the investigation and recovery from an incident is to be expected even in the best of cases. In this case, Saint Paul has done, in my opinion, the best they could do in terms of preparedness and are following the process that's been established. What they are experiencing is completely normal.
When I watched the first press conference about this incident it was clear that they were following their incident response plan. They had the right team members there, they each understood their roles and they had a plan for communicating with the public and the media. I could tell simply by watching that press conference that they were prepared. And bravo to them!
Can you only imagine the outcome if the incident response plan didn't exist? It likely would've resulted in a response that was substandard or completely absent, potentially exposing more data to bad actors and widening the scope of the incident. Even worse chaos may have ensued.
The Takeaway: Size Doesn’t Protect You. Preparation Does
The St. Paul attack shows that even large, well-funded organizations can be overwhelmed. But it also shows how preparedness can limit damage and guide recovery.
For small businesses, the lesson is simple:
- Have a plan.
- Expect downtime.
- Know who to call.
You don’t need a million-dollar budget—you need readiness, realism, and the discipline to prepare before disaster strikes.
If you’re not sure where to start, draft a simple checklist, simulate a basic outage with your team, and ask yourself: how long could we operate if everything digital went dark tomorrow?
If you'd like help with this process, let's discuss by clicking here to setup a call with us. One of our certified Incident Responders will gladly assist you in developing or tabletop testing your plan.
Because in today’s world, it’s not “if, ” its “when.” And resilience starts with planning.
We are here for you
Have a technology issue or question? Wondering if your current IT is really working for you?
Setup a free, no obligation 15 minute chat with Darren by clicking below.